Pilot mental health – can AI-powered psychological assessment help?
IFA comments: For crew, engineers and controllers, the ‘predictive’ approach is desirable. However, the practicalities when applied to the immensely diverse sets of circumstances in aviation may create even more uncertainty.
In the week that an expert committee recommended to the FAA that pilots be allowed to get ‘talk therapy’ from mental health professionals without having to disclose it in their medicals, MARC ATHERTON MRAeS, from the RAeS Human Factors Specialist Group investigates to what extent risk can be monitored and quantified by means of the psychological assessment of safety-critical staff.
Poor mental health is now an acknowledged safety risk in civil aviation, From Germanwings in 2015 through to Alaska Airlines in 2023 the issue of the mental wellbeing of safety-critical staff has become an issue for the industry.
One key question relating to this challenge is to what extent can the risk be monitored and quantified by means of the psychological assessment of safety-critical staff to inform safety management strategies.
The word ‘psychological’ comes from the Hellenic words ‘Psyche’ and ‘Logos’. Psyche translates to soul or spirit; Logos translates to knowledge.
Competencies and personality
In April 2024 an expert committee formed by the FAA on pilot mental health recommended that pilots be allowed to get ‘talk therapy’ from mental health professionals without having to disclose it in their flight medical. (Pixabay)
In aviation typically we use psychological assessments in two areas: competencies and personality.
Competencies are relatively ‘simple’ to assess. Define an action in objective terms (% correct, errors made), and assess whether an individual meets a defined value on that criterion as a pass/fail mark.
Personality attributes are more challenging. There is an generally accepted model of personality, the Five Factor Model (FFM), which describes ‘normal’ personality as a profile with a ‘score’ on each of the five elements. How this personality is linked to mental health concepts is the subject of academic debate, but currently mental health characteristics are seen as distinct from, but linked to, FFM characteristics. The reference sources for mental health issues are the Diagnostic Systems Manual v5.0 (DSM5), and the World Health Organisation (WHO) International Classification of Diseases v.11 (ICD11).
One way of standardising assessments, to provide validity, reliability, and utility information as a part of an assessment is through the use of psychometric (psyche – soul, metric- measure or enumerate) tools.
The topic of mental health in aviation is salient; aviation technology (techne – machinery, logos – knowledge) is incredibly safe and reliable. The human element is the opaque risk factor. When we say ‘safety-critical’ stakeholder groups we are tacitly accepting that they need to be included in our safety management systems as ‘risk vectors’.
There is a lot of interest in whether these ‘risk vectors’ (i.e. people) can be psychometrically assessed for ‘quality fit’ in the way that a component can be for risk management. Currently we have a reactive approach (somebody has an event, the behaviour is problematical, a professional assessment is made, and a diagnosis and treatment plan defined).
There is, I would argue, a siren song informed by a fundamental misunderstanding of individual psychology and the limits of measurement. When we perform psychological assessments, we are using proxy measures of overt behaviour (speech, actions, social and digital ‘footprints’ etc) to infer the internal state of an individual. The general mapping is neither precise, prescriptive nor predictive, even in a statistical sense, beyond population characteristics. For example, one in six people will have a common mental health disorder in any given year according to agreed definitions and criteria (in typically rich Western countries). Remember that going from the population statistics back to an individual risk probability is statistically forbidden.
A poor Psyche
Poor mental health is now an acknowledged safety risk in civil aviation and the issue of the mental wellbeing of safety-critical staff has become an issue for the industry. (Pixabay)
Where does this leave civil aviation if we want to identify and mitigate the risk posed by poor mental health to safety and performance? If we accept there is a thing called ‘poor mental health’ (a poor psyche), and that it can pose a risk to operational safety and performance then, in a Safety 2 environment (data-driven predictive approaches to safety management rather than reactive approaches), is there a means of creating the data (metros) to inform our decision making and mitigation approaches?
As Thorndike (a founder of scientific psychology) said in 1914 said: “If a thing exists, it must exist in some quantity, and if it exists in some quantity, it can be measured in some way.”
Given the points above then, could we do better than throw our hands in the air and say, ‘too difficult’. Surely, we must have the potential to have a valid, reliable, practical, and usable current state assessment of a person, and perhaps even a predictive states assessment, to inform our risk assessments?
I would argue that there are several possible ways to move towards the ‘metros’ of ‘psyche’ that may be worth considering, although they all have both knowledge and practical limitations in achieving the goal of assessing an individual for the level of risk posed to safety. These need to be both remembered and considered when looking at the role of psychological assessment in civil aviation.
Standardised psychometric assessment tools
1. Most psychological assessments use a blend of both quantitative (psychometric) and qualitative (interview) elements. In most jurisdictions the assessment must be delivered by a suitably qualified and accredited individual to have any legal standing.
2. Currently the assessment for pilots (as only one stakeholder group) is done by an AME on an annual or bi-annual basis as part of the medical requirements for licencing. Under EASA Regulations new entry pilots must have a ‘psychological assessment’ undertaken by an accredited aviation psychologist, as do pilots changing airlines. This is outside the Part Med rules pertaining to AME annual licencing assessments. Standardising this is a major challenge as neither the psychometric or clinical aspect of the assessment is a core competency of the typical AME.
a. Given this limitation, is there perhaps an assessment that, with normative benchmarking to the relevant pilot population (not all pilots are from a Western culture), could provide a valid, reliable, scalable, and practically useable assessment type that could point to diagnostic standards of metal health conditions of the type in the DSM 5 and the ICD 11 at the time of delivery?
i. In the USA currently the Minnesota Multi-Phasic Personality Inventory (MMPI) is used in areas of aviation recruitment. The MMPI was originally developed to address personality and psychopathology issues for individuals. It was primarily an instrument for assigning a diagnostic label to people with an assumed psychiatric condition. The short version currently used has 338 questions and can only be delivered and assessed by a licenced professional. It is highly researched and validated and is used outside of clinical diagnostic situations in aviation selection.
ii. The MMPI having a clinical background, being prescriptive in nature, and requiring a licenced psychologist or other medical professional to deliver it may pose a challenge for a more widespread use in the industry.
iii. What might be a better approach would be a non-prescriptive assessment, linked to the DSM5/ICD11 structure, which is shorter and could be delivered at scale by supervised non licenced staff,
3. I would suggest that there are already several contenders that could, with limited development, meet the validity, reliability, scalability, and usability criteria that the industry would require using the same fundamental approach of the psychological assessment being embedded in a professional led interview linked to the annual licencing requirement.
a. The first contender is the Structured Clinical Interview for a DSM SCID 5 assessment (SCID 5)
i. Information on the SCID 5 can be found here: https://www.appi.org/products/structured-clinical-interview-for-dsm-5-scid-5 .
ii. The possibility exists for this to meet the ‘current state’ assessment in an industry standardised fashion so that all individual assessments are comparable on a large scale.
iii. Unlike measures like the existing MMPI, it does not require statutory licencing to use, and can be deployed at scale under the supervision of a licenced professional by trained administrators in an organisation.
iv. The validity and reliability of the SCID 5 lies in the measure itself, not the individual delivering it.
b. The second form of contender might be based on the online assessment technology approach of the type represented by the US based Clinicom technology.
i. This approach uses an online assessment which is ‘mapped’ to the DSM 5 categories the output of which can inform a clinicians decision- making regarding an individual’s mental health state.
ii. The approach also uses machine learning to facilitate the assessment, but is a decision aid, not a prescriptive diagnostic tool.
c. The issue for safety risk with these forms of assessment is that they are a ‘state now’ assessment, not a predictive assessment.
d. The use of this form of assessment, enhancing what is done now in supporting professional judgement with standardised procedures and data should be relatively uncontentious, their utility high, and their cost relatively low.
e. Either of these approaches, or a similar equivalent, may be the basis of a standardised ‘now’ assessment with implications for individuals to be negotiated and defined.
f. By using a standardised approach, collection and collation of population data becomes feasible, possibly allowing the use of data analytics and machine learning to identify patterns which could inform risk management approaches on a large scale.
4. Predictive psychometric assessment tools
The ‘holy grail’ of the concept of psychological assessment is one that can tell the assessor what the future state of the individual will be based on a ‘now’ assessment. In an age of Artificial Intelligence (AI) and Machine Learning (ML) are there possibilities that we could find an answer there.
5. Since what we assess are overt actions: speech, actions, social and digital ‘footprints’, and associated criteria, can AI and/or ML perhaps provide us with appropriate insights to create a ‘predictive assessment’? AI and ML have the ability to look across large data sets and see connections and correlations that defy human understanding. They are promoted as having the ability to ‘see’ what humans cannot in a sea of data and develop new insights beyond human capabilities.
a. Given this possibility, are there perhaps tantalising possibilities in existing research that could point the way towards a predictive approach to individual mental health issues that could further de-risk the industry and allow a Safety 2 approach to mental health in civil aviation?
b. Well, there are some interesting technologies based on AI and ML that could inform a debate on the topic of predicative assessments in civil aviation from a clinical, operational, technological and legal perspective.
i. DCAPS (Detection and Analysis of Psychological Signals). A technology developed by MIT for the US Veterans Administration to remotely monitor specific mobile-phone registered mental health and wellbeing related behavioural indicators for a dispersed population of individuals undergoing clinical support programmes. The system automatically can flag up to a designated clinician when an individual’s indicators exceed specified bounds, triggering and intervention action.
ii. Prosodic and linguistic analysis of naturalistic speech patterns when using mobile-phone technology on an idiographic and normative basis (the personal patterns and a designated normative group pattern). Research at Columbia University has shown that the technology can predict psychotic episodes with a high degree of reliability in a clinical population based on changes in their prosodic (tonal patterns) and linguistic (vocabulary used) behaviour compared to their own ‘typical’ patterns and a group pattern.
iii. Monitoring Heart Rate Variability in individuals using mobile technology on an idiographic and normative basis (the personal patterns and a designated normative group patter). HRV has a high degree of correlation with stress levels, both episodic and chronic. High levels of chronic stress are a strong precursor to CMHD (Common Mental Health Disorders).
iv. In cockpit monitoring and assessment of video imagery of an individual mapping behavioural indictors to both idiographic and normative responses to situational variables as a predictive approach.
v. Technology using electronic and social media ‘digital footprint’ data to assess the psychological and mental health state of individuals has been deployed before (think Cambridge Analytica) with evidence that it cannot only assess psychological state, but also provide insight to alter it in a chosen direction at an individual level. Could this technology, or as an assessment approach based on it, be used to monitor individuals continuously?
vi. Using Chatbots to assess individual responses to structured conversations, and monitor their ‘digital footprints’ continuously which could integrate some, or all, of the preceding possibilities into a ‘pseudo-clinical on- condition monitoring’ approach as a standardised psychological assessment.
c. I would suggest that these AI and ML approaches are at best distant potentials that would require significant research and development before we could have any view of how they could meet the validity, reliability, scalability, and usability criteria that the industry would require, let alone the legal issues related to the possibility.
d. The possibility for this form of approach to meet even the ‘current state’ assessment in an industry standardised fashion so that all individual assessments are comparable would need to be established before any predictive use was considered.
e. The challenge with the form of assessment would be in its probabilistic nature, and possibly being seen as a ‘black box’ technology, if used as a predictive assessment. Any AI or ML approach would likely be highly contentious, its’ utility uncertain, and its’ cost relatively high (developing AI and ML models is expensive). All of these would need consideration before any policy could be formulated.
Summary
Currently the assessment for pilots is done by an AME on an annual or bi-annual basis as part of the medical requirements for licencing. (Pixabay)
a. The purpose of this note is to hopefully provide a quick summary of both the possibilities and the limitations of what I see as the two potential approaches to assessing the psychological state of safety-critical individuals in civil aviation in an enhancement to what is done currently.
i. The ‘now’ approach enhances what we do now in a standardised and scalable way. Whether this has a meaningful and positive impact on risk and safety is not clear and would require consideration. Whether if would have benefits for the individual is also a point for discussion.
ii. The ‘predictive’ approach, based on AI and ML approaches has many facets, but they are all currently subject to many uncertainties and I am of the view that we need to avoid getting caught up in what Gartner call the ‘Hype Cycle’ around AI and ML, step back, take a breath, and do some thinking before we leap on a possible bandwagon,
b. We need to remember that we are dealing with a complex industry, with a global presence, that already has an enviable safety and operational performance record. The mental health and wellbeing of its’ staff, for both safety and duty-of-care reasons, is one being taken more seriously by all key stakeholders.
c. We also need to remember that staff work for organisations, and the mental health and wellbeing of those staff is also the responsibility of the organisations, and those organisation’s work patterns and culture, to support as a matter of regulation and legal rules.
d. Even if we could perfectly predict that an individual will have a mental health episode in the future, we would have to ask the question ‘to what extent did the existing work patterns and culture contribute to the episode?’
No man is an island, entire of itself; every man is a piece of the continent, a part of the main. (John Donne 1572 – 1631)
Disclaimer: The author has no commercial or professional connection to any of the approaches identifies in this document. Any approaches identified are used as exemplars only and there may alternatives available. Readers are advised that identified approaches are not product recommendations, and independent decisions should be made prior to adopting any approach.
