Understanding ‘one in a billion’ in aircraft system safety assessments
by John Vincent 8th Dec 2021
Link to view presentation given during Rotorcraft and VTOL Seminar
We take a flight believing that our aircraft’s design and construction are based on sound foundations. It’s not a leap of faith. Safe flight can only be made with rock solid aircraft design certification. An integral part of that aircraft design certification is system safety assessment. The experience of use, accidents and incidents have guided the refinement of aircraft system safety assessment. Briefly we’ll go on a journey through the last 70 years of civil aviation, visiting the basis for the probabilistic approach to system safety assessment that is now shaping development of a new generation of eVTOL aircraft. Next, we’ll look at applying safety objectives and finally, we’ll consider the case for the use of a performance-based approach to aviation systems safety.
Looking back through the history of civil aviation, rules were found necessary to drive down the numbers of fatal accidents. Civil airworthiness codes (e.g., FAR/CS 25, 23, 29 and 27) have evolved from their original empirical and prescriptive nature.
To quote an early airworthiness code: “Chain sprockets shall be guarded, so that it is impossible for the chain to jam or override the sprocket.” Empirical and prescriptive regulation leaves an aircraft designer with little scope. It’s limiting even when there may be a better alternative solution at hand.
Before the 1950s, civil aircraft used comparatively simple systems. They were self-contained so, most often, the failure of one aircraft system did not influence the continued safe operation of another. One exception to this condition was that of an aircraft’s electrical system, where the effects of combinations of failures swiftly became evident. Electrical systems were used to power several systems upon which an aircraft’s safe flight was dependent.
In the U.S., the Code CAR 4b started to recognize interdependences in its text, e.g., “The system should not be rendered inoperative by any probable malfunction, if operation of this system is necessary to maintain controlled flight or effect a safe landing for any authorized flight operation.”
Thus, the terms of risk and probability started to become a key part of aircraft certification.
In the early 1960s, it was the advent of automatic landing systems that changed the game. These aircraft systems were significantly more complex, and not obviously safe when compared to what had gone before because of the number of interfaces, connections, and dependencies.
Conventional requirements were considered too constraining. It was then that the language of probability took center stage, and numerical analysis came to the fore. The certification authorities developed objective requirements where an expected level of safety was specified. Then it was for the systems designer to do the analysis and tests that were needed to show that the safety objectives had been met. Safety related interdependencies had to be addressed. Therefore, potential failures needed to be considered separately, and in combination with each other.
Then Concorde came along with advanced technology and highly complex systems, which accelerated the need for safety objectives. Supersonic transport standards were created to facilitate certification and make use of the concept of system safety assessment. Gradually, as often in aviation, a method that was proving to be practical and successful spread to the airworthiness codes for large airplanes.
Next came the agreement of principles and the definition of generic safety objectives. For the first part, when introducing a new aircraft there must be no increase in risk of accidents caused by system failures. This simple principle demands a study of aviation safety records.
In the 1970s, considering all causes, there were roughly four accidents per million hours of operation worldwide. Accepting the proposition that 10% of those accidents can be related to aircraft systems, that equates to four accidents per 10 million hours. On the basis that an objective should aim to improve aviation safety, a systems safety objective of less than one accident per 10 million hours was set for aircraft systems.
Aviation accident statistics are expressed in diverse ways. They can be expressed as per million flights, per million departures, or per million flight hours. However, it is time of exposure to risk that relates best to our lives.
The generic system safety objective for all systems of less than one accident per 10 million hours can be written as 1 x 10^-7 per hour. Assuming that there are 100 failure conditions in a typical large aircraft which can cause a fatal accident, then a factor of 100 is introduced. Thus, the safety objective for each of these failure conditions becomes a maximum of one accident per billion hours or 1 x 10^-9 per hour — which is the failure rate that the European Union Aviation Safety Agency (EASA) plans to apply to “Category Enhanced” eVTOL aircraft that will carry passengers over congested areas.
Working with small probabilities can be hard to picture in terms of day-to-day experience. If someone is lucky enough to live for 100 years, that equates to roughly 876,000 hours and is consistent with an objective of one fatal accident per million hours.
The next fundamental principle is that of proportionality. Having fixed a safety objective for the most severe failure outcome and knowing that at the other end of the scale there are failures of no consequence, it’s necessary to define several intermediate classifications. There can be a series of conditions that reduce the capability of an aircraft, or the ability of the crew to cope with adverse operating conditions or have effects on its occupants.
Thus, the inverse relationship between the probability of an occurrence of an event and the severity of its effect that is detailed in the Advisory Material to Certification Specification CS 25.1309 for large airplanes. In the case of EASA’s Special Condition – VTOL, the 1 x 10^-9 rate applies specifically to catastrophic failures in Category Enhanced aircraft, while minor failures can have a probability of occurrence of 1 x 10^-3, or one in a thousand flight hours.
The case for revisiting assumptions
With principles in place, the process of system safety assessment presents a system designer with seven key questions. What is the system? What does it do? What can go wrong? What happens if it goes wrong? What can cause it to go wrong? What is the risk? And can we accept the risk?
The first three questions contribute to a functional hazard assessment (FHA) which is used to establish the safety objectives for system functions. At the aircraft level there are few conditions that can lead to a catastrophic event: for example, unrecoverable loss of aircraft control or misleading or loss of all displayed cockpit information for flight in instrument conditions.
The FHA is the first step of the three parts of a system safety assessment. Next step is a systematic safety analysis undertaken to determine how an aircraft system meets the safety objectives that have been set. Finally, evidence to demonstrate regulatory compliance, provide limitations, support emergency and abnormal procedures, and deliver candidate maintenance requirements is completed.
Aircraft certification authorities have given recognition to the industry standards on safety assessment. For example, SAE ARP 4761: “Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment” is extensively used.
In a fully performance-based approach to aviation systems safety regulation, it’s not enough to establish that design safety objectives have been met at one moment in time.
Today, there is no systematic process to ensure that the assumptions made in safety assessments are valid with respect to the whole operational life of an aircraft type. In the event of accidents and incidents, the original system safety assessments used for certification may be revisited. There’s a case for a full-time process to make more use of operational experience.
We need to view safety assessments as living documents. The assumptions made at certification can be validated with operational experience. New generations of aircraft can provide streams of operational data at low cost. Algorithms may then perform analysis at speed and the use of digital twins may provide a means of detecting problems before they occur.
In this short article we’ve touched upon the systematic methodology of system safety assessment. It’s not just about numbers. It’s not just on the “to do” list to satisfy the certification authorities. It’s a compendium of qualitative processes supported by quantitative methods that has proven to be successful.
These methods will continue to develop apace as technology challenges are unrelenting. A digital transformation is underway and new methodologies will be needed to address artificial intelligence, expert systems, and cloud connected avionics. Will this be an evolution or revolution? Whichever, be sure it’s better that change is built on solid foundations.
