INTRODUCTION

Despite a lot of papers published about how to assess occurrence safety risk, many times during the implementation of our Galiot SMS - Safety Management System we have encountered inadequate risk assessment method selection, lack of understanding of the difference between safety issues, and historical events assessment and what are the goals of event risk assessment.

Therefore, I believe that this brief review of main risk evaluation methods might be useful for safety practitioners involved in the investigation and classification of safety occurrences in aviation

MAIN OUTPUTS OF EVENT RISK ASSESSMENT

Outputs of occurrence risk assessment should give answers to these two questions:

1. What should be done about the event (qualitative output value)
Typical answers to this question are: a) Investigate immediately and take actions, b) Investigate and carry out a further risk assessment, c) Use data for improvements

2. What is the magnitude of event risk (quantitative output value)
In the transition from traditional compliance-based prescriptive schemes towards a performance-based approach, measuring events' safety risk values is recognized as one of the top priorities in the context of a Safety Management System.
Quantitative risk values are the relative risk values between events, calculated differently in the methods discussed below.

SIDE-BY-SIDE COMPARISON

Accident/Serious Incident/Incident Classification

This is the oldest method used from the beginning of commercial aviation when safety performance has been calculated by a simple counting number (or rate) of distinct occurrence categories.
This method has three shortcomings: a) qualitative output limited to max 3 categories, b) no quantitative risk value, c) focused only on actual consequences and neglecting potential but not reached outcomes.
Nowadays, when accidents are very rare events (less than 3 per 10M flights), from a service provider perspective, this method is even more limited to 2 output values, resulting in poor and insignificant performance measurement, even for several years period calculation.

ICAO Risk Matrix

ICAO Risk Matrix is a safety issue risk assessment method based on “projected likelihood and severity of the consequences or outcomes from an existing hazard or situation”.
The process of the risk assessments with this method could be divided into 3 steps:
1. Assessing probability (by selecting one of the predefined values such as Frequent, Occasional, Remote, Improbable, and Extremely Improbable) where probability is defined as “the likelihood or frequency that a safety consequence or outcome might occur”.
2. Assessing severity (by selecting one of the predefined values such as Catastrophic, Hazardous, Major, Minor, and Negligible) where severity is defined as “the extent of harm that might reasonably occur as a consequence or outcome of the identified hazard”.
3. Selecting safety risk index (from 1E to 5A) and related risk tolerability (Red/Yellow/Green) from the risk matrix.

While this method has been successfully used for safety issue risk assessments, it has some general limitations and some specific drawbacks for individual event risk assessments:
a) Poor resolution
Qualitative output limited to 3 categories (Inacceptable, Tolerable, Acceptable) is unable to distinguish the relative importance of risk in the same category.
b) No quantitative ranking of the risk
c) Inputs subjective to cognitive and centering bias
d) Severity of what? Probability of what?
There are no predefined methods for event probability and severity evaluation
e) Arbitrary adapted matrix
Changing the number of probabilities/severities options and/or risk coloring schema without an understanding of Cox’s Matrix Theorem can lead to “worse-than-random” risk level selection.
f) Probability evaluation
The probability part is based on a concept of overall probability (product of average event probability and average conditional probability) of a certain consequence for a certain hazard for ALL similar events and NOT on the probability of the consequence at the moment when a SPECIFIC event occurred.

Event Risk Classification (ERC)

The key objectives of the Event Risk Classification (ERC) method developed by the ARMS working group are to overcome limitations of the ICAO Matrix and to reduce analyst subjectivity about the probability of the consequence.

In contrast to the ICAO Matrix where risk evaluation is based on all similar events of the same type (events resulting from the same hazard), the ERC method is based on the concept of “event-based risk level” where event risk is considered as the risk which was there when the event happened. The event risk classification is performed by use of ERC matrix (ARMS propose a 4x4 matrix but it could be customized to meet specific requirements) where a safety expert is required to answer two questions:

1. If this event had escalated into an accident, what would have been the most credible outcome?
2. What was the effectiveness of the remaining barriers between this event and the most credible outcome?

While the first question related to severity assessment is similar to the ICAO Risk Matrix approach, the second one, related to the probability of the outcome, is based on the evaluation of the remaining barriers preventing event escalation to the accident. This approach follows the “Swiss-Cheese” accident causation model developed by James Reason.

The following answers to the second question are proposed by the ARMS group:

Not effective: The accident occurred, or could only be prevented by either pure luck or exceptional skills
Minimal: Some barriers were still in place, but their total effectiveness was minimal
Limited: The effectiveness of the remaining safety barriers was limited. This is usually an abnormal situation, which is more demanding to manage, but with still a considerable remaining safety margin
Effective: The safety margin was effective, typically consisting of several good safety barriers

ERC model has two outputs:

Risk Tolerability: qualitative color values (Red/Yellow/Green) indicating what should be done about the event.
Risk Index: quantitative risk value in the range from 1 to 2,500 suitable for safety performance measurement.

Several authors proposed enhanced matrix versions with better resolutions (like Jochen Mickel), but these models will not be discussed here because they are following the same principles explained above.

EASA European Risk Classification Schema (ERCS)

The European Risk Classification Schema (ERCS) follows core principles of the Event Risk Classification (ERC) method:
a) event-based risk level assessment
b) probability assessment based on the effectiveness of the stoping and remaining barriers
c) qualitative and quantitative safety risk score of an occurrence and not its actual outcome

Compared to Event Risk Classification ERC, this model introduces identification of the key risk areas (including a comparison of their risk levels) and harmonized approach for event severity and probability determination.

The ERCS shall consist of the following steps:

1. Determination of the most likely type of accident that occurrence could have escalated to
The so-called key risk areas are: a) airborne collision, b) aircraft upset, c) collision on the runway, d) excursion, e) fire smoke and pressurization, f) ground damage, g) obstacle collision in flight, h) terrain collision, i) other injuries and j) security.
2. Determination of the potential loss of life category based on aircraft size and key risk area
Proposed categories are X=more than 100 possible fatalities or equivalent size aircraft for cargo, S = between 20 to 100 possible fatalities or equivalent size aircraft for cargo, M = between 2 to 19 possible fatalities or equivalent size aircraft for cargo, I = 1 possible fatality, E = serious and minor injuries as there are no fatalities and A =no likelihood of an accident.
3. Selection of severity score (potential loss of life) from severity matrix based on the input from step 1
Proposed severity scores are A - for no likelihood of an accident, E - for an accident involving minor and serious injuries, I - for an accident involving a single fatality, M - for a major accident with a limited (2-19) number of fatalities, S - for a significant accident with potential for 100 fatalities, X - for a catastrophic accident with the potential for more than 100 fatalities.
4. Identifying the stopping barrier from the ERCS barrier model
Stopping barrier is barrier prevented event to escalate into an accident (if exists).

5. Identification the effectiveness of the remaining barriers
The remaining barriers are barriers that were placed between stopping barrier and the potential outcome. Barriers placed before stopping barrier should not be considered in the calculation because they do not prevent accident causation.
The effectiveness of each barrier shall be classified as:
a) Stopped, b) Remaining Known, c) Remaining Assumed, d) Failed Known, e) Failed Assumed and f) Not Applicable.
6. Calculation of barrier weight sum and corresponding barrier score
Barrier weight sum and corresponding barrier score are calculated by summing barrier weights for all barriers classified as Stopped, Remaining Know and Remaining Assumed.
Barriers classified as Failed or Not Applicable shall not be counted for the final score).
7. Selecting safety score and corresponding numerical risk value from ERCS matrix
The numerical value of risk magnitude is mandatorily required to facilitate the aggregation and numerical analysis of multiple occurrences.

GALIOT SMS implementation of the European Risk Classification Schema (ERCS)

CONCLUSION

Use Accident/Serious Incident/Incident Event Classification only for historical reasons because of the low, qualitative only, risk resolution (practically only two risk levels)

Do not use ICAO Risk Matrix for single event risk classification because:
a) it has been designed for safety issue risk assessment only
b) qualitative only risk level output
c) inappropriate probability assessment based on a concept of overall probability (product of average event probability and average conditional probability) which is not applicable for historical events

If for any reason you are forced to use ICAO Risk Matrix do not arbitrarily change coloring schema and/or matrix format without the understanding of Cox’s Matrix Theorem because it can lead to “worse than useless” risk results.

Use Event Risk Classification (ERC) because it overcomes ICAO Risk Matrix limitation by providing two risk attributes (qualitative risk level and quantitative risk magnitude) and an appropriate probability assessment approach for a single, historical event.

Use EASA European Risk Classification Schema (ERCS) because in addition to all advantages of Event Risk Classification (ERC) it provides identification of the key risk areas and harmonized methodology, which is less bias-prone for event severity and probability assessment.

Marino Tudor

Founder & CEO
Galiot Aero Ltd
February, 2021