losa as a collaborative safety tool captain vern jeremica senior safety pilot boeing commercial...

LOSA as a Collaborative Safety Tool

LOSA as a Collaborative Safety Tool

Captain Vern Jeremica Senior Safety Pilot

Boeing Commercial Airplanes

International LOSA & TEM Workshop Kuala Lumpur, Malaysia 13 Sept. 2005

Int’l. LOSA & TEM Workshop

Today’s TopicsToday’s Topics

Boeing’s Safety Approach

TLC-Boeing Collaboration

Brief Safety Data and Tools Discussion

Boeing’s use of LOSA data

The Future

Boeing’s Safety Approach

TLC-Boeing Collaboration

Brief Safety Data and Tools Discussion

Boeing’s use of LOSA data

The Future


Past Accident/Incident & Customer Input-DrivenPast Accident/Incident & Customer Input-Driven

Safety change was accident driven

Safety change was driven by customer complaint/input

Safety change was accident driven

Safety change was driven by customer complaint/input


Boeing’s Approach to Safety Boeing’s Approach to Safety

Be proactive in research and promoting safety

Learn from accidents as well as incidents

Learn from the experience of our customers

Continuously improve the Airplane Life Cycle process

Seek Global Involvement – Airlines– Other Manufacturers/Sub-contractors– Regulatory agencies– Safety and flight crew organizations

Be proactive in research and promoting safety

Learn from accidents as well as incidents

Learn from the experience of our customers

Continuously improve the Airplane Life Cycle process

Seek Global Involvement – Airlines– Other Manufacturers/Sub-contractors– Regulatory agencies– Safety and flight crew organizations


Boeing’s Safety Approach: Human Factors Research


Rely on data to drive safety related decisions

Conduct and support internal safety research

Share mutually beneficial data and results

Rely on data to drive safety related decisions

Conduct and support internal safety research

Share mutually beneficial data and results




Collaborate globally on external research– NASA, FAA, NLR

– Madrid BRTE

– Customer airlines

– Universities (Illinois, Ohio State, Cranfield, Massey, etc.)

– University of Texas HF Project/The LOSA Collaborative

Create safety tools to support SMS at airlines– Flight Operations Support Program (FOSP)

– Procedural Event Analysis Tool (PEAT)

– Maintenance & Ramp Error Decision Aids (MEDA, REDA)

– Cabin Procedure Investigation Tool (CPIT)

– Line Operations Safety Audits (LOSA)

Collaborate globally on external research– NASA, FAA, NLR

– Madrid BRTE

– Customer airlines

– Universities (Illinois, Ohio State, Cranfield, Massey, etc.)

– University of Texas HF Project/The LOSA Collaborative

Create safety tools to support SMS at airlines– Flight Operations Support Program (FOSP)

– Procedural Event Analysis Tool (PEAT)

– Maintenance & Ramp Error Decision Aids (MEDA, REDA)

– Cabin Procedure Investigation Tool (CPIT)

– Line Operations Safety Audits (LOSA)


Texas LOSA Collaborative (TLC)Boeing Collaboration AgreementTexas LOSA Collaborative (TLC)Boeing Collaboration Agreement

• Agreement signed May 2002

• Assures airline and crew member confidentiality

• Enables TLC to provide customers with a two-way channel of communication with Boeing

• Establishes a mutually valuable research and safety partnership

• Agreement signed May 2002

• Assures airline and crew member confidentiality

• Enables TLC to provide customers with a two-way channel of communication with Boeing

• Establishes a mutually valuable research and safety partnership


The Value of CollaborationThe Value of Collaboration

The LOSA Collaborative

– Gains manufacturer’s perspective and expertise in global aviation safety

– Scientific and expert technical support

– Detailed design knowledge

– Ability to compare results from other sources

– Increased opportunity to benefit all airlines

– Economic and manpower resources

• Improved LOSA data collection software

The LOSA Collaborative

– Gains manufacturer’s perspective and expertise in global aviation safety

– Scientific and expert technical support

– Detailed design knowledge

– Ability to compare results from other sources

– Increased opportunity to benefit all airlines

– Economic and manpower resources

• Improved LOSA data collection software


The Value of CollaborationThe Value of Collaboration

TLC .......................................... Value

Boeing–Synergistic research partnership–Invaluable, objective insights into how our

products are operated in “normal” situations–Access to information about crew errors

during normal operations (ARCHIE)–Insights about how crews recover from

errors –Objective crew performance data to guide

future design


USE OF TECHNOLOGICAL DATA!USE OF TECHNOLOGICAL DATA!


T H E D A T A W A V E T H E D A T A W A V E


Improving the Human InterfaceImproving the Human Interface

Accidents

Incidents (ASAP)

FDM/FOQA

LOSA

BOEING FOSP

Accidents

Incidents (ASAP)

FDM/FOQA

LOSA

BOEING FOSP

Some Potential Data Sources:

Experienced-Based Airplane Lifetime Safety Cycle

Continuous feedback of information

Boeing design and regulatory

requirements

Validateand

certify

Delivery

ProduceDesign

ProblemResolution

Reports

•Previous DR&O

•FAR’s, •JAR’s•Customer requirements

NewTechnology

•SR&O’s•Design reviews

•Analyses•Testing

Analyses

Flight Testing

LaboratoryTesting

DATA• 300,000

telexes per year

• 85,000 accidents, incidents to date

Lessons Learned

ModificationReviewBoard

Testrequirements

In-ServiceOperation

Servicebulletins, etc.

BCA In-Service Safety Process

SRB’s*

Data Corrective Action Management

PotentialIssue

Analysis

AccidentInvestigation

PASC

Functional Safety

Councils

Issue Classification

7-23-02 SPS-032

Safety Review Board Members

CSD

SRP

AccidentData

In-Service Events

Lead Airline Process Involvement

Flight Test

Factory

COSP filterFAA EIB = Engineering Investigation Board

SRB = Safety Review Board*

IncidentData

COSP Bulletin Board EIBs*

Potential Supplier

Safety Issues

Issue Database

BCA In-Service Safety Process

LOSA


Normal Operations

AbnormalOperations

PEAT

LOSA

ASAP BASISASRS

SURVEYSBoeing FOSP

Lessons LearnedOperations

DesignTraining

Global Safety Outreach

Mitigated ThreatsTrapped Errors

Unmitigated ThreatsConsequential ErrorsIncidents, Accidents

THREATS & DEFENSES

Correlation and Convergence

FOQA/FDM


Boeing’s Use of LOSA DataBoeing’s Use of LOSA Data

Search for threats and errors related to safety of our fleet (e.g., automation related errors)

Close the “loop” on design

Develop capability to correlate results with other data and gain convergent validity

Capitalize on observations in ARCHIE to discover trends and patterns

Seek to improve latest data mining tools to – Enable efficient analysis of narratives

– Still limited to case-by-case basis

Search for threats and errors related to safety of our fleet (e.g., automation related errors)

Close the “loop” on design

Develop capability to correlate results with other data and gain convergent validity

Capitalize on observations in ARCHIE to discover trends and patterns

Seek to improve latest data mining tools to – Enable efficient analysis of narratives

– Still limited to case-by-case basis


LOSA DATA CAN ALSO:LOSA DATA CAN ALSO: Can proactively discover new safety issues/

through manufacturer/airline inquiry

Data which will assist in prioritizing risks and identify the real safety issues

Can provide necessary data to help define “Best Practices!”

Validate or invalidate other safety initiatives

Comfortable with de-identified data, but interested in accessing data on different Boeing types and models

Can proactively discover new safety issues/ through manufacturer/airline inquiry

Data which will assist in prioritizing risks and identify the real safety issues

Can provide necessary data to help define “Best Practices!”

Validate or invalidate other safety initiatives

Comfortable with de-identified data, but interested in accessing data on different Boeing types and models


LOSA DataLOSA Data Structured format that accurately captures

– Facts and events in normal operations

– Background that puts facts & events in context

Provides more than single flight context

Can provide better detail than incident reports

Greater accuracy than anecdotal analysis

Can support rate of occurrence analysis

Reveals how crews successfully manage errors

Enables focused observations on issues of interest

Structured format that accurately captures – Facts and events in normal operations

– Background that puts facts & events in context

Provides more than single flight context

Can provide better detail than incident reports

Greater accuracy than anecdotal analysis

Can support rate of occurrence analysis

Reveals how crews successfully manage errors

Enables focused observations on issues of interest


LOSA Collaborative Visit August 2005LOSA Collaborative Visit August 2005

2 Day Visit, August 2005

Reviewed approximately 400 sets of data, primarily 747-400 and 777– Airfield and Dates De-identified

– Airline De-identified

2 Day Visit, August 2005

Reviewed approximately 400 sets of data, primarily 747-400 and 777– Airfield and Dates De-identified

– Airline De-identified


Mining and Evaluating the DATA!Mining and Evaluating the DATA!

Mining/Refining/Evaluating/Refining/Evaluating/

Refining/etc.

((((PRODUCT)))))((((PRODUCT)))))

Mining/Refining/Evaluating/Refining/Evaluating/

Refining/etc.

((((PRODUCT)))))((((PRODUCT)))))


WHERE ARE THE THREATS?WHERE ARE THE THREATS?

Threats in the pre-departure phase of flight are highest.

Where English is not the native language

Traffic compression is a growing threat

Pre-flight after major maintenance often do not adequately prepare the airplane for the aircrew

Threats in the pre-departure phase of flight are highest.

Where English is not the native language

Traffic compression is a growing threat

Pre-flight after major maintenance often do not adequately prepare the airplane for the aircrew


Phase 4 – Design SolutionPhase 4 – Design Solution

New checklist alerting will be integrated into new airplane designs to prevent:

- omission

- forgetting, and

- late execution of critical normal checklists

Plus other data-driven enhancements to 777 and 787 baseline checklist design

New checklist alerting will be integrated into new airplane designs to prevent:

- omission

- forgetting, and

- late execution of critical normal checklists

Plus other data-driven enhancements to 777 and 787 baseline checklist design

Other Design

Issues

Other Design

Issues


Design Design

ACARS Alerts & Printers distract crews in the low altitude environment

– Inhibit function?

Missed approaches are often mismanaged

– More automated: e.g. auto altitude reset function?

Failure to arm the glide slope during an ILS

– Aircrew alert message?

Reversion from heading select, HDG SEL


VNAV-High vulnerability to error during vertical flight!

ACARS Alerts & Printers distract crews in the low altitude environment

– Inhibit function?

Missed approaches are often mismanaged

– More automated: e.g. auto altitude reset function?

Failure to arm the glide slope during an ILS


Reversion from heading select, HDG SEL


VNAV-High vulnerability to error during vertical flight!


Design Issues & Feedback Design Issues & Feedback Crews set the altitude incorrectly by several hundred feet

– Sensitivity Issue? Mode Control Panel worn?

– Can it be adjusted? Do MCP’s differ? Design Issue?

During departure process when PAX and Cargo weights change, crew fail to update the FMC – Is there an engineering solution?

777 firm landing-Need for feedback

747-400 landed excessively long during autoland-Need for feedback

Crews set the altitude incorrectly by several hundred feet

– Sensitivity Issue? Mode Control Panel worn?

– Can it be adjusted? Do MCP’s differ? Design Issue?

During departure process when PAX and Cargo weights change, crew fail to update the FMC – Is there an engineering solution?

777 firm landing-Need for feedback

747-400 landed excessively long during autoland-Need for feedback

Operational

Issues

Operational

Issues


Altitude Deviations Are CommonAltitude Deviations Are Common

Many occur due to distractions and high workload situations– Deviation around weather

– Fuel in center tank operational requirements

– ATC

– Deviation around traffic

– Missed Approaches

– Traffic compression

Many occur due to distractions and high workload situations– Deviation around weather

– Fuel in center tank operational requirements

– ATC

– Deviation around traffic

– Missed Approaches

– Traffic compression


Altitude Deviation SolutionAltitude Deviation Solution

Possibly Design

More emphasis in the Flight Crew Training Manuals to bring about awareness

More emphasis during LOFT and CRM training to bring about awareness

More structured/defined cross check process

Possibly Design

More emphasis in the Flight Crew Training Manuals to bring about awareness

More emphasis during LOFT and CRM training to bring about awareness

More structured/defined cross check process


VNAV DeviationsVNAV Deviations

Busting the speed limit below 10,000 feet due to mismanaged vertical profile setting

High vulnerability to crew error during VNAV operation and reversion between other modes

Busting the speed limit below 10,000 feet due to mismanaged vertical profile setting

High vulnerability to crew error during VNAV operation and reversion between other modes


VNAV Deviation SolutionVNAV Deviation Solution

Increased emphasis on proper cross checking in training and flight manuals

Input to Boeing Flight Technical, Training, and Standards division for operations manual, and flight crew training manual revisions

Input to Boeing Flight Training and Alteon for incorporating emphasis in recurrent and transition training

Increased emphasis on proper cross checking in training and flight manuals

Input to Boeing Flight Technical, Training, and Standards division for operations manual, and flight crew training manual revisions

Input to Boeing Flight Training and Alteon for incorporating emphasis in recurrent and transition training


Flight Crew Training Manual Improvement

Flight Crew Training Manual Improvement

LOSA data indicates that cross checking catches a great deal of errors.

Need further input from industry and the LOSA Collaborative on an industry accepted “Best Cross Checking Practices”

LOSA data indicates that cross checking catches a great deal of errors.

Need further input from industry and the LOSA Collaborative on an industry accepted “Best Cross Checking Practices”


Each of the Boeing fleets: The crews' response to the automation error crossed with the outcome of the error3 possible responses: detected by the crew and action taken; detected by the crew but ignored (intentional noncompliance); undetected by the crew3 outcomes: inconsequential, additional error, undesired aircraft stateNote: the percentage of automation errors that went undetected by the crew ranged from 32% on the 777 to 50% on the 757.

FLEET Inconsequential Undesired a/c state Additional error Total

737 Classic Detected and Action Taken 111 4 2 117 41%95%

Detected and Ignored 42 6 1 49 17%Undetected 32 58 30 120 42%

27%185 68 33 286 100%

737 NG Detected and Action Taken 56 0 0 56 42% 100%


13%80 35 17 132 100%

747-400 Detected and Action Taken 82 5 3 90 55%91%


15%96 46 21 163 100%

757 Detected and Action Taken 33 1 0 34 49%97%


23%41 22 7 70 100%



7%81 31 10 122 100%



29%79 20 7 106 100%

Error outcome


Data ValidityData Validity

Aircraft Type 737 Classic

737NG 747-4 757 767 777 Total

# Flights in Archive

July 05

877 415 458 301 450 291 2792

Error Prevalence, % of Flights W/Automation Error

26% 24% 28% 18% 22% 29% 25%

Average # of Flights With Automation Error

0.33 0.32 0.36 0.23 0.27 0.36 0.30

Error Mismanagement

%Automation Errors

Mismanaged

35% 39% 41% 41% 34% 25% 36%

Average # Mismanaged Automation Errors/Flight

0.115 0.125 0.146 0.096 0.091 0.093 0.114


FUTURE STRATEGYFUTURE STRATEGY

Most errors in themselves are not significant

As data is analyzed, the amount/rate and type of errors & threats determine significance

The narrative has important details that make the data more meaningful

With regards to mitigation of risk and reduction of threats and errors:– Customers as well as manufacturers will have the

quickest and greatest impact through modifications of processes, standards, training, and procedures

– However, there will be cases for engineering solutions to past, present, and future design

Most errors in themselves are not significant

As data is analyzed, the amount/rate and type of errors & threats determine significance

The narrative has important details that make the data more meaningful

With regards to mitigation of risk and reduction of threats and errors:– Customers as well as manufacturers will have the

quickest and greatest impact through modifications of processes, standards, training, and procedures

– However, there will be cases for engineering solutions to past, present, and future design


FUTURE QUESTIONS TO THE LOSA COLLABORATIVE

FUTURE QUESTIONS TO THE LOSA COLLABORATIVE

•Are crews proactive or reactive with regards to Are crews proactive or reactive with regards to potential TA/RA’s? How is TCAS being used in an potential TA/RA’s? How is TCAS being used in an operational context?operational context?•Why do some crews commit more errors than other Why do some crews commit more errors than other crews?crews?•Why do crews commit errors?Why do crews commit errors?•Boeing would like information on “The Best Crew Boeing would like information on “The Best Crew Practices observed during LOSA.” Practices observed during LOSA.”

-What are the attributes of crews who make -What are the attributes of crews who make relatively “zero” errors?relatively “zero” errors?-What are the attributes of crews who do an -What are the attributes of crews who do an excellent job in managing or trapping excellent job in managing or trapping

errors?errors?


FUTURE QUESTIONS TO THE LOSA FUTURE QUESTIONS TO THE LOSA COLLABORATIVE-ContinuedCOLLABORATIVE-Continued

FUTURE QUESTIONS TO THE LOSA FUTURE QUESTIONS TO THE LOSA COLLABORATIVE-ContinuedCOLLABORATIVE-Continued

•Regards to Long Haul Operations, do crews returning from vacation or extended periods of time away commit more errors?

•Regards to LOSA report findings, where are the weak areas in industry?

-What training, procedural, regulatory, and manuals changes occurred?

•From the opinions gathered in the aircrew surveys, where will the next accidents occur?


FUTURE QUESTIONS TO THE AIRLINE INDUSTRY

FUTURE QUESTIONS TO THE AIRLINE INDUSTRY

Regards to the airline crews that commit the most errors, what can the analysis tell us about why?

What training, standards, and operational improvements were implemented to reduce identified errors and to mitigate threats?

Is there any correlation between threat rates versus performance errors? – In areas where there are a lot of threats are sharper crews

required?– Stress/ threats improve performance to a point? Where is

that point where threats begin to decrease performance? Can it be identified?

Regards to the airline crews that commit the most errors, what can the analysis tell us about why?

What training, standards, and operational improvements were implemented to reduce identified errors and to mitigate threats?

Is there any correlation between threat rates versus performance errors? – In areas where there are a lot of threats are sharper crews

required?– Stress/ threats improve performance to a point? Where is

that point where threats begin to decrease performance? Can it be identified?


FUTURE QUESTIONS TO THE AIRLINE INDUSTRY-Continued

FUTURE QUESTIONS TO THE AIRLINE INDUSTRY-Continued

When Boeing equipment doesn’t work per design or as intended by the airline, Boeing would like to know about it.

Up to the discretion of the operator as to whether or not they chose to provide critical information to Boeing

There is a Boeing process in place, but sometimes operators fail or neglect to communicate

Boeing will provide process information to the LOSA Collaborative for distribution to the operator if an issue arises

When Boeing equipment doesn’t work per design or as intended by the airline, Boeing would like to know about it.

Up to the discretion of the operator as to whether or not they chose to provide critical information to Boeing

There is a Boeing process in place, but sometimes operators fail or neglect to communicate

Boeing will provide process information to the LOSA Collaborative for distribution to the operator if an issue arises


ConclusionsConclusions

LOSA is a unique safety tool that provides a window into human performance in context of normal ops

By cooperating, we can greatly multiply LOSA’s potential for improving global aviation safety– Consistent implementation is the key– Will learn together what we don’t know– Will help us to anticipate future system risks– More questions are being raised than answered

Success is critically dependent on the protection of safety data

LOSA is a unique safety tool that provides a window into human performance in context of normal ops

By cooperating, we can greatly multiply LOSA’s potential for improving global aviation safety– Consistent implementation is the key– Will learn together what we don’t know– Will help us to anticipate future system risks– More questions are being raised than answered

Success is critically dependent on the protection of safety data


Conclusion-ContinuedConclusion-Continued

Acquisition soft ware improvement will improve the process.

The larger the data base the better and more meaningful product

Level of expertise/experience level who is mining, evaluating, and refining the data is important.

Boeing is committed to strongly supporting LOSA as a key safety tool.

Boeing is in in the process of using data from TLC to provide input into 787 development, current production airplane design, as well as operational procedures & flight crew training!

Acquisition soft ware improvement will improve the process.

The larger the data base the better and more meaningful product

Level of expertise/experience level who is mining, evaluating, and refining the data is important.

Boeing is committed to strongly supporting LOSA as a key safety tool.

Boeing is in in the process of using data from TLC to provide input into 787 development, current production airplane design, as well as operational procedures & flight crew training!


Thank You For YourThank You For YourAttention!Attention!

losa as a collaborative safety tool captain vern jeremica senior safety pilot boeing commercial...

Documents