development of an industry standardized auditing and surveillance tool: minimizing maintenance...

Development of an Industry Standardized Auditing and Surveillance Tool: Minimizing

Maintenance Errors

Web-based Surveillance & Auditing Tool

Human Computer Systems Laboratory

Industrial Engineering

Clemson University, Clemson, SC 29634

www.ces.clemson.edu/ie/research/hcsl/websat

Clemson WebSAT Team:Dr. Anand K. Gramopadhye (Professor and IE Department Chair) Nikhil Iyengar (Manager, HCSL)

Dr. Joel S. Greenstein Kunal KapoorPallavi Dharwada

Sponsor: FAA

Industry Partner: FedEx


Research Objective

Identify a list of impact variables (process measures) affecting aviation safety across aircraft maintenance organizations.

Develop data collection/reduction and analysis protocol toanalyze errors for impact variables identified.

Develop and implement an application performingsurveillance/monitoring maintaining a consistent levelof oversight.

Slide 1


Deliverables

Slide 2

Year IIdentify list of impact variables to be used. Identify limitations and assess use of specific data sources in theauditing and surveillance tool.

Year II

Establish final design specifications.

Report on web based tool development process.

Analyze data collected using the web based tool.

Year III

Deliver prototype tool incorporating the trend analysis module.

Provide guidelines and recommended practices to use auditing/

Surveillance tool for overcoming maintenance oversight.

Deliver final auditing/surveillance tool source code and documentation.


WebSAT Mission Statement

Slide 3

Web-based Surveillance & Auditing ToolProduct description

Application incorporating recommended categorization, data collection, reduction and analysis schemes to facilitating preliminary data examination for surveillance and auditing of maintenance operations.

Key business goals

Achieve standardized data collection, reduction and analysis of maintenance errors across the airline industry.

Develop proactive system to capture maintenance errors.

Accomplish trend analyses in future versions of WebSAT.

Market Air Transportation Industry

Assumptions Develop WebSAT adhering to FAA standard research software design specifications (e.g. SQL server, ASP.NET, PHP)

Stakeholders FAA

Quality Assurance Department of Air Carriers (incl. QA representatives and auditors).

Carrier Information Technology Department


Phase I: Identifying process measures• Define process measures to incorporate response and observation-based data collected

during surveillance, audits, and airworthiness directives control processes.

• Discover, Identify and Validate using data gathering methods adopted interviews, focus groups, observation sessions, studying documentation and questionnaires.

• WebSAT modules identified are surveillance, technical audits, internal audits and airworthiness directives.

– Note: Airworthiness directives module will include display of the airworthiness data. It will not include a risk model.

Slide 5

Year I (A)

Identification of Impact Variables and

Data Sources(A1)

Validation(A2)

Deliverables to FAA(A3)

Identified List of Process Measures

WebSAT

Surveillance Auditing AD Control

Technical Audits Internal Audits

Administration

Records

Manuals

Procedures

Safety

Training

Compliance/Documentation

Inspection

Facility Control

Training & Personnel

Procedures

Data Control

Safety

Loading & Tracking

Information Verification

Documentation Surveillance

Facility Surveillance

Procedures and Manual Violation

In Process

Verification

Final Aircraft Walk around


Step I with FedEx

Step II with additional partnering airlines (e.g. Representatives of NWA, ATA, AA)

Slide 6

Year I (A)


Data Sources(A1)

Validation(A2)


Online-survey for process measure validation



Slide 7

Year I (A)


Data Sources(A1)

Validation(A2)

Year I deliverables to FAA:Identified list of impact variables to be used.

Identified limitations and assessed use of specific data sources in the auditing and surveillance tool.

Status Update

Year II & III

Work Accomplish

ed (B)

Work in Progress

(C)

Prototyping

(B1)

Concept Selection

(B2)

Software Developme

nt (B3)

Technical Audit

Module Validate with Controlled

Studies

Interface

Design to

Match the

User Persona

s(C1)

Identify Data

Reduction Techniqu

es

(C2)

Risk Model

for Data

Analysis

(C3)


Work accomplished:Revised product map using the feedback provided by QA audit Managers at

FedEx.The team developed three concepts for WebSAT:– Concept 1: Based on Google Search Engine with multiple search criteria– Concept 2: Based on Microsoft Outlook– Concept 3: Based on Tab metaphor Generated Prototypes.Selected Final Concept.Designed and Developed the Interface for technical audit module.

Next Slide shows Product Map

Prototyping

(B1)

Concept Selection

(B2)

Software Developme

nt (B3)

Technical Audit

Module

Work Accomplishe

d(B)

Slide 8


Concept Selection

(B2)

Software Developme

nt (B3)

Technical Audit

Module

Work Accomplish

ed (B)

Prototyping (B1)

Slide 9


Selection of final concept

Performed concept selection with HCI Experts, MIS Expert & Domain Experts

Users preferred organization structure of the tab metaphor.

Also preferred grid feature of the MS Outlook concept.

Software Developme

nt (B3)

Technical Audit

Module

Work Accomplish

ed (B)

Slide 10

.Concept Selection

(B2)

Prototyping

(B1)

Final Concept


Designed a database model.

Developed technical audits module using ASP.NET

with SQL Server.

Developed scenarios for a technical auditor.e.g., Thomas entered the audit data partially a week before and would

now want to enter the remaining data. He logs into the system, finds the appropriate audit into which the data needs to be entered. Enters the data and saves the changes.

Thomas also wants to search for all the audits performed in the past on the same vendor.

Concept Selection

(B2)

Work Accomplish

ed (B)

Prototyping

(B1)

Slide 11

.Technical

Audit Module

Software Developme

nt (B3)


Status Update

Year II & III

Work Accomplish

ed (B)

Work in Progress

(C)

Prototyping

(B1)

Concept Selection (B2)

Software Development (B3)

Technical Audit

Module Validate with

Controlled Studies

Interface

Design to

Match the

User Persona

s(C1)

Identify Data

Reduction Techniqu

es

(C2)

Risk Model

for Data Analysi

s

(C3)

Identified 3 research areas for enhancing product

Design interfaces to match the user personas.

Reduction of maintenance data to predict vendor performance and aircraft safety index.

Development of a risk model to analyze maintenance data.

Work in Progress – Research Inputs to Enhance WebSAT

(C)

Risk Model

for Data

Analysis

(C3)

Slide 12

.

Interface Design - User

Personas

(C1)

Identify Data

Reduction Techniqu

es(C2)


WebSAT report generation will involve:Identification of various fields for the report.

Indicate the required format of the report.

Generate report.

WebSAT risk model generation involves:Collection of historic data.

Identification of various variables/factors which may contribute to variance

in the responses.

Each work function will have a set of variables.

Hierarchical arrangement of the variables in an order such that the lowest level will represent the dependent variable. e.g., audit responses.

Generate data files using the established order of hierarchy.

Carry out hierarchical logistical analyses to determine the variables with significant effect.

Use significant variables to generate models for the auditing and surveillance work functions.

Slide 13


Personas

(C1)

Risk Model

for Data

Analysis

(C3)


(C)

Identify Data

Reduction Technique

s(C2)


WebSAT risk model application

Will use various risk models to answer specific questions asked by the

representative.

Some of these typical questions for the Technical Audit work function are:

a) How has the vendor performed over a period of time for an audit type?

Where β1, β2, β3 represent the regression coefficients of the various significant factors X1, X2, X3.

b) How has the vendor performed on the current audit or substantial maintenance event?

c) How has the vendor performed on the current audit or substantial maintenance for a particular process measure?

d) How has a particular audit type or type of aircraft performed?

Slide 14


Personas

(C1)

Risk Model

for Data

Analysis

(C3)


(C)

Identify Data

Reduction Technique

s(C2)

1 1 2 2 3 3 0vendorY X X X


Data Model Example

Rep.Variable

Name Description Min Max Range Name

X1 Audit type

This variable describes the number of different audit types

1 4 1 Fuel

2 Line

3 Ramp

4 Supplier/vendor

X2 DurationThis variable represents the time taken to complete an audit

Continuous var.

X3 ContinentThis variable describes the geographic location of the audit

1 3 1 America

2 Asia

3 Europe

X4Process measure

This variable refers to the categorical collection of the responses

1 7 1Compliance /

Documentation

2 Inspection

3 Facility Control

4Training and Personnel

5 Procedures

6 Data Control

7 Safety

Slide 15


Data Model Example contd.

Y = - 3.9665 (X11) - 0.9310 (X12) - 2.2.568 (X13) + 0(X14) - 0.01486 (X2) - 2.1142 (X41) - 0.9973(X42)

- 1.6173(X43) - 0.6776(X44) - 1.2207 (X45) -1.7407 (X46) + 0 - 0.00868 (X2*X31) + 0.04759(X2*X32)

+ 0(X2*X33) + 0.01608(X2 * X41) + 0.008069(X2 * X42) + 0.01098(X2 * X43) + 0.003374(X2 * X44)

+ 0.01030(X2 * X45) + 0.004742(X2 * X46) + 0(X2 * X47) + 3.4952(X31*X11) - 0.2180(X31*X12)

+ 1.0189 (X31*X13) + 0(X31*X14) + 0 (X32*X11) + 0(X32*X12) + 0(X32*X13) + 0(X32*X14)

+ 0(X33*X14) + 1.2708(X31*X41) + 0.8188(X31*X42) + 1.4562(X31*X43) - 0.1469 (X31*X44)

+ 1.2363(X31*X45) + 0.8479(X31*X46) + 0(X31*X47) + 0.6220(X32*X41) + 1.1018(X32*X42)

- 0.6775(X32*X43) - 1.0791 (X32*X44) - 0.06895(X32*X45) + 0.3316 (X32*X46) + 0(X32*X47)

+ 0(X33*X41) + 0(X33*X42) + 0(X33*X43) + 0(X33*X44) + 0(X33*X45) + 0(X33*X46) + 0(X33*X47)

+ 0.01822(X2*X31*X41) - 0.00209(X2*X31*X42) - 0.00213(X2*X31*X43) + 0.008828 (X2*X31*X44)

- 0.00121(X2*X31*X45) + 0.004484 (X2*X31*X46) + 0 (X2*X31*X47) - 0.09046(X2*X32*X41)

- 0.09971 (X2*X32*X42) - 0.09662 (X2*X32*X43) -0.02079(X2*X32*X44) - 0.07381(X2*X32*X45)

- 0.09860 (X2*X32*X46) + 5.0835

Model based on preliminary analysis

Slide 16



The GLIMMIX Procedure - Type III Tests of Fixed Effects

Effect DF DF F Value Pr > F

Audit Type 3 15844 10.12 <.0001

Duration mean 1 15844 25.26 <.0001

Continent 2 15844 1.4 0.2455

Process measure 6 15844 9.61 <0.0001

Duration mean * Continent 2 15844 3.63 0.0265

Duration mean * Process measure 6 15844 3.32 0.0029

Continent * Audit Type 3 15844 6.5 0.0002

Continent * Process measure 12 15844 3.25 0.0001

Duration * Continent * Process measure 12 15844 3.15 0.0002

Slide 17



X1 X2 X3 X4

Y (Logit)

Odds ProbabilityAudit type Duration Continent PM

X11 X12 X13 X14 X2 X31 X32 X33 X41 X42 X43 X44 X45 X46 X47

1 0 0 0 5 1 0 0 1 1 1 0 0 0 0 3.1163 22.5626 0.9575

0 1 0 0 21 0 1 0 1 0 0 0 1 0 1 1.5324 4.6290 0.8223

Example 1: Predict the probability of receiving a ‘yes’ for a Fuel Audit in Los Angeles which would last for 5 days. This audit would cover process measures of compliance documentation, inspection and facility control.

Example 2: Predict the probability of receiving a ‘yes’ for a Line Audit in Lake Charles which would last for 21 days. This audit would cover process measures of compliance documentation, procedures and safety.

Slide 18


Significant Milestones Ahead

Slide 19


Conclusion

• WebSAT will provide overview of activities associated with commercial aircraft maintenance.

• Identification of maintenance related process measures and related safety indices.

• Identification of potential problems at geographically dispersed maintenance sites through data standardization.

• Promotion of accessible display of trends in data.

Slide 20


Dissemination

• Iyengar, N., Kapoor, K., Dharwada, P., Greenstein, J. S., and Gramopadhye, A. K., WebSAT: Development of Process Measures for Aircraft Safety, International Journal of Applied Aviation Studies, September 2005.

• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., Strategy to Identify Process Measures for Surveillance in Aviation, Proceedings of the Industrial Engineering Research Conference, Atlanta, May 2005.

• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., Strategy for Evaluation of Aircraft Maintenance Operations Using Process Measures, International Symposium of Aviation Psychology, Oklahoma City, April 2005.

• Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J. S., and Gramopadhye, A. K., A Strategy for the Development of a Web-based Tool to Reduce Aviation Maintenance Errors, Proceedings of Human Factors and Ergonomics Society, New Orleans, September 2004.

• Iyengar, N., Kapoor, K., Greenstein, J. S., and Gramopadhye, A. K., Selection of Data Gathering Methodologies for the Aviation Maintenance Industry, Proceedings of the Industrial Engineering Research Conference, Houston, May 2004.

• Dharwada, P., Iyengar, N., Kapoor, K., Greenstein, J. S., and Gramopadhye, A. K., Standardized Auditing and Surveillance of the Aircraft Maintenance Operations, Proceedings of the Industrial Engineering Research Conference, Houston, May 2004.

• Dharwada, P., Iyengar, N., Kapoor, K., Gramopadhye, A.K., and Greenstein, J.S., Web-Based Surveillance and Auditing Tool (WebSAT): A Proactive System to Capture Maintenance Errors, Proceedings of Safety Across High-Consequence Industries, St. Louis, Missouri, March 2004.

Slide 21


Acknowledgements

development of an industry standardized auditing and surveillance tool: minimizing maintenance...

Documents