approaches to large, unstructured, complex problems ... · approaches to large, unstructured,...

Approaches to Large, Unstructured, Complex Problems

Problem Solving ApproachGeoff Vining

Professor of Statistics

Virginia TechTÍTULO

Outline

• What Is Statistical Engineering?

• Scientific Method

• The Strategy for Solving These Problems

• The Toolsets

• NASA Case Study

What is Statistical Engineering?

• The Art and Science for Solving Large, Unstructured, Complex Problems.

• Problems that Keep People Awake at Night

• Often, High Profile/ High Impact Problems

• Key: Proper Understanding of Scientific Method

What Is Statistical Engineering?

• Currently, No Text Discusses Large, Unstructured, Complex Problems!

• Emerging New Discipline: Statistical Engineering

Solutions Require:

• Interdisciplinary Teams

• Proper Blend of Talents

– Technical – Engineering/Scientific

– Statistical/Analytical

– Organizational Psychology/Anthropology

– Project Management

Scientific Method

• The heart of Statistical Engineering is the scientific method.

• Most theories underlying statistical engineering involve strategic application of the scientific method.

– Deming-Shewhart PDCA (Plan, Do, Check, Act)

– DMAIC (Define, Measure, Analyze, Improve, Control)

Scientific Method

• The Scientific Method Is

– Fundamental Approach for Discovery and Problem Solving

– Inductive/deductive problem solving process

Scientific Method: Basic Process

1. Define the problem (deductive)

2. Propose an educated theory, idea or model (deductive)

3. Collect data to test the theory (inductive)

4. Analyze the results (inductive)

5. Interpret the data and draw conclusions (inductive)

Scientific Method: Basic Process

• This process continues until a reasonable solution emerges.

• The scientific method is a sequential learning strategy!

Summary of the Scientific Method

• Understand the Real Problem at Hand

• Define the Problem

Summary of the Scientific Method

• Discover Solutions

– Abstract from the concrete to the abstract

– Develop a theory

– Test the theory using data

– Modify the theory as necessary

• Strong Need for Interdisciplinary Collaboration

Scientific Method and Data

• Data are the keys to the successful application of the scientific method

– Data collection

– Data analysis

– Data Interpretation

Strategy of Statistical Engineering

• Identify Problem

• Provide Structure

• Understand Context

Strategy of Statistical Engineering

• Develop the Solution Strategy

• Develop and Execute Tactics

• Deploy Final Solution

Tactics of Statistical Engineering

• Data Acquisition

• Data Exploration

• Analysis

– Traditional Statistical Methodologies

– Modern Analytics (Big Data)

Tactics of Statistical Engineering

• Inference to the Process/Problem

• Deployment of Tentative Solution

– Does It Work?

– Is It Sustainable?

Overarching Methodologies

• Data Visualization

• Project Management

• Teamwork

• Organizational Culture

Keys to Success

• Strong Management Support

• Interdisciplinary Teams

• Systematic Approach to the Process

• Understanding the Right Tool for the Right Job

• Communicate Results to all Organizational Levels

• The Science Is How to Put Everything Together!

More on Tactics

• The Basic Tactical Tools Are Taught in Statistics/Industrial Engineering

• Data Acquisition:

– Working with Historical Data

– Planning Data Collection

– Designing Experiments

– Data Cleansing

More on Tactics

• Data Exploration:

– Exploratory Data Analysis

– “Magnificent Seven”

– Analytics!

– Preliminary Models

– Follow Up

More on Tactics

• Analysis

– Statistical Inference: Estimation and Testing

– Formal Model Building

– Predictive Analytics

– Critical Point:

• Right Tool

• Properly Applied

More on Tactics

• Inference Back to the Problem – What Did We Discover?

– Statistical Inference

– Engineering Inference!

• Propose, Evaluate, Deploy Solutions

• Statistical Engineering: The Science for Deploying the Tools!

Analogy with Chemical Engineering

• Origins of Chemical Industry

– Germany, 1800s

– Chemists working with Mechanical Engineers

– Batch Processes for “Specialty Chemicals”

Analogy with Chemical Engineering

• Origins of Chemical Engineering

– United States, late 1800s, early 1900s

– Large Continuous Processes

– Commodity Products

– Process Efficiency Critical for Profit

Analogy with Chemical Engineering

• Critical Concept: “Unit Operations”

– Mass transfer

– Heat Transfer

– Separation Processes

– Chemical Reaction Processes

• Chemical Plant: System of Unit Operations

Analogy with Chemical Engineering

• Overarching Methodologies:

– Process Economics

– Process Control Systems

– Problem Solving/Data Analysis

– Managing Processes/People

• Key: Systems Thinking

Analogy with Chemical Engineering

• Statistical Engineering Unit Operations

– Problem Definition

– Data Acquisition

– Data Analysis

– Inference

– Solutions

Analogy with Chemical Engineering

• Statistical Engineering Overarching Methods– Data Visualization

– Project Management

– Teamwork

– Organizational Culture

• Systems Thinking

• Statistical Thinking!

NASA Example - COPVs

• Relatively Small Statistical Engineering Project

• COPV – Carbon Overwrapped Pressure Vessel

• Overarching Question of Interest: Reliability of COPVs at Use Conditions for Expected Life of Mission

NASA Example - COPVs

• Issues:

– Many different types of COPVs used in spacecraft

– Vessel tests are very expensive: money and time

• NASA Engineering Safety Center (NESC) Project

Core NESC Analytics Team:

• Reliability Engineers:

– JPL

– Langley Research Center

– Glenn Research Center

• Statisticians:

– Marshall Space Flight Center

– Virginia Tech

NASA Example - COPVs

• NASA Team’s Approach: Focus on Strands Used to Wrap Vessels

– Less expensive

– Can have many more experimental units than for vessels

• Still Issue with Time to Test

• Problem: How Do Strands Predict Vessel Behavior?

Previous Strand and Vessel Tests

• Relevant strand study conducted at a national lab:

– 57 strands at high loads for 10 years

– Net information learned: Strands either fail very early or last more than 10 years

• Vessel studies:

– Also 10 years

– Weibull model parameters similar to strands

Team’s Initial Concept

• Much larger study

• Censor very early

– Reduces time

– Allows the larger study in a practical amount of time

• Proceed in phases

• Have detailed data records to track any problems

NASA Example - COPVs

• Phase A: Conducted During Shake-Out of Equipment

• Small study (bigger than the national lab study!)

• Statistical goal: Determine if the parameters from the national lab study are valid as the basis for planning the larger study!

• Note: Phase A gave the team an opportunity to re-plan the larger experiment, if necessary!

NASA Example - COPVs

• Phase B: “Gold Standard” Experiment

• Planned time required: 1 year

• Used 4 “blocks” of equal numbers of strands

– Allowed the team to correct for time effects

– Allowed the team to mitigate problems

• Size assured ability to assess model

NASA Example - COPVs

• Total Size of the Database: Huge

• Kept data from start of specific strand test to failure on the second

• Kept the last 2 minutes at the .01 second from buffer

• Buffer allowed team to investigate unusual phenomena at failure

• Essential for proper data cleansing

NASA Example - COPVs

• Parallel Vessel Study

• Reasonably large ISS study targeted to end early (< 10 yrs)

• Opportunity to step up loads to mimic strands

• Censored but longer censor time than strands

Results to Date

• Phase A: Surprisingly similar to national lab study

• Phase B:

– Serious problem occurred in the first block

– Serious conversations with possibility of replacing!

– Other three blocks produced better than the planned precision for the estimates

– Residual analysis confirmed the Weibull model

Why is COPVs Statistical Engineering?

• Application of Scientific Method to a Complex Problem

• Sequential Data Collection/Experimentation

• Each Phase Targeted Different Questions

• Clearly Documented Assumptions, Assessed via Data

• Took Proper Steps to Cleanse Data

• Real Research Question Involves System of Systems

The Future

• Success Requires Providing Value to Organizations

• Solving Large, Unstructured, Complex Problems Provides Great Value

• Current State: People Know Tools, Often Limited Number of Tools

The Future

• Future State: How Do We Use the Tools Most Effectively

– Most Understand the Full Range of Tools Required

– Most Understand Best Practices for Using the Tools

– Must Understand Leadership in the Broad Sense

• The Future Is a Journey!