simulation contract management v0.1€¦ · 5/24/11 2 objectives and scope objectives introduction...

5/24/11

1

Simulation Contract Management

31 May 2011 Peter Hill

Simcon Services Pty Ltd

31 May 2011 © Simcon Services Pty Ltd 1

Introductions Peter Hill

  Engineer / Project Manager   Industrial control, then simulation   Collins Ships Management System   F-111C Mission Simulator   Train Driving Simulators

You (30secs each!)   Name and organisation   What you do   Why you’re here


5/24/11

2

Objectives and scope Objectives

  Introduction to general Contract development and management principles

  Focus on specific simulation-related issues!

Outcomes   To be equipped with both general and simulation-

specific contract management concepts   To appreciate some of the difficulties arising from

simulation projects   To identify areas where further more detailed courses

would be beneficial


Objectives and scope Scope

  Mainly Acquisition – Support briefly mentioned   All domains – not just defence


5/24/11

3

Scope Aimed at

  Newly appointed or aspiring Project / Contract Manager of a simulation project

  Experienced Project / Contract Manager who has just taken on their first (or second!) simulation project.

  Business manager

  Involved with the tendering process (issuing or responding to)

  Those interfacing with industry, or customers


Program


5/24/11

4

General Contract Management Principles

  Success

  Enablers

  Tendering



  What is success?


Success

5/24/11

5


  On-time

  On-budget

  Acceptable Quality


Success Easy!

Pick any two!

General Contract Management Principles - Success

  Stakeholders:   Users   Funders   Owner / Champion

  Steering Committee

  Objectives of the system

  Business case   Sets expectations   Obtain funding   Defines timing


Pre-conditions

Identify! Communicate / Socialise

Allow contingency!

5/24/11

6

General Contract Management Principles - Success

  Training Needs Analysis – see later

  Constraints   Funding   Facilities   Throughput

  Staffing


Pre-conditions (ctd.)


  Managing Risks

  Communication

  Specifying responsibilities

  Managing relationships

  Managing resources

  Keeping records

  Behaving ethically


Success - Enablers

5/24/11

7



Reference

www.anao.gov.au/Publications/Better-Practice-Guides

Developing and Managing Contracts Getting the right outcome, paying the right price


Phases

  Request For Information

  Request For Tender / Proposal

  Evaluation

  Negotiation


Tendering

5/24/11

8

General Contract Management Principles - Tendering

  Who’s in the market

  Range of solutions

  Technical / other issues

  Possible risks

  Cost information

  Commercial-in-confidence


Request For Information

General Contract Management Principles - Tendering

  Cf Australian Defence Contracting (ASDEFCON) templates (www.defence.gov.au/dmo/DMO/function.cfm?function_id=45#group60)

  Conditions of Tendering   Evaluation criteria

  Schedules to be completed

  Draft contract (terms and conditions)   Standard T’s and C’s   Special T’s and C’s

  Statement of Requirements

  Statement of Work


Request For Tender

5/24/11

9

General Contract Management Principles – Tendering - RFT

  Pricing

  Payments (milestone based)

  Delivery schedule

  General information   Tenderer’s team

  Experience and References   System description

  Clause by clause response


Schedules

Ask: Why do I need this info?


  Defines the systems to be delivered

  Include the system’s objectives

  Number requirements!

  Requirements need to be testable!


Statement of Requirements

5/24/11

10


  Defines the management and VV&A processes to assure quality and reduce risk

  Understand the risks and put in place appropriate systems

  Include Contract Data Requirements List (CDRL) (including requirements traceability)


Statement of Work


  Sample CDRL


Contract Requirements Data List

5/24/11

11


  Sample CDRL Item


Contract Requirements Data item

General Contract Management Principles – Tendering

  Site inspection and briefing

  Probity – any issues / answers must be distributed to all

  Evaluation

  Negotiation

  => Contract in place!


Process

5/24/11

12

General Contract Management Principles – Tendering - Process

  Plan in place before RFT released!   Helps define the evaluation criteria   Helps define the Information requested in the tender

  Process:   Stakeholders   Separate technical and pricing => determining value for money.

  Through-life support costs

  Clarification questions - probity

  Risks   Technical   Management - reference checks   Cost

  Evaluation criteria weightings


Evaluation

General Contract Management Techniques

  Work Breakdown Structure

  Tools

  Principal / Contractor Liaison

  Verification, Validation and Accreditation

  Introduction into Service


5/24/11

13


Concept

  A tool used to define and group a project's discrete work elements in a way that helps organize and define the total work scope of the project

  End objective and successively subdividing it into manageable components in terms of   Size   Duration   Responsibility

  An element may be a product, data, a service, or any combination   Outcomes rather than actions!

  100% rule – sub-elements add up to 100% of the item


Work Breakdown Structure

systems, subsystems, components, tasks, subtasks, and work packages




5/24/11

14


Exercise

  Develop a WBS for a car




Helps define and align:

• Work Packages – criteria: • Complexity of the interfaces – loosely or close coupled • Knowledge / experience of the parties • Power to influence an outcome – eg getting data from an

OEM at platform procurement time • Management risk

• SoR and SoW structure

• Pricing

• Delivery schedule



5/24/11

15


  Schedule (eg Gantt)

  Budget and earned value

  Risk register

  Issues register

  Actions register – including customer’s!

  Correspondence register


Tools


  Kick-off meeting

  Regular reports and meetings

  Consider an on-site representative

  Encourage informal communication   Within limits!

  Clarify authorities

  Document decisions, directions


Principal / Contractor Liaison

5/24/11

16


Definitions (in layman terms)

  Verification: Have the requirements been achieved "Are we building the product right”

  Validation: Have the right requirements been developed to achieve the desired functionality

“Are we building the right product”

  Accreditation: Quantifies how well the device has achieved training functionality. (against a known standard)


Verification, Validation and Accreditation

General Contract Management Techniques – VV&A

Reviews   Requirements

  Verify that the design requirements match the Contract’s requirements

  Preliminary Design Review   Check the allocation of design requirements to

subsystems   Check the risk of the chosen architecture   Check proposed size, weight, heat, power etc

  Critical Design Review   Check that the design and documentation is complete

before implementation begins (or continues)


Verification

5/24/11

17


Reviews (ctd.)

  Factory Acceptance Test   Verify that the system conforms to the Contract’s

requirements

  Often performed at the Contractor’s premises for efficiency reasons

  Site Acceptance Test   Verify that the system conforms to the Contract’s

requirements, after transport / installation, and in the final environment


Verification


  Requirements Traceability Matrix   Shows linkages from design requirements (at each

phase of the design) back to the contract’s requirements

  Can check that each contract requirement has been addressed, and has been tested

  Documentation check   Check that each requirement is fully addressed

  Presentations   Useful for understanding the system architecture

and the development risks


Verification - Techniques

5/24/11

18


  Demonstrations   Look and feel of the software   High performance risk items

  Testing   “White-box” – eg for internal calculations

  “Black-box” – check outputs against inputs   Regression – re-testing following changes


Verification - Techniques


  Demonstrations / testing by stakeholders

  Beta release to users

  Issues:   Change in stakeholder expectations

  Change in operating procedures


Validation - Techniques

5/24/11

19


  Communicate to stakeholders

  Ensure support system is in place   Repair   Help-desk

  Train operators

  Dry runs


Introduction into Service

Simulation Specific Contract Management

  Access to customer Subject Matter Experts

  Data

  Fidelity

  How to Accept

  Standards

  Accreditation standards

  Configuration management


5/24/11

20


  Often “owned” by the client or operations section

  Difficult to extract from their normal responsibilities

  No “contractual” obligations for performance


Access to customer Subject Matter Experts



Data Data

5/24/11

21


  Why do you need data?

  Examples of data

  Availability

  Ownership / Intellectual Property

  Quality

  Cost

  Security classification

  Where Data can create problems


Data

Simulation Specific Contract Management - Data

  Simulators / simulations are trying to replicate reality

  Data defines this reality


Why do you need data?

5/24/11

22


  Dimensions of a cockpit

  Locations of controls / instruments

  Colours and labels

  “Feel” of controls

  Flight characteristics

  Sound characteristics

  Operations of sub-systems (eg flight computers)


Examples of data   Fault conditions and

consequences

  Air / sea / road / ground characteristics

  Weapon characteristics

  Visual environment

  Human characteristics – behaviour, morale


  Does the data exist?

  Particularly an issue with:   Old systems – little engineering data – may be some

measured data   New systems – may be engineering / predictive data

  Fault conditions

  Scope (eg outside the operational envelope – such as air-to-air refueling – but this is exactly what the simulator is for)


Availability

5/24/11

23


  Background and Foreground

  Data may be owned by the OEM (and not the client or tendering party)

  Data is often highly commercial-in-confidence (eg car industry)

  Deliverables (such as visual databases) may be a mixture of Background and Foreground

  An issue for support contracts – where a party other than the developer or IP owner intends to support the system


Ownership / IP


  What is the heritage / methodology used of the data?

  Is it for exactly the system being simulated?

  Is the anticipated usage within the scope of the data generation?


Quality

5/24/11

24



Quality - Scope

B737 / Wedgetail

B737


  Can be extremely high cost to:   Purchase   Obtain by measurement

  Needs to be taken into account in:   Work-packages / suppliers allocation (see later)

  Business case   Tender


Cost

5/24/11

25


  May prevent data being obtained

  Can cause extensive delays


Security classification


  Cost blow-outs / contractual impasses

  Schedule over-runs

  Quality shortfalls


Where Data can create problems

Cf the criteria for success!

does

5/24/11

26


  You are buying a new fleet of hovercraft from a major overseas hovercraft manufacturer, which are fairly standard but need some customisations.

  You also want a training simulator, but are unsure whether to procure this from the OEM, or from a specialist simulator supplier.

1.  What choices do you have?

2.  What are the pros and cons of each?


Exercise


Procure simulator with platform


Exercise

Procure simulator separately

Pros: • Get data • Interfaces are managed by the OEM

Cons: • You get what you get • Minimal customization • OEM may not be sim experienced • Little management influence – sim swamped by the platform

Pros: • Get what you want • Max customization • Low risk sim development

Cons: • Data difficult • You manage the interfaces

5/24/11

27



Exercise (ctd.) Hybrid – get sim from the OEM but locally manage sim

customizations Pros: • Get data • Get what you want • Most interfaces are managed by the OEM

Cons: • Needs to be provided for in the contract • Split accountabilities


  How to specify

  Training Needs Analysis

  Getting sign-off on Fidelity decisions

  Where Fidelity can create problems


Fidelity

5/24/11

28

Simulation Specific Contract Management - Fidelity

  Definition: The accuracy of the representation when compared to the real world.

  “As accurate as possible / necessary”

  Data

  Design criteria   Objective, testable characteristics of an element of the

design   Also specifies specific characteristics for a particular

“tail number”

  How close is good enough? – Depends on the TNA


How to specify


Purpose and importance

  Specifies the blended learning system - the mix of:   Class-room   Part-task trainer   Mission simulator   Real platform

  Sets the objectives for each element of the course

  Outputs   Analysis of which aspects of the course are allocated to

which medium   High-level syllabus


Training Needs Analysis

5/24/11

29


Process   Analysis Framework / tool

  For each element of the training task, look at what is required for:   Knowledge   Skills   Competencies   Physical – eg muscle memory

  Prioritise and weight

  Desk analysis by TNA expert with SME input

  Impact on Fidelity

  Example: Car driving simulator


Training Needs Analysis (ctd)


Impact on Fidelity

The fidelity has to be suitable for the particular training modality

Eg car driving (to teach safety to a novice driver)

  Steering wheel – physical size, weight, feel?

  Headlight switch - physical size, position, feel?

  Radio – size, position, operational, volume, sound quality?

  Air-conditioning controls - size, position, operational?


Training Needs Analysis (ctd)

5/24/11

30



Risk Based TNA

ie the risk

•  Looks at Hazards, their causes and control measures

•  The control measures inform the "Importance" part of the DIF

1 Managing Risks Through Training and Competence Management - Megan Brown, Llyod's Register Intermodal Conference, 8th November 2007

1


• Output of Training Needs Analysis – ie objective

• Subject Matter Experts

• Relate to the Training objectives!

• Stakeholders need to have a common view!


Getting sign-off on Fidelity decisions

5/24/11

31


• Working groups eg for Visual databases – gets buy-in from all stakeholders

• Examples: • Mark-up a video of a train’s OTW showing what features

outside the rail corridor are important • Airport – take photos of important cultural features

• Document all decisions with photos and GPS location!

• The contractual chain (ie from the ultimate user with the SME’s to the simulation supplier) can get in the way – need a short-cut – provide this mechanism in the contract.


Getting sign-off on Fidelity decisions (ctd.)

Any questions?

Peter Hill

Simcon Services Pty Ltd

PO Box 226 LINDFIELD NSW, 2070, Australia

Tel: +61 417 457 105

Email: [email protected] 31 May 2011 © Simcon Services Pty Ltd 62

5/24/11

32


Where Fidelity can create problems

• Negative training:   American Airlines Flight 587   The Airbus A300-600 crashed in an area of Queens,

New York on 12 Nov 2001   Negative training lead to the aircraft exceeding the

design envelope killing nine crew, 251 passengers and five people on the ground

  The negative training1 received within the simulator was directly attributed the incident.

  The three success factors again


Getting sign-off on Fidelity decisions (ctd.)

1Previous simulator training did not properly reflect "the actual large build-up in sideslip angle and side-loads that would accompany such rudder inputs in an actual airplane"


  Objective technique rather than subjective   Design criteria   Qualification Test Guide (for certification)

  Proof of Match

  Subjective (eg “golden arm”)   Avoid if possible!   Reserve for things impossible to objectively define

(eg the “look” of a radar, or ultrasound, or visual database)

  Tie to training outcomes

  Validation and associated issues


How to Accept

5/24/11

33


  An objective, well-defined benchmark

  Civil aircraft standards   Eg CASA Part 60 – 4 levels

  Applicability to military systems

  Standards for the platform (to be simulated) are often not applicable for the simulator   Air-worthiness

  Safety-critical software


Standards


  Few standards for development – only Civil aircraft and some rail

  Regular re-certification required


Accreditation standards

5/24/11

34


  Sources of change   Platform changes   Platforms have different configurations – concurrently

  Customer changes   Supplier deliverables


Configuration management


  How to manage   Baselines   Traceability of components including data

  Assign a “Tail Number” and upgrade the aircraft simulator as the aircraft themselves are upgraded

  Incremental upgrades to the simulator:   Are less traumatic than a “mid-life” crisis

  Minimize negative training / user rejection caused by difference between the platform and the simulator

  What if the aeroplane etc is not designed yet?   Planned, iterative development


Configuration management (ctd)

simulation contract management v0.1€¦ · 5/24/11 2 objectives and scope objectives introduction...

Documents