2015 doe acquisition and project management workshopenergy.gov/sites/prod/files/2015/04/f21/workshop...
TRANSCRIPT
2015 DOE ACQUISITION AND PROJECT MANAGEMENT WORKSHOP
‘EVMS – FROM DATA TO DIAGNOSIS’
David Kester DCMA EVM Policy and Strategy
March 24, 2015
CONTENTS ***
1. DCMA At a Glance 2. Current EVMS Environment 3. The PM Connection 4. DCMA EVMS Mission Transformation 5. All Things Data 6. Mapping the EVMS Genome – EIA-748 Guideline 9 7. EVMAS – The Technology Solution 8. Role of the ACO in the EVMS Mission
2
DCMA AT A GLANCE ***
Vision One team, one voice delivering global acquisition insight that matters. Mission We are the independent eyes and ears of DoD and its partners, delivering actionable acquisition insight from the factory floor to the front line… around the world.
3
DCMA AT A GLANCE ***
• Number of civilian personnel: 11,473 • Number of military personnel: 528 • Number of contract management offices: 47 • Number of contractors: 20,155 • Number of active contracts: 353,200 with a total obligated
value of more than $1.9 trillion. • Number of countries with operations: 26 • Number of service lines: 15 - Earned Value Management Systems
4 Data as of September 2014
VALUE FOR MONEY ***
• Change The Way The Department Does Business By Changing The Way The
Department Thinks About EVMS
. . . Is EVMS A Burden Or Necessity?
8
Strategic Level Inter-Relationships Critical Path to DCMA EVMS Operations
USD (AT&L) PARCA
DoD EVM Policy DPAP
DoD Contractor Business Systems Rule
Office of Management &
Budget
DCMA PMI-V
EVMS Policy
DCMA-O DCMA-I DCMA-SP
EVMS Execution
ANX-205 EVMAA
IPMR Guide
DFARS 252.234-7002
EVMS
DFARS 242.302 Contract Admin
Functions
DoDI-5000.02 EVM Implementation
Policy
OMB Circular A-11 EVMS ANSI-748
Compliance
Title 48 - FAR Subpart
34.2
INST-210 SSI
INST-208 CRI
DFARS 234.201 EVMS Policy CFA
DFARS 234.203 Solicitation
Provisions & Contract Clauses
Execution
DFARS 252.234-7001
Notice of EVMS
ANX-412 CIA
INST-205 MPS
INST-131 CBS
DAES
Performance Indicators (Feedback)
INST-214 EVMCP
DFARS EVMS POLICIES
5 PMI EVMS Policies
CMO utilize Contractor EVMS
data To generate
monthly PARS
242.302 Contract Administration Functions (S-71) DCMA has responsibility for reviewing earned value management system (EVMS) plans and for verifying initial and continuing contractor compliance with DoD EVMS criteria. The contracting officer shall not retain this function.
234.201 EVMS Policy (3) The Defense Contract Management Agency is responsible for determining earned value management system compliance when DoD is the cognizant Federal agency.
DoD EVMSG 9
DFARS 252.242-7005
Contractor Business Systems Rule
SUBPART 234.2 - EARNED VALUE MANAGEMENT SYSTEM
*** One Part Management Strategy, One Part
Management Control System
10
PROJECT MANAGEMENT ***
Project Management Institute, Inc. (PMI) defines project management as "the application of knowledge,
skills, tools and techniques to a broad range of activities in order to meet the requirements of a
particular project."
11
5 PHASES OF PROJECT MANAGEMENT
*** 1. Project conception and initiation 2. Project definition and planning 3. Project launch or execution 4. Project performance and control 5. Project close
12
5 PHASES OF PROJECT MANAGEMENT
*** 4. Project performance and control
Project managers will compare project status and progress to the actual plan, as resources perform the scheduled work. During this phase, project managers may need to adjust schedules or do what is necessary to keep the
project on track.
13
• A System of Systems • Value Comes Through A Smart Design and An Even
Smarter Implementation Strategy
• Requires Data That Is Timely, Trusted, And Relevant
Anatomy Of An EVMS
14
• Minimum allowable acceptance conditions:
• Ensures a clear definition of work prior to beginning that work
• Presents a logical plan for accomplishing the work • Provides an objective measure of accomplishments • Early and accurate identification of trends/issues • Accurate picture of contract status • Basis for course correction
. . . Better Data (Ingredients), Better Decisions (Pizza)
Defining EVMS Success
15
1
4
9
2 1
3
9
1 1 2 2
7
2 2 2 1
5
2
5
1 1 1
15
5 5
2 2
0
2
4
6
8
10
12
14
16
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Num
ber o
f Rev
iew
s
Guideline
FY13-14 Total Non Compliant EVMS GLs
• Total of 93 Guidelines were found to be Non Compliant in the 23 major reviews • GL 6 (schedule with network logic) – 9 of 23 reviews (40%) • GL 10 (create work/planning packages) – 9 of 23 reviews (40%) • GL 16 (record direct costs) – 7 of 23 reviews (30%) • GL 27 (revise estimate at completion) – found non compliant in 15 of 23 reviews
(65%) S P
A
EAC
16
∂
BIG DATA - Transforming How DCMA Executes Its DFARS EVMS Compliance
Mission ***
1. Provide Clarity 2. Ensure For Consistency 3. Reduce or Eliminate Costs
17
Initiative 1.3.3 – Alternative Approaches to EVMS Compliance
*** Revise current EVMS Policy and CONOPS to streamline how DCMA conducts the EVMS mission (Compliance, Surveillance, and Analysis), and target cost savings of 20%
18
PRIMARY GOALS ***
• Influence industry partners to field effective (and efficient) EVMS that are compliant with the 32 ANSI-748 guidelines
• Improving Program Management by the way we think about (and use) EVMS
• Strengthen the purchasing power of the Department
19
BLUF – DATA DRIVEN ***
• Rethinking the way in which EVMS compliance (initial) and surveillance (continuing) can be performed to simultaneously test the reliability of core management processes, and to analyze data sets to summarize, detect patterns, and draw conclusions about contract performance • Data-driven compliance to help predict when and where
issues will occur
• Targeted Compliance and Surveillance • Predictor variables identify and localize the issue, optimizing
surveillance and minimizing disruptions • Surveillance driven by data /emerging issues, not by the
calendar • Consistent application regardless of EVMS specialist
20
A Drop of Blood (Data), An Instant Diagnosis – Early Detection,
Prevention
Source: Theranos - Elizabeth Holmes, President and CEO
21
THERANOS = Aligning
Technology with Strategy
PREMISE ***
• The success of the data-driven EVMS compliance approach is reliant upon the availability and quality of the data gathered and the effectiveness of its analysis and interpretation.
• Human error can creep into the EVMS compliance process at any stage of the endeavor and serious consequences can result when they do.
• A data-driven approach to EVMS compliance places value on decisions that can be backed up with data that can be verified.
22
WHY A DATA DRIVEN APPROACH? ***
• Yields a more structured and disciplined analytical approach to evaluating the assessment
• Highlights potential risks in EVMS implementation that might otherwise be unknown
• Better allocates resources where pay-back is greatest • Reduces (and works towards) eliminating unnecessary
and inconsistent data requests and CAM discussions • Projected time (cost) savings for the enterprise (and aligns
with the AT&L Better Buying Initiative)
23
Source: From Keeping Up with the Quants: Your Guide to Understanding and Using Analytics (HBR Press)
A Focus On The Quantitative Analysis Of Systems And Processes
*** Developing numerical (or quantitative) techniques for the
analysis and verification of system properties such as reliability (trustworthiness) and relevance (usefulness) of data
24
DATA - The New Raw Material ***
• More Data Doesn’t Guarantee Better Decisions • More Data Is Useless If We Aren’t Asking The Right
Questions • Asking The Right Questions Requires Looking At The
Right Data In The Right Way • Knowing How Long Data Must Be Retained • Data Access - 252.234-7002 Earned Value
Management System as prescribed in 234.203 • (f) The Contractor shall provide access to all pertinent records and data
requested by the Contracting Officer or duly authorized representative as necessary to permit Government surveillance to ensure that the EVMS complies, and continues to comply, with the performance criteria referenced in paragraph (a) of this clause
• Data Formatting 26
COMMON DATA FORMATTING ISSUES ***
• Inconsistent data content • Tied to paper based forms and reports that are useless in a digital environment (scanned documents or PDFs) • High volume of non-value added data content • Lots of human effort to collect, enter, sort through, or manipulate the data
27
UN/CEFACT /XML ***
• Allows for the combination of more efficient processing, more accurate searching, and more flexible linking that will significantly reduce costs
• Toolset neutral –program management software vendors are supporting as a common means to import or export data
28
MAPPING THE EVMS GENOME ‘Cracking The Code’
• 32 ANSI-748 Guidelines
• 73 Guideline Attributes • 127 Tests
• 189 Metrics
• Tailored Protocols • Engineering and Development • Construction • Production Assembly
29
30
MAPPING THE EVMS GENOME ***
DCMA role for executing compliance and implementing remains unchanged by the OSD EVMSIG as stipulated : • Section 1.2: “…when DoD is the cognizant federal agency,
the Defense Contract Management Agency (DCMA) is responsible for determining EVMS compliance.”
• Section 1.1: “… agencies and organizations charged with conducting initial and continuing EVMS compliance activities will establish amplifying agency procedures and/or guidance to clarify how the respective agencies and organizations are implementing this guide to include the development of evaluation methods for the attributes associated with each of the 32 guidelines.”
Guidelines 32
EIA-748 GUIDELINE CLASSIFICATIONS ***
2.31 Per Guideline
Attributes* 155 Before
63 After 73 EVMSIG
Test Steps 330 Before
136 After 127 EVMSIG
1.72 Per Attribute
*Attribute - A Quality Or Characteristic Of A Guideline That Makes It Distinct From Other Guidelines
52% Reduction In Attributes 62% Reduction In Test Steps
31
Testing Algorithm ***
Example: Write an algorithm to determine a student’s final grade and indicate whether the grade is ‘passing’ or ‘failing’. The final grade is calculated as the average of four marks.
32
Testing Algorithm ***
Step 1: Input M1,M2,M3,M4
Step 2: GRADE ← (M1+M2+M3+M4)/4 Step 3: if (GRADE < 60) then Print “FAIL” else Print “PASS” end
33
Testing Algorithm Flowchart
PRINT “PASS”
GRADE←(75+65+80+70)/4
IS GRADE<
60
PRINT “FAIL”
STOP
Y N
START
INPUT 75+65+80+70
73
TAKE ACTION
34
ARTIFACT
CALCULATION
METRIC
THRESHOLD
ROOT CAUSE
EIA-748 Guideline 9 Testing Protocol
*** • EVMS Category: Planning, Scheduling, and Budgeting
• EIA Standard Guideline: 2.2d - Authorize and Budget by
Cost Elements
• Intent of Guideline: Establish budgets for authorized work with identification of significant cost elements (labor, material, etc.) as needed for internal management and for control of subcontractors
35
• Distributed Budgets reflect the contractually authorized efforts allocated to WBS elements
• Distributed Budgets include budgets assigned to control accounts during baseline establishment as well as work packages and planning packages
36
• Are the distributed budgets assigned to control accounts planned and identified in terms of the following cost elements:
(1) Direct labor dollars and/or hours? (2) Material and/or subcontract dollars? (3) Other direct dollars?
• Accomplishing the cost account resource plan is the crux of this criterion question. It requires that the budgetary resource plan be devised by element of cost; i.e. labor (in terms of dollars or hours), material (dollars), and other direct charges (dollars)
• This type of budget segmentation adds discipline and integrity to the control account's resource plan and ensures that the correlation is made between every budget dollar and the entity that dollar is supposed to buy
• The work authorization system is what translates the contractor's plans into practice
37
EVMAS – Earned Value Management Analytics System Is Intended To Empower
Program Teams (PMs, ACOs, IPTs) ***
39
EVMAS – Aggregate Data To Identify (And Predict) EVMS Non-compliance
*** • Correlation and Causation
• Identifying The Degree And Type Of Relationship Between Any Two (Or More) Quantities, And Then Understanding The Reasons Why
• Predictor For The Likelihood Of An Issue
40
GL 1 0101
0102
0103
GL 17
1701
G
GL 2
0201
0202
GL 18
1801
G
GL 3 0301a
0301b
0301c
0301d
0301e
0301f
0301g
0301h
0301i
0303a
0304a
0304b
GL 19
1901a
1902a
G G G G G G G G G
GL 4 0401
0402
GL 20
2001
2002
GL 5
0501a
0502a
0503a GL 21
2101a
2102a
2102b
2102c
2103a
2104a
2105a
G G G G G G G R
GL 6
0601a
0602a
0603a
0604a
0605a
0606a
0607a
0608a
0609a
0610a
0611a
0612a
0613a
0614a
0615a
0616a
0617a
0618a
0622a
0619a
0620a
0620b
0621a
0621b
GL 22
2201a
2202a
2202b
2202c
2202d
2202e
2202f
2202g
2202h
2202i
2202j
G G G G G G G G R G G G G G G G G G G G G G G G G G G G G G
GL 7
0701a
0702a
0702b
0703a
0704a GL 23
2301a
2304a
2306a
R G
GL 8 0801
0802
0803
0804
0804
0805
GL 24
2401
2403
G G G
GL 9
0901a
0902a
0903a
0905a GL 25
2501a
2501b
2501c
2501d
2501e
2501f
2501g
2501h
2502a
2502b
2502c
2502d
2502e
2502f
2502g
2502h
G G R G G G G G G G G G G G G G G G G G
GL 10 1001
1002
1003
1004
1004
1005
1006
1006
1007
1008
1009
1010
1011
1012
1013
GL 26
2601
2602
2603
G G R G G G R G G G G G R G G
GL 11 1101
1102
GL 27
2701
2702
2702
2702
2702
2703
2704
2705
2706
2707
2708
2709
2710
2711
2712
G G R G R R G R G G G G G G
GL 12 1201
GL 28
2801
GL 13 1301
GL 29
2901
2901
G G
GL 14
1401a
1402a
1403a
1404a
1405a
1406a
1407a
1408a
1409a
1410a
1411a
GL 30
3001a
3001b
3001c
3001d
3001e
3001f
3002a
3003a
G G G G G G G G G G G G G G
GL 15
1501
1502
GL 31
3101
G G
GL 16
1601
1602
1602
1603
1603
1604
1605
1605
1605
1606
1606
1606
1607
1608
1609
GL 32 32
01
3202
3203
3204
3205
3206
3207
G G G G G G G G G G G G G G R
DCMA PROOF OF CONCEPT
41
… An Instant Diagnosis – Early Detection, Prevention
MARKERS
The use of an EVMS maturity model will allow for a contractor to have its methods and processes categorized against a clear set of criteria
Maturity levels could be used to determine ongoing Government surveillance requirements
Mechanism to communicate EVMS risks to customers and stakeholders
Results Matter
Source: Credit Karma
42
Role of ACO in EVMS Mission ***
• Identify EVMS Contract requirements through Contract Receipt and Review (CRR)
• Maintain the Contract Business Analysis Repository (CBAR) • Review EVMS functional reports submitted in accordance
with Contractor Business System (CBS) policy when a system review is conducted or material deficiencies are identified
• Approve/Disapprove EVMS based upon specialist findings • Progress Payment Administration
• Percent of physical completion; Estimates-at-completion 43
• DFARS 252.242-7005 clause added to contracts requiring contractor’s to certify that they have no major (or significant) deficiencies in their systems:
1. Cost Accounting 2. Purchasing 3. Estimating 4. Property Management 5. Earned Value Management System 6. Material Management
• Contracting officers would have the authority to withhold payments on cost reimbursement, incentive-type, time-and-materials, and labor-hour contracts
• Required for one or more 16 high-risk (guidelines 1, 3, 6, 7, 8, 9, 10, 12, 16, 21, 23, 26, 27, 28, 30, or 32), discretionary for the other 16 guidelines
• Enforcement relies on auditors or other functional specialists to document any business system deficiencies
Contractor Business Systems Rule
44
Step 1 EVMS Compliance Review & Report Step 2 DACO Initial Determination Step 3 Supplier Written Response Step 4 DACO Board of Review Step 5 DACO Final Determination (2-5%) Step 6 DACO Initiates Withholds Step 7 Supplier Response (CAP) Step 8 DACO Verify and Validate Corrections Step 9 DACO Considers Withhold Reduction Step 10 DACO Removes Withholds
CMO Functional Specialists working through the COO
are to Assess the Contractor’s corrections within 90 Days of receipt
Contractors are Required to Respond to the ACO
within 30 Days
COO Functional Specialists are to generate a
Business System Analysis Summary within 45 days
DCMA Business Systems Process ***
45
WHAT IS A SIGNIFICANT DEFICIENCY?
*** • DFARS 252.234-7002 - Earned Value Management
System • DFARS 252.242-7005 - Contractor Business Systems
“Significant deficiency means a shortcoming in the system that materially affects the ability of officials of the Department of Defense to rely upon information produced by the system that is needed for management purposes.”
46