statistical analysis of kpis: the missing link in the vdc decision-making process
DESCRIPTION
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process. Calvin Kam (PI) - Civil and Environmental Engineering Martin Fischer (PI)- Civil and Environmental Engineering Sadri Khalessi (PI)- Statistics Devini Senaratna (RA)- Statistics. - PowerPoint PPT PresentationTRANSCRIPT
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Calvin Kam (PI) - Civil and Environmental EngineeringMartin Fischer (PI) - Civil and Environmental EngineeringSadri Khalessi (PI) - StatisticsDevini Senaratna (RA) - Statistics
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
2
A set of statistical models for innovative:benchmarking,
prediction of performance and adoption, decision-prioritization and
in Virtual Design and Construction projects
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Problem Statement
Why?
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
“What Gets Measured, Gets Done” –Peter Drucker
What do AEC firms need to “Get Done” using VDC?
4
Minimal waste in the process
Shorter total duration
Tight synchronization between design & fabrication
Last responsible moment benefit from VDC
More alternatives evaluated earlier
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Ref: McGraw and Hill (2012):The Business Value of BIM
AEC may not see the value in KPIs for VDC
5
Ref: Kam, McKinney, Xiao & Senaratna (2013): The Formulation and Validation of the VDC Scorecard
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Apparel industries: 80%VDC in AEC: 40%
AEC firms monitor KPIs for VDC less than other industries
Ref: Esin, et al. (2009) Are KPIs and Benchmarking actively used
among organizations of the Swiss apparel industry to assure revenue?
6
Ref: Kam, McKinney, Xiao & Senaratna (2013): The Formulation and Validation of the VDC
Scorecard
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
What information do AEC firms want from the KPIs?
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Frequency ofIntegrated
Meetings (Monthly)
Extent ofStakeholder Involvement(% of Stakeholder involved in VDC)
AEC project teams want to know where they stand relative to other AEC project teams
Where do we stand relative to other projects?
One data-driven benchmark that combines KPIs?
8
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Frequency ofIntegrated Meetings
Extent ofStakeholder Involvement
AEC project teams want to quantify and understand interdependencies among KPIs for VDC
How much more involved would the
stakeholders be?
How much more frequently would the stakeholders attend
the integrated meetings
9
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Our on-going research using the VDC Scorecard Tool
Ref: Kam, McKinney, Xiao & Senaratna (2013)
A survey based benchmarking tool
10
108 cases and counting50+ indicators13 countries11 types of facilities
• More cases for statistical modeling• Adding information to existing cases• Data-driven benchmarks• Data-driven prioritization of KPIs
What more do
we need?
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
10 Divisions
VDC Scorecard
Planning Adoption PerformanceTechnology
4 Areas
Objective Organization QualityQuantityIntegrationPreparation MaturityStandard CoverageProcess
Documentation StakeholdersReduced Design Errors
StakeholderSurvey
CommunicationBudget DepthGuidelineLevel ofDetail
Project
Metric TrainingReduced
LaborPost
Occupancy S.
Interoper-ability
Tool BreadthBenchmarkModel Use
Life Cycle
BroaderContext
Etc.
50+ Measures
Etc.Etc.Etc.
1 Score
11
VDC Scorecard measures many types of KPIs
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Decisions based on anecdotal learning’s only won't be as strong as making decisions that are
also supported by statistically established insights
Breadth
Involvement
Model UseLife Cycle
Depth
Reduced Design Errors
Stakeholdersatisfaction
ReducedLabor
Safety
Stakeholders
Project
Meetings
Benchmark
13
1542 MeasureCombinations
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Raw Data (KPIs)
“What gets measured, gets done “How can AEC firms benefit
from this advantage ?
Information= Statistical analysis of KPIs +
VDC expert opinion
14
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
What we propose
What?
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Ref: neural.cs.nthu.edu.tw
We will benchmark with a variant of Principal Component Analysis (PCA)
Few Features 16
Many KPIsFew KPIs for
benchmarkingand ranking
Many Features
Picks up the most important signals
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
17
Clustering to find features about “projects like mine”
Features of projects that are similar to mine
Features of projects that are different to mine
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Clustering groups of projects with similar KPIs together
Used in Recommendation EnginesMethods: K-means, Hierarchical, Two-Step Clustering
Projects with similar features will
group together
Feature Combination Score 1
Feat
ure
Com
bina
tion
Scor
e 2
Numerical statistical measures built using a
collection of KPIs
18
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Observation for Cluster Groups 1 and 3:
Per
form
ance A
do
ptio
n
Common Drawbacks :1. Interoperability problems
2. Inconsistent Level of Detail for model uses
3. Only native formats used for model exchange
19
GOOD Technological Coverage but
POOR Technological Integration
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Statistical predictive models to identify interdependencies
Factor Analysis
Structural Equation Modeling
Classification Trees
Canonical Correlations
Correspondence Analysis
For Predictionand
Prioritization
20
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Structural Equation Modeling will model interdependencies
Ref: Lee, Kleinman, (2003): Statistical choices and apparent work outcomes in auditing
Example of a Structural
Equation model for Job Dis-Satisfaction
21
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
How are we going to do it?How?
22
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Timetable
23
Addition of KPIs to Scorecard
Testing of statistical methods on sample data
Surveying108 cases (Online – Web survey)
Data cleaning and cross checking
Organizing the data for analysis
Descriptive Analysis and assumption checks
Statistical model based on SEM, FA, CHAID
Test data
Using expert opinions
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Dissemination of Findings
24
Two Industry Workshops (Kam, Khalessi, Fischer)• November CIFE workshop• February CIFE workshop
Contributions towards:• Statistical KPI wiki for AEC• CIFE members (AEC firms)• National BIM Standards and McGraw Hill Smart Market
Report
Publications New findings to be posted on vdcscorecard.stanford.eduJournal paper publication
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
The data-flow process of the proposed study
AEC Professionals
VDC Scorecard
Model Analyses KPIs and their Correlations
Prioritized, Customized
feedback based on KPIs
0 50 100 150
02
46
812
Rolling Window
1:180C
umm
ulat
ive
Ret
urn
0 50 100 150
04
812
Expanding Window
1:181
Cum
mul
ativ
e R
etur
n
Statistical Model
25
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Goals
Long Term Goal: Automation of this process for faster decision-making;and to make decision-making more consistent across
an organization
Short Term Goal: Statistical analysis of KPIs for VDC decision-making
26
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
References
• Abeysekera, S. (2006) Multivariate methods for index construction• Alarcón, L., Mourgues, C., O’Ryan, C., and Fischer, M. (2010) Designing a
Benchmarking Platform to select VDC/BIM Implementation Strategies• Bloom, N. and Van Reenen, J. (2006) Measuring and explaining management
practices across firms and countries• Bloom, N. and Van Reenen, J. (2007) Why do management practices differ across
firms and countries?• Esin, C., Von Bergen, M., and Wüthrich, R. (2009) Are KPIs and Benchmarking
actively used among organisations of the Swiss apparel industry to assure revenue? • Kam, C., McKinney, B., Xiao, Y., Senaratna, D. (2013) • Kam, C., Xiao, Y., McKinney, B., Senaratna, D. (2013) • McGraw and Hill (2012) The Business Value of BIM.• Lee, Kleinman, (2003): Statistical choices and apparent work outcomes in auditing• Li, W., Fischer, M., Schwegler, B., Bloom, N., Van Reenen, J. (2012)• Zevenbergen, J., Gerry, J., and Buckbee, G., (2006) Automation KPIs Critical for
Improvement of Enterprise KPIs.
27
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Don't you want to find out about which VDC practices will give
you the best results?
28
Copyright 2013
CIFE TAC 2013
Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process
Thank You!
29