structuring performance data to effectively drive management decisions tim gribben, josh bull, &...
Post on 01-Apr-2015
216 Views
Preview:
TRANSCRIPT
Structuring Performance Data to Effectively Drive Management Decisions
Tim Gribben, Josh Bull, & Mat Pascarella
OFFICE OF PERFORMANCE MANAGEMENTU.S. SMALL BUSINESS ADMINISTRATIONDecember 5, 2012
1
Embedding Performance Management
Cascading Data-Driven Decision-Making
Framework of SBA’s Training Course
Permeation & Permutation of Training
2
Embedding Performance Management
Cascading Data-Driven Decision-Making
Framework of SBA’s Training Course
Permeation & Permutation of Training
3
Performance Management as Organizational Culture
• Senior Leadership must use data-driven quarterly performance reviews of CAP and Priority Goals
(GPRA Modernization Act of 2010)• Many agencies run this exercise
out of their respective Performance Management Offices
• However, for performance management to effect dramatic change, the more difficult task will be to integrate the same data-driven decision-making throughout the agency, especially among program managers and mid-level analysts.
Senior Leadersh
ip
Program
Managers
Mid-level Analysts
4
Recognizing a Problem & Finding the Solution
IDENTIFIED GAP
SBA not effectively
analyzing the wealth of data available nor using data
effectively to drive
management decisions
LEADERSHIP BUY-IN
Office heads select interested
managers or analysts with basic experience in data analysis and willing to commit 3-4 hours per week to training
PMO RUNS TRAINING
PMO develops six week training
course to teach the skills via
lecture, activities, and homework as
well as “real-world” final presentation
5
‘Lesson Learned’ continually improves course content & format
Final presentations raised by course participants turn into agency-wide projects
Future presentations to mgmt incorporate class teachings
Making The Training “Stick”
Goals
Expectations
MANAGERS INSTITUTIONALIZATION
Getting buy-in from managers for maximum impact on participants, from
executive leadership is key for institutionalization
Invest in personal & professional development of employees
Surface actionable, interesting analytic questions for participants to explore Required to meet with their participating employees to discuss training goals and expectations
Asked to check in on participants to proffer help or feedback during training
Including phrase into SBA’s new strategic plan –
“Investing in SBA’s employees through business, technical, and analytical training”
6
Summary of SBA’s Path
Identified skills gap related to data analysis / analytical presentations Relatively few used data effectively to inform management decisions Reliance on data summarization without insights
WHERE WE WERE IN 2009
Created a training course to build new skills Included it as an objective & strategy in the new strategic plan Senior leaders and managers were also trained in the process
WHAT WE DID
Data is being presented more effectively to inform management decisions
Senior leaders are using data to address management challenges
WHERE WE ARE NOW
7
Embedding Performance Management
Cascading Data-Driven Decision-Making
Framework of SBA’s Training Course
Permeation & Permutation of Training
8
Structure the Presentation
Communicate the analysis
Synthesize & Develop
Recommendations
Session
KeyLessons
Structuring a management presentation
Developing a story-line
Presenting and communicating with data
Creating visual charts and graphs
Synthesizing findings into management recommendations
Developing data-driven ‘so-what’ insights
PURPOSE OF TRAINING: To help participants develop and structure analytic presentations that inform SBA
management decisions
Training Purpose & Format9
WHAT THE TRAINING IS
WHAT THE TRAINING ISN’T Training to help develop
presentations that inform management decisions
Structuring
Communicating
Data Synthesis
Way to EMPOWER employee to drive SBA business decisions
Χ Excel 101
Χ Powerpoint 101
Χ How to Report Data
Χ Oral Presentation Coaching
Χ Understanding SBA’s Programs
Risk-free environment Χ Formal Assessment of . Employee’s Abilities
Interactive with assignments
Group oriented
Χ US lecturing THEM
Χ Rigid Curriculum
Data Analysis Training Ground Rules
10
Structure the Presentation
Communicate the analysis
Synthesize and Develop
Recommendations
Two step process – “Structure the Presentation”:• Structure the management question (using an
Issue True) Organize the issues into buckets Anticipate the data and analyses that will
answer the questions
• Build a story line (using a “Ghost Deck”) Plan the work to be done Align with managers on direction before
investing the time
1
2
SESSION #1: Structure the Presentation
11
While developing the ‘Issue Tree’ to structure issues, we might consider two
principles to answer ALL and the most RELEVANT questions
1
…and keep digging deeper (ask “So-
What?” or “Why?”) to ensure the issues are
relevant
Example: Why is my credit card bill
increasing?
Be “Mutually Exclusive,
Collectively Exhaustive” (MECE) to identify all issues…
Example: How can I save money for a
vacation?
Structure the Presentation12
How can I save money
for a vacation?
Spend LESS money
Earn MORE money
Questions and analyses to consider
Can I earn more interest on existing income?Can I invest in stocks/funds instead of checking account
Can I earn new income?Can I sell garage items by having a yard sale
Can I buy cheaper items?Coffee machine, not Starbucks; Shop at discount stores
Can I buy fewer items? Spend less on gas by driving at 55 mph on the highway
Be “Mutually Exclusive, Collectively Exhaustive” (MECE) to identify all issues
1Structure the Presentation: Organize the Issues
13
Situation ComplicationsOptions to Consider
Depending on the purpose, a senior management presentation might include
the following components
Inform a
Decision
Problem(s) to address
Recommendations & Rationale
Implementation Plan
Build a Proposa
l
Purpose Potential Considerations
Reason for Action (or context)
Actions TakenResults Achieved
Showcase a
Success
2 Build the Story-line14
• How to Build an Issue Tree
• Mutually exclusive, collectively exhaustive (MECE)
Start as MECE as possible. It’ll help as you build.
• How to Turn an Issue Tree into Relevant Questions
Keep asking “so-what” and “why”
• Approaches for building a story line
Situation, complication, resolution
• Constructing a “ghost deck”
Key Lessons: Structure the Presentation
15
Synthesize and Develop
Recommendations
Structure the Presentation
Communicate the analysis
• Determine your message
1
• Identify the comparison
2
• Select the best chart type
3
MATERIAL ADAPTED FROM GENE ZELAZNY’S “SAY IT WITH CHARTS”
SESSION #2: Communicate the Analysis
16
The chart type you choose depends on the
specific point or message you want to
conveyTherefore, the first step is to identify
what message you want to share
1 Determine Your Message17
5 TYPES OF COMPARISONCOMPARISON
TYPEPURPOSE EXAMPLE
Component Displaying the size each component as a percentage of the total
X% of government contracts go to small businesses
Item Compare how things rank
Center X ranks fourth among all regions for program activity
Time Series Trends or changes over time
Student loan debt increased significantly
Frequency Distribution
Number of items that fall into numeric ranges
Most grants fall into the $15 million -$25 million range
Correlation Identify the strength of a relationship between two variables
There is a strong relationship between employee satisfaction and employees’ awareness of responsibilities
Source: Gene Zelazny, “Say it with Charts”
2 Identify the Comparison18
KEY TYPES OF CHARTS
Pie chart
Bar chart
Column chart
Line chart
Dot chart
Component
Item Time series
Frequency
Correlation
Pie
Bar
Column
Line
Dot
3 Select the Best Chart Types19
SCENARIO COMPARISON TYPE
CHART TYPE
Percentage of government contracts by business size classifications (i.e., small business)
Relationship marketing outreach & attendance at national parks
Change in student debt levels from 2007-2010
Breakdown of the value of goods exported by region, as a percentage of the total
Breakdown of grant awards by size buckets (i.e. 0-$500k; $500k-$1M, etc.)
Frequency of small business government contracts by NAICS code
Employee satisfaction vs. AWS participation rate
Speed of disaster assistance for a given disaster by time buckets (i.e., 1 week; 2 weeks; etc.)
2009-2010 weekly count of White House press releases
3Exercise!!!
20
• Determine the message
• Identify the comparison
Component, Item, Time Series, Frequency Distribution, Correlation
• Select the chart type
Pie, Column, Bar, Line, Scatter, or Other
Key Lessons: Communicate the Analysis
21
Communicate the analysis
Structure the Presentation
• Examples of Synthesis
• How to Synthesize & Develop Management Recommendations
Synthesize and Develop
Recommendations
1
2
Session #3: Synthesize & Develop Recommendations
22
Don’t just summarize. Summary re-states all the facts briefly; synthesis includes insights that help managers make decisions
Ask “so-what”, “why”, or what does this mean for my manager, office, team, constituent, etc.?
Lead with the answer (but it depends on the context)
Foreshadow: Your initial synthesized statement should map to the remainder of the actions/recommendations/analysis
Develop actionable recommendations that are supported by the facts
1 Tips for Synthesis23
1 • Structure/highlight the major issues
2 • Spell out the recommendations
3 • Present the action plan (or path forward)
4• Ask for agreement/commitment (if
appropriate)
5 • Close off with “next steps”
1Potential Components of a Synthesis
24
Dear Elizabeth W.,
How are you? I hope this note finds you well. I must say, I have not been doing too well—everything seems to be going wrong. My life is in shambles. For example, I had to pull two all-nighters last week for a very big report that was due last Friday. My boss said he told me about in two months earlier, but I totally forgot about it until he asked for a semi-final draft last Wednesday. I scrambled and just barely got it done—though I am so exhausted now. OMG.
And then this weekend was really sad because it was my 30th birthday on Saturday, and nobody came to my party!!! On Wednesday, I sent people a text message Friday night saying “come to my house tomorrow for dinner at 8 pm”. Half of my friends already had plans, and the other half went to my old house—because I forgot to tell them I moved!! Then they got mad at me for all the hassle. So my dog and I alone ate the 5 trays of meatloaf I cooked. Ugh.
And to top it off, I got a call from my bank, and they are going to repossess my car! Apparently I missed three months of payments (I totally forgot to send them my new address for the bills!). And when I tried to use my credit card at the grocery store, it didn’t even work, as it had been shut off after reaching my credit limit. Now I’m hungry and can’t drive to work.
What is wrong with me? Why do I have the worst luck?
Sincerely,
Josh
275 Words
2EMAIL EXAMPLE: Before synthesis…
25
2EMAIL EXAMPLE: After Synthesis…
Dear Josh, It seems you have a lot going on in your life. You may need some personal training in the areas of:
• Time Management• Communication• Organizational Skills
Have you tried using a Life Coach? Let me know if I can help. Good Luck!
Sincerely, Elizabeth W.
49 Words
26
2EMAIL EXAMPLE: Model Structure for an Answer
96 Words
Dear Josh,You’re disorganized—you need to better plan/schedule your work activities, social gatherings, and finances.
• At work, you haven’t anticipated deadlines or managed workload, causing work to build. Keeping a better team calendar may help.
• In social settings, you don’t plan events with advance notice and clear instructions, leading to damaged relationships. You should communicate better with friends, and send formal party invitations.
• Financially, you have missed several due dates, damaging your credit standing. Consider on-line ‘bill pay’ and other tools to easily update your records and manage your expenses.
Sincerely, Elizabeth W.
27
Dear Josh,You’re disorganized—you need to better plan/schedule your work activities, social gatherings, and finances.
• At work, you haven’t anticipated deadlines or managed workload, causing work to build. Keeping a better team calendar may help.
• In social settings, you don’t plan events with advance notice and clear instructions, leading to damaged relationships. You should communicate better with friends, and send formal party invitations.
• Financially, you have missed several due dates, damaging your credit standing. Consider on-line ‘bill pay’ and other tools to easily update your records and manage your expenses.
Sincerely, Elizabeth W.
Actionable recommendati
ons
Track the
options
What is the
action?
Lead with the
answer
Maintain a
structure
Highlight implicatio
ns
2EMAIL EXAMPLE: Model Structure for an Answer
96 Words
28
Amount =< $25K
$25K < Amount =<
$50K
$50K < Amount =<
$100K
$100K < Amount =<
$150K
$150K < Amount =<
$350K
$350K < Amount =<
$500K
$500K < Amount =<
$1M
$1M < Amount =<
$1.5M
$1.5M < Amount =<
$2M
Regular 7a 10.4% 8.7% 8.7% 10.3% 10.4% 9.0% 8.1% 9.8% 7.2%
SBA Express 15.1% 17.7% 14.3% 6.7% 4.2%
0%
5%
10%
15%
20%
25%
Nor
mal
ized
Pur
chas
e Ra
te
Normalized Purchase Rates, 2005 Cohort
Because larger SBA Express loans are
better performing, raising the caps on SBA Express above $350K may have a modest impact on
program cost
Amount =< $25K
$25K < Amount =<
$50K
$50K < Amount =<
$100K
$100K < Amount =<
$150K
$150K < Amount =<
$350K
$350K < Amount =<
$500K
$500K < Amount =<
$1M
$1M < Amount =<
$1.5M
$1.5M < Amount =<
$2M
Regular 7a 5.8% 9.7% 10.9% 13.4% 13.3% 11.9% 10.2% 10.8% 10.7%
SBA Express 16.6% 18.0% 17.5% 5.0% 4.6%
0%2%4%6%8%
10%12%14%16%18%20%
Nor
mal
ized
Pur
chas
e Ra
te
Normalized Purchase Rates, 2006 Cohort
2Although SBA Express loans have relatively higher purchase rates, larger SBA Express loans are higher performing
Note: Normalized purchase rate is defined as SBA purchase amount as a percent of SBA-guaranteed portion of the total disbursement amount.
PowerPoint Example
29
Amount =< $25K
$25K < Amount =<
$50K
$50K < Amount =<
$100K
$100K < Amount =<
$150K
$150K < Amount =<
$350K
$350K < Amount =<
$500K
$500K < Amount =<
$1M
$1M < Amount =<
$1.5M
$1.5M < Amount =<
$2M
Regular 7a 10.4% 8.7% 8.7% 10.3% 10.4% 9.0% 8.1% 9.8% 7.2%
SBA Express 15.1% 17.7% 14.3% 6.7% 4.2%
0%
5%
10%
15%
20%
25%
Nor
mal
ized
Pur
chas
e Ra
te
Normalized Purchase Rates, 2005 Cohort
Because larger SBA Express loans are
better performing, raising the caps on SBA Express above $350K may have a modest impact on
program cost
Amount =< $25K
$25K < Amount =<
$50K
$50K < Amount =<
$100K
$100K < Amount =<
$150K
$150K < Amount =<
$350K
$350K < Amount =<
$500K
$500K < Amount =<
$1M
$1M < Amount =<
$1.5M
$1.5M < Amount =<
$2M
Regular 7a 5.8% 9.7% 10.9% 13.4% 13.3% 11.9% 10.2% 10.8% 10.7%
SBA Express 16.6% 18.0% 17.5% 5.0% 4.6%
0%2%4%6%8%
10%12%14%16%18%20%
Nor
mal
ized
Pur
chas
e Ra
te
Normalized Purchase Rates, 2006 Cohort
2Although SBA Express loans have relatively higher purchase rates, larger SBA Express loans are higher performing
Note: Normalized purchase rate is defined as SBA purchase amount as a percent of SBA-guaranteed portion of the total disbursement amount.
HIGHLIGHT THE
INSIGHTTHE SO-
WHAT
A DESCRIPTIVE LEAD
HIGHLIGHT THE
INSIGHT
PowerPoint Example
30
• Synthesize EVERYWHERE
• Ask “so-what”, “why”, or what does this mean
• Lead with the answer (depending on the context)
• Foreshadow
• Develop actionable recommendations
• Don’t just summarize. Highlight the insights
KEY LESSONS: Synthesize & Develop Recommendations
31
• Teams of 2-4 participants• Judges will be “VIP” SBA panel
Who?
• Develop and present a 5-10 page management PowerPoint presentation on any SBA-specific topic. Work should:
o Exhibit command of structuring and story-lining
o Use data-driven charts and graphs effectively
o Contain clear, actionable management recommendations and synthesis
What?
• To show everyone what they have learned• To have fun
Why?
SESSION #4: Applying the lessons…THE CONTEST
• Prizes for best weekly assignments• GRAND PRIZE for best final
presentation• Lunch at White House with Administrator
What can I win?
32
COMPETITION CHECKLIST: Criteria to Consider for Data-Driven Presentations
Mastery Area FeatureStructuring
the PresentationDoes the presentation build a compelling, coherent story?
Does the presentation identify (and prioritize) the most appropriate issues for analyzing a specific management question?
Communicating the Data
Does the presentation incorporate relevant data/evidence to inform a management decision?
Is the data presented in a visually-compelling format that sends a clear message?
Synthesizing Findings & Developing
Recommendations
Does the presentation surface interesting, high-level insights supported by the data?
Does the presentation provide actionable recommendations (or options) for management to consider for improving SBA’s operations/programs?
33
Embedding Performance Management
Cascading Data-Driven Decision-Making
Framework of SBA’s Training Course
Permeation & Permutation of Training
34
PERMEATION: Cultural shift created in the SBA
RESULTS OF HQ CLASSESo 1st Class Winner: Senior level task force formed that developed a
comprehensive plan to address retirements and led to a VERA/VSIP action.
o 2nd Class Winner: Analysis being used to inform Agency efforts for boosting volume of loans going to underserved markets.
35
Performance data highlighted a management question that needed exploration
SBA’s Average Time-to-Hire was Worse than Agency Target
36
An Issue Tree was created to ask relevant questions and inform the data analysis
How can SBA
reduce time to hire?
Cut # of steps
Cut time
per step
SBA Steps
Candidate Steps
Program Office
OHCM
Utilize hiring
flexibilities
Pre-Intervie
w
Interview
Offer
…
Direct Hire Authority
Excepted Service
Veterans
Part-Time Reemploymen
t (2010 NDAA)
STEP
PMF
Questions to Assess
• How many employees are eligible to retire?
• Ease of execution?
• Cost implications?
• Continuity in critical missions?
• Mentorship opportunities?
• Infusion of new talent?
37
Further Data Analysis Led To:
RISK In 5 years,
Retirement & VERA eligible employees will
outpace ineligible employees by
66% (2 out of 3 GS will qualify for an immediate annuity through voluntary or early retirement)
CURRENTLY
+3 Years
+5 Years
How can SBA mitigate effects of impending retirement tsunami?
38
How can SBA loan dollars more effectively go to underserved communities? Questions and analyses to
explore
Increase Demand for
Loan Programs by Underserved Community
Increase outreach to underserved
business communities
Increase the number of
businesses that qualify for loans
• What loan programs do banks use most in underserved communities?
• What are the characteristics/criteria for these loan programs?
Increase Supply of Loan to
Underserved Community
Increase the # of SBA
programs targeting
underserved businesses
Increase the number of
banks lending money in
underserved communities
• How many banks are serving small businesses in underserved communities? • What are characteristics of banks in underserved communities? (Size, national branch, local bank, etc)?
• What percentage of SBs are approved for loans through targeted underserved programs?
• Are SBA loan programs evenly distributed to underserved communities, by Region?
• Are SBA loans programs reaching the high concentration of underserved small businesses that they are created for?
39
21.4%
14.8%
12.8%
10.1% 10.1% 10.0%
6.7% 6.5%
5.2%
2.5%
0%
5%
10%
15%
20%
25%
8 6 1 3 7 4 10 5 2 9
SBA Region
% Distribution of Lenders, Underserved Businesses and SBA Dollars
% of Lenders
% of 8a/Hubzone
% of SBA USM Lending
21.4%
14.8%
12.8%
10.1% 10.1% 10.0%
6.7% 6.5%
5.2%
2.5%
0%
5%
10%
15%
20%
25%
8 6 1 3 7 4 10 5 2 9
SBA Region
% Distribution of Lenders, Underserved Businesses and SBA Dollars
% of Lenders
% of 8a/Hubzone
% of SBA USM Lending
Lender Outreach can be tailored to be region specific
.
Region 9’s Best Practices should
be shared
Highest concentrati
on of Lenders
USM Need not aligned
with results
A chart was used that highlighted insights…
40
PERMUTATIONS: The Training Course at SBA
BASICPrinciples
4 hours Covers basic
topics much like this presentation
Example Audience: Senior Leadership
CONDENSED Seminar
1.5 days More in-depth look at
principles Built-in homework Pre-defined options
and data sets for the final presentations
Example Audience: regional/national conference attendees
PRIMARYFull Course
4 to 6 weeks, each 2-3 hrs
Individual & Group homework
Deep dive into agency’s data sets & team-defined ACTIONABLE projects
Incentives for final presentation
Example Audience: Headquarters staff or dedicated field office
GOOD BETTER BEST
41
Interested in Finding Out More?
CONTACTTim Gribben
Director, Office of Performance Management
Timothy.Gribben@SBA.gov (202) 205-7565
U.S. Small Business Administration409 3rd Street, SW, Ste 6000
Washington DC 20416
Josh BullProgram Analyst & Budget Analyst,
Office of Performance Management and Office of CFO
Joshua.Bull@SBA.gov (202) 205-7603Mat Pascarella
Program Analyst, OCFOMathew.Pascarella@sba.gov (202) 205-
6102
42
top related