examiner training july 2011. the one award will identify and promote best practices in non-profit...
TRANSCRIPT
The ONE Award will identify and promote best practices in non-profit organizations and recognize our region’s Organizations of Noteworthy Excellence
Agenda
• Overview• Roles, Responsibilities, Expectations
• 2011 Criteria
• Process• Independent Review
• Consensus
• Site Visit
• Questions, Next Steps
Handouts
• Slides
• Glossary of Key Terms
• 2011 Criteria
• Independent Review Worksheets
• Share Food – Organizational Profile
• Conflict of Interest/Code of Ethical Conduct
Process Timetable
• Applications Due – July 15• Applications Distributed – July 21• Consensus Review Report Due – August 25• Site Visit Training – August 31• Site Visits Conducted – September 12-30• Site Visit Reports Due – October 7
Process Timetable
• Honorees Selected – by October 28• Feedback Reports Delivered – by November 15• Recognition Event – January 2012
Examiner Standards
• Confidentiality• Conflict of Interest• Code of Ethical Conduct
• Signed Statement Required
New in 2011
• Biennial Cycle• Streamlined Criteria• Shortened Application
• Two-page limit for Organizational Profile• Six-page limit (total) for Categories 1-3
• Applicant Workshop Series• Standards-Based Recognition
Criteria Characteristics
• Organizational Profile• Three Categories – People, Principles, Process• Approach-Deployment-Results (ADR)• “What” Questions• “How” Questions• Non-prescriptive• Results emphasis
Organizational Profile
• Key Characteristics• Programs/services• Purpose, Vision, Mission, Values• Customers and stakeholders• Suppliers/collaborators• Workforce• Accreditation, certification, regulators• Governance system• Major funder/foundation relationships
Organizational Profile
• Organizational “biography”• Accept at face value• Resource for evaluation of criteria responses• Two-pages, text or bullet format• Only source of “biography” information – cannot
go to web site to learn more
Organizational Profile
• Competitive Environment, Challenges• Key competitors• Key competitive success factors• Competitive/comparative data sources• Core competencies• Strategic advantages, challenges• Operational challenges• Human resource challenges
Categories
• Category 1 – People• Leadership, Workforce, Governance, Related Results
• Category 2 – Principles• Customer Focus, Strategic Planning, Organizational
Performance Review, Knowledge Management, Related Results
• Category 3 – Process• Value Creation, Support, Integration of
Suppliers/Collaborators, Related Results
Reviewing Smaller Organizations
• Size does not affect criteria applicability• Size does provide context for expectations• Systems likely to be less formal• Resource capacity may be limited• Have had to make choices due to resource
limitations
Important Terms/Concepts
• “Key”• System/process• Senior leaders• Results – actual data• Outputs and outcomes• Trends
Organizational Profile
• Become familiar with the organization by thoroughly reviewing Profile prior to working on the Category analysis
• Convert/condense to “key” bullet items (1-2 pages) to support analysis
• Applicant’s have been trained in what you need/expect – revise and further condense if necessary
• Share Food example
Category Review
• The applicant is the customer of your work• Selection panel depends exclusively on
examiner team’s work• This is not a compliance assessment – we are
seeking to help them understand where they have strengths to build on and opportunities for improvement
Category Review Process
• Review the criteria and keep them handy for your analysis process
• Use the Glossary to help you understand the criteria requirements
• Read the application completely prior to beginning to detail your comments
• Rely on the Organizational Profile for reference and background
Approach-Deployment-Results
• Approach – methods used to address criteria requirements
• Deployment – the extent to which the approach is implemented
• Results – outputs/outcomes achieved through approach-deployment efforts related to criteria requirements – these are actual data, not anecdotes or subjective statements concerning effectiveness
Category Review Tips
• Focus on criteria requirements
• Look for “action” verb in criteria questions to clarify what is expected
• Keep the key factors from the Organizational Profile in mind – what is important to this applicant
• Review the “small organization” tips
• Avoid generalized assumptions
• Benefit of the doubt
• Applicants want actionable feedback – they need to know what to work on
Worksheets – Analysis Process
Key Process Evidence of systematic approach Evidence of deployment
Deployment of Mission, Vision & Values
Two-way communication
Promoting legal and ethical behavior
Ensuring alignment with community goals
Creating sustainability
Item 1.1 – How do senior leaders lead?
Worksheets – Analysis Process
Key Process Evidence of systematic approach Evidence of deployment
Deployment of Mission, Vision & Values
Discussed at staff meetings, used as decision criteria in management meetings
Published in newsletters and brochures, staff members post in cubicles
Two-way communication
Forums held with employees to seek input, informal walk-around process
Monthly all-staff meetings, employee focus groups, talk one-on-one
Promoting legal and ethical behavior
Values include commitment to ethics, ethical standards documented
Discussed in staff meetings, shared at volunteer orientation
Ensuring alignment with community goals
Management reviews as part of organizational performance review
Discussed in staff meeting and in volunteer orientation
Creating sustainability
Key factors reviewed annually by Board and senior leaders
Leaders attend futurists conference, developing succession strategy
Item 1.1 – How do senior leaders lead?
Worksheets – AD ScoringItem 1.1 – How do senior leaders lead?
0-5% 10-25% 30-45% 50-65% 70-85% 90-100%
Approach Not addressed
No systematic approach, anecdotal
Systematic approach for basic
Systematic approach - all
Mature approach for all
National role model
Deployment No evidence
Early stages
Deployed in most areas
Fully deployed, few gaps
Fully deployed, no gaps
No gaps or weakness, role model
Worksheets – AD ScoringItem 1.1 – How do senior leaders lead?
0-5% 10-25% 30-45% 50-65% 70-85% 90-100%
Approach Not addressed
No systematic approach, anecdotal
Systematic approach for basic
Systematic approach - all
Mature approach for all
National role model
XDeployment No
evidence Early stages
Deployed in most areas
Fully deployed, few gaps
Fully deployed, no gaps
No gaps or weakness, role model
X
Worksheets – Results AnalysisItem 1.5 – People results
Measurement Provided/Not Provided
Levels Trends Comparisons
Leadership effectiveness
Workforce satisfaction, etc.
Fiscal Accountability
Compliance, legal and ethical behavior
Worksheets – Results AnalysisItem 1.5 – People results
Measurement Provided/Not Provided
Levels Trends Comparisons
Leadership effectiveness
360 evaluation results
Overall ranking of 4.2 on a 5 point scale
Increased from 3.8 in 2009
National average at 3.9
Workforce satisfaction, etc.
Employee and volunteer satisfaction
Volunteer satisfaction at 3.1 out of 5
One year’s data only
None provided
Fiscal Accountability
Audit and internal assessment
Zero exceptions on 2010 year-end
Internal review - fewer exceptions than in 2009
None
Compliance, legal and ethical behavior
Times in compliance, legal/ethical breaches
Four significant compliance violations
2009 and 2010 showed zero compliance issues
Similar providers averaged less than 2 issues
Worksheets – Results ScoringItem 1.5 – People results
0-10% 15-35% 40-60% 65-85% 90-100%
Levels No results Some good performance
Good performance for most areas
Good performance all areas
National role model performance
X
Trends No trend data
Little trend data, some adverse
Positive trends in key areas
Strong positive trends
Exemplary performance trends
X
Comparisons None provided
Little provided Some provided
Good comparative performance
Benchmark leadership
X
OverallXX
Consensus Process
• One set of comments for the team• Strengths and opportunities for improvement
(OFI)• Single “score” for the team on each item• Identify site visit issues• Submit feedback report with draft of strengths
and OFIs by August 25
Comments – Feedback Report
• Most important Strengths• Most important OFIs• Criteria-based• Actionable but not prescriptive• Complete thoughts – 2-3 sentences• 4-6 per category for AD items (include both
strengths and OFIs)• 3-5 per results item – 1.5, 2.6, 3.4 (include both
strengths and OFIs)
Comments – Feedback Report
• Refer to “the applicant” but not their name• Avoid prescribing solutions or making subjective
judgments – state the specifics of the strength and/or OFI
• OFIs often need a “so what” or “so that…”• Watch out for contradictions• Don’t comment on application quality or
presentation• Make sure an independent reader can
understand your comment
Comments – Format
• All comments need an item number (1.1, 2.3, 3.4, etc.)
• Bold the most significant comments • Strengths first, OFIs next• Avoid “would,” “could” or “should” • Non-prescriptive – keep objectivity• Professional, positive tone
Developing Comments
• Preliminary comments based on independent review worksheets
• Final, revised comments based on site visit findings
Developing Comments
• Possible comments relate to:• Approach-related strength or OFI• Deployment-related strength or OFI• Gap in responding to criteria requirements (not
apparent that the applicant has a systematic approach to…)
• Potential role model practice – rare but important to highlight – make sure to present in bold type
• Acknowledgment of anecdotal response while lacking a systematic approach
Sample Comments
• Item 1.1 – The applicant’s senior leaders and Board members review the organization’s Vision, Mission and Values statements as part of the annual strategic planning process. Multiple communications methods, such as monthly staff meetings, one-on-one meetings and the monthly newsletter are used on a recurring basis to make sure that all volunteers and staff members understand these three key statements. (Strength)
Sample Comments
• Item 1.4 – The applicant ensures fiscal accountability throughout the organization through a series of periodic internal audits conducted by key staff as well as an annual independent financial audit. Audit results are reviewed within 10 days of receipt by senior leaders, and the Board reviews these results as part of each quarterly meeting. (Strength)
Sample Comments
• Items 3.1 & 3.3 – The applicant reviews its core competencies – Service Delivery and Service Design – during the Strategic Planning process and relies on these competencies to strengthen its 3 key value creation processes – needs assessment, service design and care delivery. These processes are improved annually utilizing input from customers, employees, volunteers and other key stakeholders, as well as data developed from customer feedback instruments. (Strength)
Sample Comments - Results
• Item 1.5 – The applicant’s annual workforce satisfaction survey indicates that overall employee satisfaction (those giving a score of “5” on a scale of 1 to 5) has increased from 41% in 2005 to 68% in 2008. Satisfaction among volunteers, who play a key role in the applicant’s service delivery process, has risen from 75% to 82% (scores of “5”) in that same time frame. (Strength)
Sample Comments - Results
• Item 2.6 – The applicant has reduced dependency on government funding through increased donations and other forms of private support as a source of revenue, addressing a key strategic challenge for the applicant. Private support has risen from 15% of its annual revenue in 2005 to almost 40% in 2008, while growing overall revenue by 30% in that same time period. (Strength)
Sample Comments - Results
• Item 3.4 – The applicant improved its process to report monthly performance to its funding partner, shortening delivery time from 30 days in 2006 to 15 days in 2008, six days less than the funder’s requirement of 21 days. The time made available by this improved process performance has led to the creation of additional capacity for providing service to customers. (Strength)
Sample Comments
• Item 1.3 – While the applicant expresses that its customers are satisfied, and indicates that it has received numerous letters from satisfied customers, it is not clear what systematic approach the applicant uses to determine the actual level of satisfaction of its customers with its services or how the applicant gathers data on customer experiences to develop actionable data on which to base improvement efforts. (OFI)
Sample Comments
• Item 2.4 – While the applicant indicates that it conducts monthly performance reviews, both by senior leaders and with its Board, it is not clear how or if the results of those reviews are shared with those individuals responsible for making decisions in the affected performance areas. (OFI)
Sample Comments
• Item 3.1 – While the applicant identifies its two key value creation processes, Needs Assessment and Case Management, it is not clear that the applicant has a process in place to assess the effectiveness of these processes in order to make improvements and incorporate emerging best practices. Lacking such a review and improvement process, it is not clear how these processes are kept current with evolving customer requirements, a key strategic challenge for the applicant. (OFI)
Sample Comments - Results
• Item 1.5 – While the applicant indicates that its customer satisfaction results have risen by 300% over the past three years (2006 through 2008), lacking actual data on the specific level of performance achieved, it is difficult to determine whether the applicant is achieving its goal of 95% customer satisfaction. (OFI)
Sample Comments - Results
• Item 2.6 – The applicant indicates that a key factor in its ability to compete effectively in its service delivery is minimization of delivery errors, yet its service delivery results actually reflect an increase in percentage of errors each of the past three years (2006 through 2008) from 6% to 8%. Given such a trend, it is not clear how the applicant will be able to ensure its sustainability in a highly competitive market. (OFI)
Sample Comments - Results
• Item 3.4 – The applicant indicates that it is critical for real-time data to be available at all times for effective service delivery in meeting customer needs, yet its system uptime remains at a level of only 85% in 2010. While this performance level has increased from 75% in 2009, it is not clear that the applicant can meet its customer requirements if data access is not available 15% of the time. (OFI)
Comment Improvement
• The applicant’s strategic planning process is highly effective due to the use of an outside consultant with much expertise.
Comment Improvement
• Item 1.2 – The applicant has excellent two-way communication due to its open-door policy and frequent e-mails from the leaders to the staff.
Comment Improvement
• Item 2.2 - It would be easier to assess the applicant’s approach to building relationships with customers if they would include more information about what they actually do in this area.
Comment Improvement
• Item 2.3 – The applicant does not include the results of its SWOT analysis making it difficult to assess the relevance and accuracy of its strategic objectives and goals. Without this information I cannot tell how the applicant will be able to improve its sustainability.
Comment Improvement
• Item 3.1 – The applicant could do a better job of improving its value creation processes if it would just take the time to map the processes and measure the performance of the process at each key step along the way.
Comment Improvement
• Item 3.1 – The applicant’s approach to managing its support processes is clearly a best practice that others should copy.
Comment Improvement
• Item 2.4 – The applicant would benefit from using a balanced scorecard approach as the over-arching framework for its performance management system. In doing so, the applicant would create better alignment of effort and produce greater impact. This would lead to greater likelihood of funding support increases, positioning the applicant for enhanced levels of sustainability.
Consensus Process - Review
• One set of comments for the team – seek to synergize comments, not just compile/combine
• Strengths and opportunities for improvement (OFI)
• Single “score” for the team on each item• Identify site visit issues• Submit single consensus feedback report with
draft of strengths and OFIs by August 25
Site Visit Process - Overview
• Review comments and identify site visit issues• Key site visit issues only• Verify and clarify on site• Review approach, deployment and results• Revise feedback report based on site visit
findings – new strengths and OFIs will emerge• Training for examiners and applicants on August
31