Charity Navigator CN 2.0 Case Study

Presentation at Managing to Outcomes Forum

Paul Brest, President, The William and Flora Hewlett Foundation With Ken Berger, President & CEO, Charity Navigator,

David Bonbright, CEO, Keystone Accountability, Xandy Brown, Pilot Project Coordinator, Charity Navigator,

And Professor David Campbell, SUNY Binghamton June 13, 2011

The Data Proves Impact

 Estimated 3.3 million distinct visitors per year (~5 million hits)

 92% say evaluations affected their decision to support individual public charities

 CN ratings influence decisions on billions in donations annually

Financial Health

Accountability/Transparency

Results More Comprehensive Rating System

CN 2.0

Note: Strategic Plan is in draft and not yet finalized.

•  Why 10,000? Because ~70% of all revenues that come into the NP sector annually go to them.

•  How to scale this effort? A new process to move beyond our staff and involve consumers in the rating of charities in cause areas they’re passionate about.

•  now… 2.0…

10,000 CHARITIES RATED BY VOLUNTEER RATERS

Reporting Results R&D Process

What is the charity’s commitment to reporting results?

•  Clear commitment to reporting results stated. •  Specified time period for which results are

presented. •  Some mission-related results in current period

compared to earlier period. •  Reporting distinguishes between activities,

outputs and outcomes. •  Credible intention to validate results evidence.

How does the charity demonstrate the demand for its services?

•  Reports indicating the aggregate numbers of individuals accessing the charity’s outputs in a given period.

•  Evidence of demand for more than half of the charity’s outputs is provided.

•  A statement of evidenced demand set against the larger demand that the charity does not presently have the capacity to reach.

Does the charity report its activities and outputs?

•  A clear description of the majority of the charity’s mission-related activities and outputs for a defined reporting period

•  A significant portion of mission-related activities and outputs reported are compared to an earlier reporting period

•  Activities and outputs reported can readily be related to reported expenditure

Does the charity report its outcomes (medium and longer-term results)?

•  A clear description of mission-related outcomes achieved by the charity in the reporting period.

•  Outcomes are described for more than half of the charity’s mission-related activities or those mission-related outcomes described relate to activities that consume more than half of total charity’s expenditure.

What is the quality of evidence for reported results?

•  Evidence of most mission-related outputs includes some element of independent validation

•  Low level of outcome evidence •  Medium level of outcome evidence •  High level of outcome evidence:

beneficiary feedback •  High level of outcome evidence:

independent validation

Does the charity adjust and improve in light of its results?

•  Evidence that the organization assesses its mission-related performance in light of what was planned for the reporting period

•  Admits mistakes and publicizes corrective actions •  Admits mistakes, publicizes corrective actions and

commits itself to validating proposed corrective measures through dialogue with those affected

•  Admits mistakes, publicizes corrective actions, and provides evidence that it has in fact validated proposed corrective measures through dialogue with those affected

Framing Questions •  Given the low level of current reporting of

results, what are the best initial rating criteria? •  What is the best way to communicate and

implement rating criteria that will ratchet up over time?

•  What initial criteria will be (a) possible to rate reliably by volunteers and (b) sufficiently achievable to have a basic level of compliance within 12 months, but (c) not be so easy as to be credible and widely gamed.

•  Who else needs to be on the bus?

What do student raters learn? •  More questions raised than answered:

– How to differentiate activities, outputs, and outcomes when they are not explicitly labeled?

– How does a charity report results at this level and still have a website that is accessible and clear to lay people?

– How to decrease subjectivity?

How was student feedback incorporated? •  Materials

– Online platform – Centralized communication

•  Rating –  Illustrative examples of where to find

information – 4 level confidence scale

•  Process – Thanking volunteers – Providing background

Binghamton University Courses •  Two courses

– Public Administration •  16 Students •  Issues in Nonprofit Administration

– Social Work •  Advanced Social Work Practice with Communities •  19 Students

•  Unique Features – Local Organization Assessment – $3,000 in “Philanthropy Incubator” grants – Student Blog (“Navigating Southern Tier

Charity”)

Course Structure •  Public Administration

–  Course Focus: Effectiveness and Accountability –  CN Recommended Readings, Forces for Good

•  Social Work –  Forces for Good

•  Inter-disciplinary Dimensions –  Charity Navigator Assessment Teams –  Local Organization Assessment Teams –  Class Discussion

CN: Students’ Practical Issues •  Assessing Confidence Levels

•  Positive Response to 5 Minute Rule

•  Most Diligent: Assessments Time Consuming

•  Importance of Preparation, Knowledge (SW vs. PA)

•  Technology Challenges (CN Responsiveness)

CN: Students’ Philosophical Issues

•  Standards Incomplete –  Mistrust –  “Stars” system simplistic. –  Good organizations negatively affected.

•  Resource Limitations Constrain Performance •  Assessing Transparency, not Performance

–  Performance data depend on trust (not verifiable). •  Discomfort with “Admitting Mistakes” •  Donor vs. Beneficiary Focus

Standards and Students’ Funding Decisions

•  Real-world Challenge of Assessment •  Charity Navigator Standards OK After All…

– Financial Measures – Transparency

•  Role of Executive Compensation •  Frustrations with Limitations of Secondary Data

– Assessment Incomplete •  Creation of New Standards

– Role of Social Media – Student Concerns (small vs. large, effect of

contribution)

End of Semester CN Rating Attitudes

•  The perfect not the enemy of the good.

•  Value of standards •  Not all performance standards apply. •  Create standards for stakeholder

dialogue •  SW students: Limited utility for

beneficiaries

End Semester Performance Assessment Attitudes

•  Effectiveness a social construct •  Negotiate standards with

stakeholders. •  More questions than answers; “it

depends” •  Importance of responsiveness to

performance perceptions.