developing a system for evaluating small microcredit programs interim research report january 10,...
TRANSCRIPT
Developing a System for Evaluating Small Microcredit Programs
Interim Research ReportJanuary 10, 2012
York University Community Finance Project (YUCFP) in partnership with
ACCESS Community Loan Fund
Brenda Spotton Visano, PhD and Neil Britto
2
Overview
Topic1. Project Objective…………………………………………………
2. Process and Outcomes to Date……………………………
3. Goals of Evaluation and Program Evaluation Commitments of ACCESS.......................................
4. Comparison of Industry Options ………………….…….
5. The ACCESS Alternative: Proposed Program Evaluation Indicators…….........................................
6. The Flow of Social Performance Information……….
Slides#3#4
#5-7
#8-11
#12-17
#18
10 January 2012
3
Project Objective
• Develop a system of self-evaluation for small volunteer microcredit programs in Canada that – Has comprehensive coverage options spanning an array of
performance objectives– Offers an evaluation of the program’s effectiveness,
efficiency, impact, and social performance– Is informed by program evaluation industry standards– Offers an independent and credible set of metrics for
volunteer organizations to use when reporting their performance
– Is low cost, flexible, accessible, and easy to use
10 January 2012
4
Process and Outcomes to DateWhen What Who
Summer 2010 Review existing microcredit program evaluation instruments and services YUCFP
Fall 2010 - Winter 2011 Review existing literature on Social Performance Indicators YUCFP
Summer 2011Assess advantages and disadvantages of existing program evaluation instruments and services in relation to Social Performance Literature
YUCFP
Summer 2011Proposal to Pilot the Development of a System to Evaluate Small Microcredit Programs in Partnership with ACCESS Community Capital Fund
YUCFP, ACCESS
Fall 2011 Review evaluation commitments and representations of ACCESSIdentify/clarify program evaluation objectives ACCESS , YUCFP
Fall 2011 Identify Indicators to be included ACCESS, YUCFP
Winter 2012 Interim Research ReportDraft Program Evaluation Data Report YUCFP, ACCESS
Winter 2012 Complete Survey Instruments for Data Collection and Pilot System for Data Management YUCFP, ACCESS
10 January 2012
5
Goals of Program Evaluation
• Effectiveness
– for investigating whether a program has demonstrable effects on specific targeted outputs and activities.
– Illustrative Question: “Is the program achieving the goals and objectives it was intended to accomplish?”
• Efficiency– for assessing the cost-effectiveness of the program. – Illustrative Questions: “Are the program’s activities being produced with appropriate use of
resources such as budget and staff time?”
• Policies and Practices– For assessing the integrity and effectiveness of program policies and practices. – Illustrative Question: “How well does the program manage client information?”
• Impact– for investigating any impacts beyond the immediate outputs and activities. – Illustrative Question “Do clients experience changes in personal income through microloans?”
10 January 2012
6
ACCESS’ Evaluation Commitments and Representations Trillium Grant
Indicator SourceProvided to Board
Provided to
Trillium
Loan Portfolio InformationVolume of loans granted by each Chapter Portfolio report Monthly QuarterlyQuality of loans at each chapter Portfolio report Monthly Quarterly
Other Financial DataAdditions to Guarantee Fund Financial statements Monthly QuarterlyDonations received Financial statements Monthly Quarterly
Other DataChapters formed Loan Officer report Monthly QuarterlyChapter Agreements executed Loan Officer report Monthly Quarterly
Chapters in dialogue with Access Loan Officer report Monthly QuarterlyVolunteers trained in lending, fundraising, other as required Loan Officer report Monthly QuarterlyTrainers identified at each chapter Loan Officer report Monthly QuarterlyFund raising (investors and donors) formulated Loan Officer report Monthly QuarterlyCalls made on potential investors and donors Loan Officer report Monthly QuarterlyFundraising events held Loan Officer report Monthly QuarterlyArticles published, other media exposure Loan Officer report Monthly Quarterly
10 January 2012
7
ACCESS’ Evaluation Commitments and Representations Presented by Alex Kjorven to Future Fund (Trillium) Evaluation Workshop July 19, 2010
as posted http://futurefundevaluation.wordpress.com/projects-evaluations/
Evaluation metrics currently under consideration to evaluate the effectiveness of loans disbursed and the social impact of our lending program include:
1.Number of new businesses created or new lines of business initiated2.Number of referrals for mentorship services (internally/externally)3.Number of new employees hired by the business since receipt of loan4.Financial metrics such as profits and total spending5.Improvement to lenders credit rating as a reflection of their improved ability to access capital in the future
Indicator Current Goal
Number of loans disbursed 15 30
Number of open houses N/A 3
Number of articles published N/A 2
Number of chapters formed and operating
1 3
Number of active volunteers 35 50
Amount of guarantee fund $220,000 $300,000
Amount of donations $3,000 $10,000
10 January 2012
8
Comparison of Industry Options – Operation
CERISE’s SPI Tool
Planet Rating’s Social Rating Microfinanza Microrate
Impact Reporting & Investment Standards (IRIS)
Cost to Acquire as @ August 2011
$1,0000 - $4,000 w/ the
use of a supporting analyst.
$4,000 - $9,000 $17,000 to $20,000 $5,000 - $10,000Freely available on
open access website
Number of Consultants Required/Provided
1 Provided 1 Required 2 to 3 Required
Unknown (Consultants who provide financial
rating, also provide social rating)
None
Timeline: On site consultant visit
1 to 2 days 4 to 5 days 8 to 10 days 4 to 5 days Not applicable
Timeline: Draft report
N/A1 to 2 months after
on-site visit1 to 2 months after on-
site visit2 to 4 weeks after
on-site visitNot applicable
Timeline: Final report
1 to 5 months after on-site visit
1 to 3.5 months after on-site visit
2 to 3 months after on-site visit
2 to 5 months after on-site visit
Automatically generated
10 January 2012
9
Profile: Planet Rating: Social Rating Product
Planet Rating, headquartered in Paris, France, is a specialized microfinance rating agency offering evaluation and rating services to microfinance institutions (MFIs), using the GIRAFE and the Social Performance methodologies. To date, Planet Rating has conducted over 600 missions in more than 75 countries and continues to expand its international presence. Planet Rating has been recognized by the Rating Fund I, the Rating Initiative, the Moroccan APP and the Latin American Rating Fund II.
-From their website: http://www.planetrating.com/EN/
10 January 2012
Relevant for small, volunteer microloan programs in Canada?
NO… Costs are prohibitive
….designed for large formal microfinance organizations…benchmark measures are unavailable for Canada
10
Profile: MicrofinanzaSocial Rating Product
MicroFinanza Rating is a leading microfinance rating agency. Since 2000 we strive to produce independent and qualitative information that promotes responsible flow of investment towards the microfinance industry. This is achieved through provision of a comprehensive package of institutional and social performance assessment services to microfinance institutions (MFIs), investors, regulators, donors and networks. MicroFinanza Rating has performed about 600 assessments so far in over 60 countries worldwide. - From their website: http://www.microfinanzarating.com/
10 January 2012
Relevant for small, volunteer microloan programs in Canada?
NO… Costs are prohibitive
….designed for large formal microfinance organizations
11
Profile: Impact Reporting and Investment Standards (IRIS)
Financial Service Sector Supplement
Impact Reporting & Investment Standards, or IRIS, is a common language for describing the social and environmental performance of an organization. IRIS provides an independent and credible set of metrics for organizations to use when reporting their impact. IRIS indicators span an array of performance objectives and include specialized metrics for a range of sectors including financial services, agriculture, and energy. Like financial accounting standards, IRIS provides a basis for performance reporting and organizations need only report on applicable metrics from the IRIS library
-From their website: http://iris.thegiin.org/
10 January 2012
Relevant for small, volunteer microloan programs in Canada?
NO… no scope to assess volunteer and capacity building contributions
….designed for large formal microfinance organizations…bases of comparison irrelevant for Canadian context
12
The ACCESS Alternative: Performance Indicators, Reporting Format and Information Management
► The proposed social performance assessment includes assessments of both efficiency and effectiveness. ► The proposed measures express greater consideration for the small size of Canadian microcredit
organizations. ► The social performance assessment requires significantly less resources to complete. ► The proposed indicators avoid using any summary or composite score in the final report. ► The proposed assessment rebalances and refocuses the set of indicators to better reflect an urban setting. ► The proposed measures are intended for use in the Canadian context. ► The proposed social performance assessment is designed to produce information valuable to internal
stakeholders as well as external audiences. ► The provision of supports to help manage social performance information is unique to the proposed
assessment.
10 January 2012
13
Proposed Indicators - Effectiveness
10 January 2012
Category Indicator Measure
Client Relations - Client Targeting
Whether the loan portfolio indicates success of the organization's mission to target specific groups
Whether the groups mentioned in mission statement correspond to groups in the loan portfolio: OS11 corresponds to OS13
Client Relations - Client Targeting
Whether the organization successful in targeting the groups that its mission statement suggests.
Whether the groups targeted in the organization's outreach efforts are the groups represented in the loan portfolio over the last 12 months. (a) Yes(b) No
Operational Performance - Loan Delivery Application success rate The total number of loans granted divided by the total number of
microloan applications this calendar year.
Operational Performance - Loan Delivery Effective delivery of loan product The number of microloans provided this calendar year.
Social and Client Impact Gender diversity of loan recipients The number of female clients as % of total during the reporting period.
Social and Client Impact Client Diversity The number of clients that identify as: (a) Caucasian, (b) visible minority, (c ) aboriginal, (d) differently abled, (e ) other
Social and Client Impact Personal changes identified by the ClientThe personal changes that clients identify: (a) increase in business skills, (b) increase in personal income, (c ) neither, (d) increase both personal income and business skills, (e) other
14
Proposed Indicators – Effectiveness (Cont’d)
10 January 2012
Category Indicator Measure
Social and Client Impact Change in business performance since the microloan
The approximate change in business (a) revenues since the client received the microloan, (b ) profit since receiving the microloan; > $10K; < $10K
Social and Client Impact Contribution to financial inclusion
The number of loan recipients who progress to (a) new bank account, (b) a registered account, (c ) a credit card, (d) mutual fund products, (e) tax free savings account, (f ) additional business loans, (g ) line of credit (h) Other (please list below)
Social and Client Impact Contribution to financial inclusionThe number of loan recipients in the current loan portfolio that identify as (a) female, (b) aboriginal, (c) Caucasian, (d) visible minority), (e) differently abled, (f) low income, (g) poor credit history
Social and Client Impact Contribution to personal skills development The number of mentorship hours provided to clients
Social and Client Impact Community connections The number of partnerships with other organizations and the number purchases from local businesses.
Social and Client Impact Gender diversity of the organization The number of females as % of total who are board volunteers, non board volunteers or paid staff.
Social and Client Impact Ethnic diversity of the organization The number of paid staff, board volunteers and non-board volunteers that identify as: (a) a visible minority, (b) caucasian, (c ) aboriginal
Social and Client Impact Skills improvement for organizational members
The number of individuals who identify skill development in: (a) Time Management, (b) Money Management, (c ) Human Resource Management, (d ) Information Organization, Interpretation and Maintenance, (e ) Team Skills, (f ) Customer Service, (g ) Leadership, (h ) Cultural Sensitivities, (i) Other
15
Proposed Indicators - Efficiency
10 January 2012
Category Indicator Measure
Operational Performance - Loan Servicing
The total cost of providing skills development and mentorship to microloan recipients
Allocated salary of paid staff to loan delivery + (number of board volunteer hours x minimum wage)
Operational Performance - Loan Delivery Efficiency of pre-loan activities
The number of non-board volunteer hours needed to deliver one microloan: (total average monthly hours for pre loan activities + total average monthly hours for post-loan activities) / average number of new loans granted per month.
Operational Performance - Loan Delivery Cost of pre-loan delivery Allocated salary of paid staff to loan delivery + (number of non-board
volunteer hours on loan delivery activities x minimum wage)
Operational Performance - Loan Servicing Efficiency of post-loan activities
The number of paid staff hours needed to deliver one microloan: (total average monthly hours for pre loan activities + total average monthly hours for post-loan activities) / average number of new loans granted per month.
Operational Performance - Loan Servicing Cost of post- loan servicing Allocated salary of paid staff to loan servicing + (number of non-board
volunteer hours on loan delivery x minimum wage)
Operational Performance - Loan Servicing
Resource requirements to support the activities of one microloan
The total time of 'other activities': (a) operations/management, (b) outreach committee meetings, ( c) steering committee meetings, (d) promotional efforts and (e ) other activities, divided by the average number of new loans granted per month.
16
Proposed Indicators – Policies and Practices
10 January 2012
Category Indicator Measure
Client Relations - Client Communication
Consideration for client communication
Are clients understanding their rights, obligations and product terms when (a) submitting the loan application, (b) signing the loan agreement?
Client Relations - Client Participation Opportunities for client participation
Whether policies and/or current practices exist that enable client participation in the organization, in addition to being a loan recipient (a) Yes, (b) No
Client Relations - Client Targeting
Consideration for targeting specific client groups
Whether the organizations outreach efforts target (a) women, (b) aboriginal, (c ) caucasian, (d) visible minorities, (e ) differently abled, (f ) low income, (g ) poor credit history (h ) other priority groups
Client Relations - Communications Consideration for client feedback Whether the loan program have a process in place to allow clients to
provide feedback. (a) Yes, (b) No
Employee Relations - Employment Terms Provision of non-wage benefits
Whether the organization provides its employees and volunteers with non-wage benefits. If so, please indicate (a) long term employment, (b) health care benefits and (c ) training opportunities
Employee Relations - Performance Appraisal
Consideration for social objectives as part of staff and volunteer appraisal
Whether the MFI incorporate the fulfillment of social objectives into performance appraisal of staff and volunteers
1710 January 2012
Proposed Indicators – Policies and Practices (Cont’d)Category Indicator Measure
Employee Relations - Training
Consideration for social performance training
Whether the organization provide social performance training If so, does it provide training on (a) Payment Collection Practices, (b) Responding to Client Needs, (c ) Social Performance Reporting, (d ) Client Communication, (e ) Gender Sensitivity, (f ) Safeguarding Client Data, (g ) Over Indebtedness Prevention. Also, who is the training provided to: (a) Non-board member volunteers, (b) Paid staff, (c) Board Member Volunteers
Employee Relations - Workplace Diversity
Consideration for workplace diversity
Whether the organization has specific policies to ensure diversity in the workplace: (a) Yes, (b) No
Governance - Environment
Demonstrated concern of environmental impact and influence of the organization
A demonstration of environmental concern. (a) in the form of policy statements addressing environmental impact, (b) tracking of environmental impacts, (c ) other
Governance - Ethical Conduct
Existence of formal policies on ethical conduct
Whether the organization has a formal code of ethical conduct for (a) volunteers, (b) employees, (c ) clients
Governance - Social Performance
Formal policies addressing governance activities.
Whether the MFI has formal policies for its governance activities. This could include (a) regular Board of Directors meetings with at least three Board members, (b) at least one Female member of the board, (c ) statements to address potential conflicts of interest between members of the organization
Operational Performance - Loan Delivery
Flexibility of microloan productsThe range of products provided by the organization: (a) Business start-up loans, (b) Business Loans > $5,000, (c ) Training Loans, (d) Line of Credit, (e) Other (please list below)
Employee Relations - Workplace Diversity
Alignment between operational staff and the communities the organization serves.
Whether board members, volunteers and staff have residence or work in the neighborhoods where the organization provides services
18
The Flow of Social Performance Information
Step One: Selection of IndicatorsSelected social performance indicators are represented
in the three surveys in the form of a question.
Step Two: Survey CompletionThe respective groups submit completed surveys to the
individual managing the social performance report
Step Three: Populating the SpreadsheetSurvey data are input into the spreadsheet template.
The spreadsheet populates the key performance indicators from the survey data
Step Four: Report CompletionThe social performance indicators from the spreadsheet
template provide the foundation of the social performance report
Sequence of Tasks
Client Survey (Completed by
all clients)
Organizational Survey (Completed by one
director)
Non-Board Volunteer and Staff Survey(Completed by all
volunteers and staff)
Spreadsheet Template (Aggregates all survey information. Allows for periodic reporting
and storage of historical social performance information )
Social Performance Indicators(Effectiveness, Efficiency and Policy / Practice Indicators)
Tool Used
Social Performance Report(Completed report includes an indicator scorecard, discussion on
historical social performance of the organization and provides recommendations for potential improvements)
10 January 2012