usaf capacity building- monitoring and evaluation
DESCRIPTION
USAF expert Parvex Iftikar discusses provides a detailed look at implementing monitoring and evaluation plans for universal service and access funds and their projects.TRANSCRIPT
MONITORING & EVALUATION
for
UNIVERSAL SERVICE
PROGRAMS/PROJECTS
Introduction
• Personal – Former CEO USF Pakistan – First employee - built USF Pakistan
and ran it successfully for nearly 5 years – Now a consultant in several countries of Asia and Africa
• This Webinar – NOT about M&E itself – Rather about M&E in USOF Projects & Programs! – After an introduction to the concept of M&E, “setting the stage”:
how to set up M&E in a Universal Service Organisation
Agenda
• Part-1 – Session 1- Setting the stage – Session 2- Monitoring – Session 3- Evaluation
• Part-2 – Session 1- Organizational Capacity Assessment – Session 2- Sourcing Options – Session 3- In-House M&E – Session 4- Outsourced M&E – Session 5- Procurement – if outsourced – Session 6- Worked Example for Bid Evaluation
SETTING THE STAGE Part 1, Session 1
Project life
Focus of Monitoring
Focus of Evaluation
Goals Objectives Outputs Inputs Outcomes Impacts
Definitions
INDICATOR - a metric for measurement of inputs,
processes, outputs, outcomes, & impacts for
projects, programs, or strategies.
INPUTS - the main resources required to
undertake activities & to produce outputs.
OUTPUTS - physical &/or tangible goods / services delivered by the project.
Describes scope of the project
OUTCOME - what the project intends to accomplish by the end of it’s implementation.
IMPACT - result of the project, accrued in medium to long
term.
MONITORING Part1, Session 2
PLAN
Planning Phase Implementation Phase
PROGRESS
Monitoring is an ongoing process by which
stakeholders obtain regular feedback on the progress
being made towards achievement of their planned
objectives.
Are we taking the actions we said we would take?
Are we making progress on achieving the results that we said we wanted to achieve?
Monitoring should be able to respond to 2 questions
…all this is measured thru Indicators.
Examples of Indicators
– Funds spent in each region divided politically, geographically, etc
– No of contracts awarded
– Subsidy amount disbursed
– % of operational expenditures against contribution collected
Org
leve
l Pr
ojec
t lev
el – Telephone lines per 100 persons
– Computer usage rate in intervened area
– Mobile subscribers per 100 persons (segregated geographically)
– Internet availability per 100 persons (based on technology)
– Volume of local content developed
What to Monitor?
Quality
Risk
Scope
Contracts
Schedules
Human Resources
Other factors
Who Monitors?
Monitoring Tools
Project
• Periodic Reports
• Computer based project management tools.
• Network performance reports
e.g: network usage report, billing info,
QoS report, call success ratios, etc.
• GIS Maps
• Tech. Parameters validation through drive
test & other tech. metrics
Basic Design of Monitoring Framework
What is to be measured –indicators?
How it will be measured & recorded?
Who will measure the information?
How frequently info
will be measured &
reported?
EVALUATION Part 1, Session 3
What is Evaluation?
EVALUATION - a thorough & independent
assessment of activities to determine the extent to
which they are achieving stated objectives.
It is a periodic, objective assessment of an
ongoing or completed project, program, or
policy
Parameters of Evaluation
It relates to specific
questions regarding
design, implementation
and results.
It is carried out at discrete
points in time, often
seeking perspective of
outside experts.
Types of Evaluation – broadly
Program Performance Evaluation
addresses issues like
• relevance to overall goal of org.
• efficiency in delivery of inputs,
• effectiveness of approach,
• timeliness of approach
• other questions regarding project/program design, management & operational decision making
Outcome/Impact Evaluation
• investigates nature of relationship between planned inputs & outputs, & their outcomes
• assesses, what difference (impact) the intervention made
• and how much of the impact is attributable to the Program/Project.
When Evaluations are conducted?
Program / Project
for���long-term���Impact���Analysis
Post Evaluation
Interim Evaluations
Baseline Evaluation
Start Later ? End
When is the Post Evaluation conducted?
Whenever sponsors want to address question like:
– In case of a Pilot project: What would be potential development contributions of an innovative new program (before it is started)?
– In case of a successful small-scale Program: To what extent & under what circumstances could it be replicated on a larger scale or with different population groups?
– In case of a completed Program: What has been the contribution of the intervention (made thru the Program)?
How to conduct Impact Evaluation?
Start Time T 0 Close Time T C
Project Ben
eficiary
Evalua8on E 0 Baseline
-‐ Evalua8on E C Post Evalua3on
Evalua8on could cover • Demographic • Socio -‐ Economic • Technology • Usage pa@ern • Other relevant indicators
Impact= Ec – Eo
Approach -‐ 1
Also called “Before & After method”
How to conduct Impact Evaluation?
Start Time T 0 Close Time T C
Project
Ben
eficiary
Project
Non
-‐Ben
eficiary
Evalua8o
n Ew
ith
-‐
Impact= Ew
ith -‐ Ew
ithou
t
Evalua8on could cover • Demographic • Socio -‐ Economic • Technology • Usage pa@ern • Other relevant indicators
Approa
ch -‐ 2
Evalua8o
n Ew
ithou
t Evalua8o
n Ew
ithou
t
Also called “With & Without method”
Evaluation Tools
Project
• Surveys
• Beneficiary Assessment
• Econometric Analysis- Regression
• Case Studies
• Cost Benefit Analysis
• Cost Effectiveness Analysis
Info. required for Impact Evaluation (example Telecenter)
General information about area
Mapping of public & private facilities (schools, hospitals, post offices, …)
Demographic Info. (No. of persons - age group & gender segregated)
Number of persons who can access computers (gender-wise)
Cost per use of computer, PCO, & other services Persons using computer for educational purpose, health, others
Persons using computer for web based earning,
Average distance & time from nearest computer center
Other uses of computers & ICTs
Number of computers per 100 population
Number of persons who can access computer and internet
Number of persons who know how to use computer
Info. required for Impact Evaluation (example Telecenters)
Information on Purpose of using computers Info on Internet & Computers for learning Info on status of E-commerce Info on status of E-Health Info on status of E-governance Info on status of E-Agriculture Info on use of computer in social networking
Info on economic activities associated with internet & computer
Info on status of networking
Info on presence of local content in cyber space
Info on financial sustainability
Info on community participation in management of center
PART-2
ORGANIZATIONAL CAPACITY ASSESSMENT TEST – One of the ways!
Part 2, Session 1
Comparison of 3 scenarios for M&E implementation
Scenario 1
Sufficient capacity exists within the org.
• Independent M&E unit
• Capacity to execute M&E
• M&E plan is available
• Necessary staff available
• Necessary hardware & software are available
• Office-space is available
Scenario 2
Capacity exists but not sufficient
• Capacity to execute M&E but with gaps
• M&E plan not developed
• M&E team deficient in terms of HR
• Deficiencies in hardware & software
Scenario 3
Substantial Capacity gaps
• No M&E set up of any kind
• No dedicated human resource
• No infrastructure
OCAT*- Sample Test
Question-1 Score What is the organizational structure in context of M&E?
Response Scores A separate M&E Unit with clear reporting lines existing, indicate a strong M&E setup viable for in-house M&E
2
A dedicated M&E human resource with no clear reporting line implies a weak M&E system within an organization
1
No M&E system within an organization structure 0
* Organizational Capacity Assessment Test
(… cont.) OCAT- Sample Test
Question-2 Scores What is the current M&E setup - human, financial, and technical resources?
2.1 Human resources 2.2 Budget (financial allocation) for M&E 2.3 Equipment available for M&E
Response Scores Sufficient 2 Needs Improvement 1 Non-existent 0
(….. Cont.) OCAT- Sample Test
Question-3 Score What is the rationale for creating M&E within the organization (Management’s perspective)?
Response Scores Clearly defined objectives, and policy exists regarding M&E 2 Management aware about M&E’s importance but no written policy or objectives exist
1
No information available 0
(….. Cont.) OCAT- Sample Test
Question-4 Scores What skills exist within the organization in the following areas;
4.1 Domain Knowledge of Monitoring & Evaluation 4.2 Experience of working and demonstrated capability of managing M&E
4.3 Designing and conducting research/surveys 4.4 Data collection, management, and reporting 4.5 Statistical Analysis 4.6 Knowledge about econometrics 4.7 Using research findings for decision making through recommendations
4.8 Project management 4.9 Development of computer based M&E system
Response Scores Sufficient 2 Needs Improvement 1 Non-existent 0
(….. cont) OCAT- Sample Test
• Scoring Recommendations
– Maximum attainable score = 28
– Score above 90% indicates sufficient capacity for In-house M&E
– Score between 60-89% indicates capacity, which need to be built
– Score less than 60% indicates weak capacity, where Outsourced M&E is the most plausible option
SOURCING OPTIONS Part 2, Session 2
M&E Sourcing Options
OCAT Results
Sufficient Cap. In-house
Insufficient Cap.
In-house with Cap. Building
Cap. Gaps Out-source
Steps for M&E implementation in the 3 scenarios
Scenario 1 Scenario 2 Scenario 3
Implement M&E in-
house if not already
being done
Implement M&E in-house, but: • Acquire deficient HR • Develop M&E team • Develop capacity of team thru
professional training • Procure necessary HW & SW • Develop M&E Plan • Execute M&E Plan • Outsource functions which are
not feasible or are occasional
Outsource M&E by
engaging a professional
firm
How to decide?
Build In-House Outsource
Possible reasons of selecting one over the other
• Cost - less expensive to outsource (eg: there isn’t lot of work)
• Lack of in-house capacity too big
• Outside Providers’ research & specialized know-how exceeds by far
• External validation is required
• To keep in-house manpower small
• A legal requirement?
• Notion that third-party M&E is considered more “neutral”
• Cost - may be less expensive in long run
• Required expertise is available
• Want direct control over the process
• Desire of better quality
• Flexibility in managing system configuration
• Reliable & competent service providers not available
• Routine nature of task
In-house Out-sourced
IN-HOUSE M&E Part 2, Session 3
Structure of a good M&E Unit - Functions
Manager
Survey
Enumerators
Monitoring
Engineers Field Coordinators
Database Coordinators
Evaluation
Data Analyst
Reporting Sociologist/ Economist
HR Requirements for In-house M&E (pg 1 of 2)
S. No Designation Experience Qualification No.
1 Manager
10+ years in Project Management + Monitoring & Evaluation of similar projects in development sector
Masters Level Degree — Management, Social Sciences, & Engineering
1
2 M o n i t o r i n g Specialist 5+ years in Monitoring
Masters Level Degree — Management, and Social Sciences 1
3 E v a l u a t i o n Specialist
5+ years of conducting Evaluation
Masters Level Degree — Social Sciences 1
4 S u r v e y Specialist
5 + years in conducting surveys
Masters Level Degree — Management, and Social Sciences 1
5 R e p o r t i n g Specialist
(5+ years in Report-writing
Masters Level Degree — Social Sciences (economics/sociology or other relevant disciplines)
1
Pg 2 of 2) HR Requirements for In-house M&E S. No Designation Experience Minimum qualification Nos.
6 Soc io log is t / Economist
5+ years of in capturing socio-economic aspects of development interventions
Masters Level Degree — Social Sciences (economics/sociology or other relevant disciplines)
1
7 Engineers 3+ years Bachelors Level — Engineering Telecom
Workload dependent
8 Data Analyst 3+ years in conducting data analysis & interpretation of data
Masters Level Degree -- Management or Statistics 1
9 D a t a b a s e Coordinator
5+ years in developing & managing databases
Masters Level Degree — Computer Sciences, Database Management
1
10 F i e l d coordinators
3+ years in field coordination
Bachelors Level Degree -- Sociology + other relevant disciplines
Workload dependent
11 Enumerators 3+ years in enumerating surveys
Bachelors Level Degree -- Sociology + other relevant disciplines
Workload dependent
Material Requirements for In-house M&E
S.No Item Quantities
1 Hardware
2 Software
3 Drive Test Equipment
4 Communication Equipment
5 Furniture
6 Vehicles
7 Office
8 Misc.
OUTSOURCED M&E Part 2, Session 4
Selection Process to outsource M&E
Invitation to prequalify
Receipt of applications from Firms
Evaluation of applications
Shortlist of the pre-qualified
RfP to the short-listed
Evaluation of proposals
Selection of the Out-sourcing Firm!
Task List for the Consultant/Firm
• Develop a detailed configuration of M&E unit incl: identification of HR, job descriptions, competency levels & organization structure
• Develop M&E Plan to address: What will be done? How it will be done? Who will do it? When it will be done? How much resources are required?
• Build capacity of the staff for M&E;
• Develop computer based M&E reporting system, along with user manuals & documentation;
• Develop M&E Operational Manual for the staff;
• Install hardware & software for M&E & train operators;
• Train __ persons to conduct/supervise data-collection process;
• Design & conduct baseline surveys in project areas;
• Develop impact assessment methodology and & conduct impact assessment;
(.. Cont) Task List for the Consultant/Firm
• Identify problems with M&E system & come up with solutions to those problems;
• Develop transition plan to support USF in taking over M&E
• Provide refresher training on M&E to USF staff;
• Oversee design & conduct of various M&E activities by USF;
• Review the results of studies undertaken & propose improvements;
• Assist in data analysis & interpretation;
• Review the system & propose amendments if required;
• Provide technical support for maintaining computer based M&E system.
• Ensure that staff is receiving adequate support to implement M&E functions & that data collection & analysis is on schedule
If out-sourced, some HR would still be required
At least with 5 basic skills:
1) Project Management
2) Contract Management
3) Financial Controlling
4) Technical Monitoring
5) Socio-Economic Monitoring
Min HR required to monitor Out-sourced M&E
S. No Designation Experience Min. qualification Nos.
1. M a n a g e r M&E
5+ years in project management
Masters level —Management, Social Sciences, or Engineering
1
2. Sociologist/ Economist
3+ years in capturing socio-economic aspects of development interventions
Masters level — Social Sciences (economics/sociology or other similar disciplines)
1
3. Engineer 3+ years in field Bachelors level —Engineering Electronics / Telecom
Workload dependent
PROCUREMENT DOCUMENTATION
Part 2, Session 5
List of Important Documents for Pre-qualifications
1. Invitation to Prequalify
2. Instructions to Applicants
3. Prequalification Datasheet
4. Prequalification Evaluation Criteria
5. Prequalification Application Form.
Major parts of RfP for M&E
1. Introduction
2. General Instructions to Bidders
3. Specific Instructions for Technical & Financial parts of Bids
4. Evaluation Criteria (within the bidding docs!)
5. Draft Contract
Structure of TORs
• Preamble
• Brief Project Description
• Scope of Work
• Key Personnel
• Obligations of Successful Bidder
• Time Frame
• Deliverables
• Payment Modalities
THANKS
Backup Slides
Agenda Part-1
• Session 1- Setting the stage • Session 2- Monitoring • Session 3- Evaluation
TORs for the Evaluation Team/Firm
• Evaluation Background and Context • Evaluation Purpose • Evaluation Scope and Objective
– What aspects of the intervention are to be covered by evaluation? (time frame, geographic area & target groups to be considered, projects (outputs) are to be included).
– What are primary issues of concern that evaluation needs to address ?
• Evaluation Questions • Evaluation Methodology
…continued
TORs for the Evaluation Team/Firm
• Evaluation Products – Inception report
– Draft Evaluation report
– Final Evaluation Report
• Evaluation Team Composition & Competencies
• Evaluation Implementation Arrangement
• Evaluation Time Frame for the Process
• Evaluation Cost – Projected Budget
Planning
Goal Setting
Developing Strategy
Resource Allocation
Outlining Implementation
Sourcing Options
Steps for M&E implementation in the 3 scenarios
Scenario 1 Scenario 2 Scenario 3
Recom
mendations for activating M
&E Function
Implement M&E in-house • Execute
M&E Plan
Implement M&E in-house, but: • Acquire deficient HR • Develop M&E team • Develop capacity of team thru
professional training • Procure necessary HW & SW • Develop M&E Plan • Execute M&E Plan • Outsource functions which are
not feasible or are occasional
Outsource M&E • Engage a firm
for provision of M&E services
Possible reasons of selecting one over the other
In-House
• Cost - may be less expensive in long run
• Required expertise is available
• Want direct control over the process
• Desire of better quality
• Flexibility in managing system configuration
• Reliable & competent service providers not available
• Routine nature of task
Outsourced
• Cost - presently less expensive to outsource (eg: there isn’t lot of work)
• Lack of in-house capacity very big
• Outside Service Providers’ research & specialized know-how exceeds
• External validation is required (like third party audit validation)
• To keep the in-house manpower small
• May be a Legal requirement
• Notion that third-party M&E is considered more “neutral” than in-house.
Structure of a typical M&E Unit
Material Resources Required for In-house M&E
• Hardware • Software • Drive test equipments • Furniture • Communication Equipment • Vehicles • Office
Selection Process to outsource M&E
WORKED EXAMPLE FOR EVALUATION OF BIDS
Part2, Session 6
Worked example - Evaluation of Bids
Items Assigned weights
Evaluated Rating Points
1- Qualifications of Bidder (150)
a- Experience in similar projects 50 100% 50
b- Experience in similar geographic areas 50 90% 45
c- Prior experience of working with USPF 50 0% 0
2- Approach and methodology (400)
a- Understanding of objectives 100 90% 90
b- Quality of methodology 100 100% 100
c- Innovation in proposed approach 50 90% 45
d- Work plan/ time lines 50 80% 40
e- Presentation of the Bid 50 90% 45
f- Partnership with a local firm 50 70% 35
(…cont)Worked example - Evaluation of Bids
Items Assigned weights
Evaluated Rating Points
3- Personnel (450)
a- Team leader 100 80% 80
b- Monitoring specialist 50 70% 35
c- Evaluation specialist 50 90% 45
d- Survey specialist 50 100% 50
e- Sociologist/Economist 50 80% 40
f- Reporting Specialist 50 90% 45
g- Other Staff 100 90% 90
Total Evaluated Points 835