Case study

Title: Overview of Impact Assessment Tools

Sub-title: Measuring quality of services through qualitative data

Country: Nepal

Specific thematic: Physical Rehabilitation

Presented by: Reena SHAKYA

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Project context

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Project Title: "Enabling local partners to deliver sustainable and quality physical rehabilitation services for persons with disabilities in Nepal“ Jan 2010 – Dec 2013

Global Objective: To improve functional capacities and autonomy of persons with physical disabilities in Nepal, and thereby enhance their full and effective participation in all sectors of society, as well as in the post-conflict reconciliation of the country.

Specific Objective: To ensure and optimize quality physical rehabilitation services in Nepal, accessible for all - especially for isolated and vulnerable persons with physical disabilities - on a sustainable way and based on locally capacitated actors.

Project Indicators

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Key Project Indicators:• Service delivery: Quality physical rehabilitation services (including

therapy sessions and assistive devices) provided to persons with physical disabilities

• Client satisfaction: 70% of clients satisfied with the services provided during year 1 and 80% from year 2 onwards

• Client centred goal setting: Technical staff able to establish and set rehabilitation goals together with clients/carers in year 1. 60% of clients achieve set goals in year 2, 70% in year 3 and 80% in year 4

• Sustainability: 4 locally managed physical rehabilitation structures operate autonomously (institutionally, financially and technically)

• Accessibility to remote area through outreach mobile camps, Community workers mobilisation

• Clients’ Satisfaction Survey Questionnaire (CSSQ): to measure clients’ satisfaction on the service provision (Counsellor)

• Case Studies: to document the outcome of the project on client’s life (Rehabilitation staff)

Impact Assessment Tools used:

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

• Technical Staff Skills Assessment: to enhance the technical skills of the partner rehabilitation staff, (HI Technical team)

• Client Centred Goal Setting: to set SMART goals for a smooth rehabilitation process (Client and the Rehabilitation staff)

• Rehabilitation Management System (RMS): to measure quality of services, (Management team)

• Project Database: to maintain the systematic data collection system, (Data entry person)

Tool/study implemented

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

What How Measures MethodologyCSSQ Structured CSSQ

form - clientsSatisfaction of the clients: Quality & timeliness of service, physical accessibility & attitude of service providers

Clients at centre uses CSS form during their follow up visit. The form is analysed & used to improve the services by the service providers.

Case Studies Standard Template – tech. staff

Outcome of project activities on individual client’s life

Identified clients based on defined criteria, record during regular follow up and summarize case study

Tech. Staff Skills Assessment

Skill assessment for PT and P&O based on their job resp. & prof. skills- Senior tech staff

Improvement of technical knowledge and skills of rehab staff

HI Senior technical staff assess partners’ technical staff skills once/year for PTs and twice/year for P&Os. Personalized mentoring plan follows the assessment

Goal Setting Set goal together with client and career –tech. staff

Functional improvement of the client

Client centered approach, clients and family members are consulted and jointly set goals for achievement during the rehabilitation process, SMART goals setting

Rehab ManagementSystem

5 core areas of centre mangt. checklist review - service providers

Measures quality of services in 5 core areas (users, service output, finance, staff & internal mangt.)

Performance in 5 core areas of management are measured against standard in a participatory way by center staff. This allow planning for improvement in priority areas.

Outcome or Impact that the tool/study was able to

demonstrate?

• CSSQ: satisfaction of beneficiaries; 88% of the total beneficiaries were shown to be satisfied by the service provision

• Case Study: Significant changes in the quality of life of an individual

• Technical Staff Skills Assessment: measures the level of technical skills and capacity of the rehabilitation staff

• Goal Setting: effectiveness/ appropriateness of the interventions in a given timeframe for each clients, quality of services

• Project Database : Global overview of service provision, including geographical coverage, equity and effectiveness, utilization rate, prevalence on impairment with respect to age group, time of intervention etc.

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Project Database

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Clients Satisfaction Survey Questionnaire

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Outcome or Impact that the tool/study was able to

demonstrate?

• Case study before & after photos

Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

Facilitating factors and obstacles

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

Tools Facilitating Obstacles/ ChallengesCSSQ - Presence of

Counselor- Bias (positive answers)- Low literacy of beneficiaries

Case Study - Case study template guide

- Home visits and presence of CDWs

- Language is barrier for local staff - Time consuming (2-3 hrs to complete a

form) - Long process (3-6 months)

Goal Setting - Technical resources in the PRCs

- Client centered approach

- Time management (for those coming from remote areas)

- Negotiation with family members, always have high expectation

- Follow up not always possible

Skills Assessment

- Presence of Senior PT and P&O

- Internal process, not institutionalized; takes time to measure the outcome/ effect

Project Database

- HI skilled staff - Poor computer skills of the partner staff- File corruption and power cuts

To use these tools effectively, long term project visibility is essential (since it involves development of tool, field testing, use, review and analysis); need of skilled staff / capacity building of staff at HI and partners level

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh

How to duplicate

Tools RecommendationsCSSQ: involve external stakeholder for evaluation, FGD to have global

survey

Case study: identify the case early during the project period to give enough time for follow up of the case and see the outcomeuse defined criteria for identification of cases

Skill Assessment:

refer to international standard to be more accurate

Goal Setting : promote client centered approachEnsure strong clinical skills of technical staff

Database: Disaggregate data as much as possible by type of impairment, client pathway, socio-economic category, ethnic group, age group…

Multiple tools allow us to cross check the impact, but there is still scope of improvement in collection of qualitative data.

Limitation: Quality of impact measurement depends on the capacity of the partners/ service providers: human resources, tools, clinical skills, management skills

Reference to international standardized tool and definitions in the rehabilitation sector will help to improve the impact assessment tools, has to be adapted to the local context

Summary points:

Impact Evaluation Seminar 2 to 6 Decembre, Phnom Penh