DISCUSSION

RESULTS

REFERENCES AND CITATIONS

INTRODUCTION METHODSWe collaborated with CBOs based in Mchinji, Malawi, that were trained to use the CSI tool. CBO staff generated CSI scores for the children that they serve. We then selected a sample of the same children and/or their caregivers (depending on the child’s age) to gather more in-depth information on the same dimensions as those captured by the CSI.

A local, experienced research team collected data by administering a detailed questionnaire we called the Comprehensive Child Welfare tool. We developed the CCW tool by combining validated instruments or best practice indicators in the same domain as those in the CSI. Data were collected immediately following collection of the CSI scores to reduce the likelihood that changes in the welfare of the children would bias our results.

We tested construct validity by examining the correlation between CSI scores and the indicators from the CCW in the same domains. We compared the CCW data with the CSI scores for the same children. We examined the relationship between the domains from the two sources by computing the Spearman's rank correlation coefficient, which measure the strength of the relationship between variables with a ranked correlation coefficient., and does not require that the variables have a linear relationship.

The study was approved by the ethical review boards of Boston University and the Malawian Ministry of Health.

In Malawi, we were unable to validate the CSI tool given the lack of relationship between the CSI and the CCW scores. Although some CSI and CCW scores yielded weak to moderate correlations, there were still important conflicts between the food security, housing, wellness, and education scores, such that children in severe distress were not identified as so in the CSI. Without meaningful correlation in the remaining domains, we concluded that the CSI scores did not accurately assess the welfare of children in the areas of nutrition, care, vulnerability for abuse, legal protection, emotional and behavioral wellbeing, and educational performance. We therefore cannot confirm the validity of the CSI tool in assessing child vulnerability in this particular setting.

In this population, children are experiencing serious problems in many domains of their lives and yet the CBO staff that visited these children did not score children as though they recognized these as issues of concern. These findings underscore the scope of unmet needs among OVC children and their households, as well as the severity of problems that OVC encounter. The CSI tool may not be able to accurately assess child welfare. Tool developers may use this evidence to modify the CSI to further improve its performance and utility.

The USAID | Project SEARCH, Orphans and Vulnerable Children Comprehensive Action Research (OVC-CARE) Task Order, is funded by the U.S.

Agency for International Development under Contract No. GHH-I-00-07-00023-00, beginning August 1, 2008. OVC-CARE Task Order is implemented

by Boston University. The opinions expressed herein are those of the authors and do not necessarily reflect the views of the funding agency.

Evaluation of the Child Status Index Tool: A Validation Study in Malawi

The Child Status Index (CSI), was developed by MEASURE Evaluation at the Carolina Population Center to help community based organizations (CBOs) serving orphaned and vulnerable children (OVC) assess children’s needs.

The CSI was designed to help CBO staff assess children in the following domains: food and nutrition, shelter and care, protection, health, psychosocial wellbeing, education and skills training

1. O'Donnell K, Nyangara F, Murphy R, Nyberg B. Child Status Index (CSI): Public Domain: Developed by the support from the U.S. President's Emergency Fund for AIDS Relief through USAID to MEASURE Evaluation & Duke University, 2008.

2. O'Donnell K, Nyangara F, Murphy R, Nyberg B. Child Status Index: A Tool for Assessing the Well-Being of Orphans and Vulnerable Children -- Manual: United States Agency for International Development, 2009.

3. Miller C., Brooks B., Tsoka M., Rybasack H., Themba Z., Dyless A., Chitekwe M., Kambalame E., and Sabin L. (2010). Evaluation of the Child Status Index: A Validation Study in Malawi. Boston, MA. BUSPH, Center for Global Health and Development. OVC-CARE Project.

CSI Domains and selected tools used in the Comprehensive Child Welfare toolCSI Category Validation tool and developer Tool measures

Food and Nutrition: Food Security Household food insecurity access scale (HFIAS) and Dietary Diversity Scale (DDS)

United Nations (UN) / Food Agriculture Organization (FAO), Food and Nutrition Technical Assistance (FANTA),

Household access to food and anxiety about food; Diversity & nutritional value of diet

Food and Nutrition: Nutrition and growth Anthropometry Weight, height or length, Middle Upper Arm Circumference (MUAC)Shelter and Care: Shelter Shelter Assessment Survey

Questions from World Bank, UnitedNations Children’s Fund (UNICEF) Multiple Indicator Cluster Survey 3 (MICS3) and Demographic Health Surveys (DHS), used in Evaluation of Cash Transfer in Mchinji Surveys

Type, construction, safety of shelter; water and sanitation; cooking practices and fuel

Shelter and Care: Care Indicators from UNICEF OVC guide Dependency ratio, relationship to head of household, illness and other OVC in homeOrphan connection ratio: ratio of average level of connection for OVC to average level of connection for non-OVC

Support received by child; emotional and physical

Protection: Abuse and Exploitation Child Abuse Screening Tool Children's Version (ICAST-C)

International Society for the Prevention of Abuse and Neglect (ISPCAN) and UNICEF

Current & lifetime exposure to violence

Child labor module from MICS3;

Conforms to UNICEF, UNAIDS indicators

Type, frequency of work activities during and outside of school hours

Protection: Legal Protection Birth registry survey from MICS3

Conforms to UNICEF, UNAIDS indicators

Legal questionnaire: Developed using indicators from UNICEF, UNAIDS best practice

(For 5-10 year olds) Registration status; For all ages, questions on inheritance, will, who child goes to for help, knowledge & enforcement of laws, experience with property theft

Health: Wellness Questions from UNICEF, MICS3 and DHS surveys, used in Mchinji Surveys

Malaria module from MICS3; HIV module from GSHS; Risky behavior survey from Malawi/Uganda Global School Based Student Health Survey (GSHS); used in Evaluation of Cash Transfer in Mchinji Surveys

Health status (reported for 11-17 year olds, observed for 5-10 year olds), episodes and severity of illness; For HIV education and knowledge among 11-17 year olds; Sexual activity and protection, for 11-17 year olds

Health: Health care services Questions from World Bank, UNICEF MICS3 and DHS surveys, Used in Evaluation of Cash Transfer in Mchinji Surveys

Utilization of healthcare, whether used during last illness; if child would receive services if very ill

Psychosocial: Emotional health Children’s Depression Inventory (for parents 5-10 years and child 11-17 years)Multi-Health Systems Inc.

Screening with psychometric properties; Separate versions for children 5-10 and 11-17. Designed for children age 7-17

Psychosocial: Social Behavior Strengths and Difficulties QuestionnaireYouth In Mind, Limited

General screening for children with emotional & behavioral problems

Education & skills training Indicators from United Nations Educational, Scientific and Cultural Organization (UNESCO) /MCIS indicators

Enrolment, grade level, attendance

Children surveyed using the comprehensive child welfare instruments, by gender and age

Characteristic Number of children:Ages 5-10

n=102

Number of children Ages 11-17

n=100

Gender Boys 49 50 Girls 53 50

CSI Domain: Food and NutritionZ-score for underweight (higher score indicates greater weight)

Z-score for BMI (higher score indicates greater weight)

Food Insecurity Composite Score (higher score indicates more food insecurity, range is from 0-9)

Food Diversity Composite Score (higher score indicates greater food diversity, range is from 0-16)

Children 5-10 years

Food Security -0.14 -0.07 0.40 *** -0.18 Nutrition -0.20 * -0.08 0.20 * -0.17

Children 11-17 years

Food Security 0.06 -0.009 0.10 -0.03 Nutrition 0.07 -0.13 -0.09 0.05

CSI Domain: CareCare Dependency ratio (higher

score indicates worse dependency ratio)

‘Bad Care’ composite (higher score indicates worse care, range is from 0-6)

Children 5-10 years -0.02 Not calculated

Children 11-17 years 0.02 0.006

CSI Domain: HealthcareHealthcare Receiving

health care when sick

Having to care for self when sick

Composite score for not seeking health despite serious illness (higher score indicates less access to healthcare) (range is from 0-11)

Using a mosquito net

Children 5-10 years 0.05 - 0.05 0.16

Children 11-17 years 0.03 0.05 0.08 0.15

CSI Domain: Emotional Health CDI sub scale of emotional depression (range is from 0-16)

CDI sub scale of functional depression (range is from 10-26)

CDI total composite score(range is from 10-42)

Children 5-10 years 0.04 -0.04 0.001

*p<.05; **p<0.01; ***p<0.001

The mean height-for-age z-score for children aged 5-10 years that were rated as having ‘good’ nutrition in the CSI was -1.08, ‘fair’ was -1.80, ‘bad’ was -1.79 and ‘very bad’ was 0.63. We would expect to see worsening mean z-scores according to the CSI ratings if the CSI scores accurately identified children with nutritional problems.

Among 11-17 year olds, the mean z-score for those rated as having ‘good’ nutrition was -1.38, -1.16 for ‘fair’ nutrition, -0.51 for ‘bad’ nutrition. No 11-17 year olds were rated as having ‘very bad’ nutrition, even though 25% were stunted (2+ SD below the mean).

DATA ANALYSIS: Spearman Rank Correlation Coefficients and associated p-values

CBO assigned CSI Tool Scores for each domain for 11-17 year olds Older kids (percentage of children) Good

0Fair

1Bad

2Very bad

3

Food & nutrition Food security 9 27 24 40

Nutrition 77 21 2 0

Shelter & care Shelter 22 36 25 17

Care 47 40 8 5

Protection & abuse Abuse 89 11 0 0

Legal Protection 94 5 1 0

Health Wellness 63 32 5 0

Healthcare 81 15 3 1

Psychosocial wellbeing Emotional Health 71 23 5 1

Social Behavior 84 15 0 1

Education & skills Performance 47 33 10 10

Education & work 59 26 8 7

Miller C., Brooks B., Tsoka M., Rybasack H., Themba Z., Dyless A., Chitekwe M., Kambalame E., and Sabin L. (2010). Evaluation of the Child Status Index: A Validation Study in Malawi. Boston, MA. BUSPH, Center for Global Health and Development. OVC-CARE Project. Miller C., Brooks B., Tsoka M., Rybasack H., Themba Z., Dyless A., Chitekwe M., Kambalame E., and Sabin L. (2010). Evaluation of the Child Status Index: A Validation Study in Malawi. Boston, MA. BUSPH, Center for Global Health and Development. OVC-CARE Project.

CSI Domain: EducationEducation Enrolment Attendance Composite based on enrolment,

attendance & work (range is from 0-3)

Children 5-10 years -0.22 * -0.06 0.17

Children 11-17 years -0.32 ** 0.21 * 0.12

Lora Sabin1, Candace M. Miller1, Mohammed I. Brooks1, and Jacqueline Cho1

1Boston University School of Public Health, Center for Global Health and Development

Despite CBO use of the CSI, there had been no rigorous evaluation of the tool to determine whether the CSI generates valid data regarding the type and degree of vulnerability facing OVC.

This evaluation of the validity of the CSI, serves as an independent, external assessment of the tool in a specific country-context.