schwartz quantitative and qualitative data

7/28/2019 Schwartz QUantitative and Qualitative Data

1/12

Combining Quantitative and Qualitative Data

in Welfare Policy Evaluations in the United StatesI

ANDREW S. LONDONSyracuse University, NY, USA

SAUL SCHWARTZCarleton University, Ont., Canada

and

ELLEN K. SCOTT *

University of Oregon, OR, USA

Summary. Researchers have made significant efforts to combine quantitative and qualitativemethods in welfare reform policy research in the United States. This paper draws on several exam-ples arising from the American experience to argue that mixed-methods research (particularly, butnot exclusively, with integrated sampling, data collection, and data analysis) can yield importantand unexpected insights that neither method alone could generate. We caution that each methodhas strengths and weaknesses that must be borne in mind so as not to oversell the promise ofmixed-methods research. 2006 Elsevier Ltd. All rights reserved.

Key words poverty, welfare reform, mixed-methods research, policy evaluation, United States

1. INTRODUCTION

As is the case among researchers working onissues of economic development and poverty indeveloping countries (Kanbur, 2003), there issubstantial interest in the promise of mixed-methods approaches among researchers evalu-

ating the effects of welfare reform and relatedpolicies and programs in the United States.This interest was evident among the partici-pants at a conference entitled Mixed-MethodsResearch on Economic Conditions, Public Pol-icy, and Family and Child Well-Being, whichwas hosted in June 2005 by the National Pov-erty Center at the University of Michigan.Many of the concerns and tensions that haveemerged in the discussion of mixed-methodsapproaches in poverty research in developingcountries also exist among poverty researchersworking in the United States.

For many years, researchers working onMDRCs Project on Devolution and Urban

Change and the Next Generation Project havebeen attempting to integrate analyses ofstructured interview data collected in thecontext of both population-based surveysand experimental evaluations with narrativedata collected during lengthy, conversational

* This paper was written with support from MDRCs

Next Generation Project, which is funded by the David

and Lucile Packard Foundation, the William T. Grant

Foundation, and the MacArthur Foundation. We thank

Martha Bonney for editorial assistance.q Earlier versions of this paper were presented on thepanel on integrating qualitative and quantitative re-search methods at the 24th annual APPAM ResearchConference, Dallas, TX, November 79, 2002 and at theQ-squared in Practice Conference on Experiences ofCombining Qualitative and Quantitative Methods inPoverty Appraisal, University of Toronto, Toronto,Ont., Canada, May 1516, 2004.

World Development Vol. 35, No. 2, pp. 342353, 2007 2006 Elsevier Ltd. All rights reserved

0305-750X/$ - see front matter

www.elsevier.com/locate/worlddevdoi:10.1016/j.worlddev.2006.10.007

342


2/12

interviews with welfare-reliant and workingpoor women. Our goal, always, has been totry to better understand how welfare reformand employment policies affect low-income wo-men, their children, and their families. Partly

out of necessity and partly by design, our strat-egy has generally been to integrate across meth-ods at the data analysis stage and to use thefindings from the analysis of one data sourceto confirm, challenge, extend, complement,supplement, or nuance the findings emergingfrom the other. Rightly or wrongly, the anthro-pologists, developmental psychologists, econo-mists, public policy analysts, and sociologistsworking on our interdisciplinary teams tendto reduce the distinction between qualitativeand quantitative to the difference between data

obtained, respectively, from structured inter-views (quantitative) and interviews primarilyinvolving open-ended questions (qualitative).We also tend to think of all attempts to inte-grate these two types of data as mixed-methodsresearch.

Following the first Q-squared conference,Kanbur (2003) proposed five key dimensionsthat can be used to situate studies along thequalitativequantitative continuum: (1) non-numerical versus numerical information on thepopulation; (2) specific to general population

coverage; (3) active to passive populationinvolvement; (4) inductive to deductive infer-ence methodology; and (5) broad social scienceversus neo-classical economics disciplinaryframework. We view our attempts to mix meth-ods in these evaluation studies as attempts to bemore broadly social scientific by drawing onboth numerical and nonnumerical informationto generate general and specific informationabout the focal population and the effects ofpolicies through inductive and deductive ap-proaches to inference. All of the primary studies

that we will discuss in this paper were designedas evaluations of policies and programs. Assuch, they were largely driven by researchersinterests with limited active involvement ofmembers of the population. The qualitativeinterview studies allow for somewhat more ac-tive participation in the interview process, buteven in those situations, the prerogatives ofthe researchers predominate.

In this paper, we review a few of the ways wehave used these quantitative and qualitativedata in our work and discuss some of the prom-

ises and pitfalls of combining qualitative andquantitative research methods and data in pov-erty appraisal and policy evaluation research.

Along with the Q-squared conference partici-pants represented in this special issue (see alsoKanbur, 2003), participants in the conferenceat the National Poverty Center, and a growingnumber of other researchers in the U.S. (e.g.,

Gibson & Duncan, 2005), we believe there aresynergies to be gained from the careful combi-nation of mixed research methods. Drawingon our own experience and the reports of oth-ers, we identify several factors that we thinkwill enhance the value of combining qualitativeand quantitative methods and data in povertyappraisal research and policy evaluations.However, we also caution that all methodsand data have their limitations, and the goalsof quantitative and qualitative researcherssometimes legitimately differ. Researchers must

be careful to specify the types of questions thatthese different types of research methodologiesand data can and cannot answer, and the kindsof contributions they can and cannot make.To fail to do so risks raising expectations toohigh.

2. DATA SOURCES

The population-based survey data and thequalitative interview data referenced in this

paper were collected during 19972001 underthe auspices of MDRCs Project on Devolutionand Urban Change. 1 Urban Change is a multi-city, multi-component study of the effects of the1996 welfare reforms in the United States onwelfare administration, caseload dynamics,poor women and their children, community-based social service organizations, and urbanneighborhoods. Urban Change had severalcomponents that were directly linked: theadministrative records component, which in-volved analysis of the universe of cases in each

city over a 10-year period; a neighborhood indi-cators component, which focused on census-tract level change over the same 10-year period;and a survey component, which involved thecollection of two waves of data from a randomsample of women selected from the administra-tive records. The qualitative interview studieswere designed to be nested loosely within butnot directly linked to other components ofUrban Change. The qualitative interview com-ponent was also designed to articulate with animplementation component, which focused on

changes in welfare offices, and an institutionalcomponent, which focused on the effects of wel-fare reform on community-based social service

COMBINING QUANTITATIVE AND QUALITATIVE DATA 343
http://-/?-http://-/?-http://-/?-


3/12

agencies. The offices and agencies that werestudied in the latter two components werelocated in the same neighborhoods (i.e., clus-ters of census tracts with particular characteris-tics) in which the women in the qualitative

interview samples lived. Thus, the overall designof Urban Change included comparative cityand neighborhood studies nested within a lar-ger, quasi-experimental analysis of administra-tive records and neighborhood change, and apopulation-based, longitudinal survey.

The first wave of the Urban Change surveyinvolved in-person interviews in 199899 withwomen who had been single mothers receivingwelfare and/or food stamp benefits in May1995 and who were living in neighborhoods(census tracts) characterized by high rates of

poverty (30% or more of households) or welfarereceipt (20% or more of households). Usingdata from administrative records, women whomet these criteria were randomly selected, witha sample size targeted to be 1,000 completedinterviews per site. In-person interviews werecompleted in 199899 with 3,960 women(78.6% of those sampled three years earlier).Follow-up interviews with the same womenwere completed in 2001.

The longitudinal, qualitative interview com-ponent of Urban Change involved three or

four semi-structured, in-person interviews con-ducted during 19972001 with approximately40 women per site. These women lived in neigh-borhoods of concentrated poverty similar tothe neighborhoods in which the survey respon-dents lived; however, no woman participated inboth the survey and longitudinal, qualitativeinterview studies. Racial/ethnic criteria werealso used to select respondents for the qualita-tive studies. In Cleveland and Philadelphia,the cities represented most consistently in thereports discussed below, all of the women in

two neighborhoods per city were AfricanAmerican, while in one neighborhood per citythey were white.

The qualitative interview samples were notdirectly linked to or embedded within the sam-pling frame for the survey component becausethe interdisciplinary team of researchers work-ing on the qualitative interview component be-lieved it was critical to recruit women throughmeans that were completely independent ofthe welfare office. Thus, the research teamsrecruited women in the community from se-

lected census tracts that had the same povertyand welfare receipt rates as the census tractsfrom which the survey sample was drawn. As

a result, the qualitative and quantitative com-ponents of Urban Change are largely indepen-dent of one another even though they weredesigned in tandem, were fielded over the sametime period, and included women from the

same population. Despite the fact that therewas no explicit linkage between the surveyand qualitative interview components of UrbanChange, we have tried to integrate them at thedata analysis stage to add breadth and depth toour answers to specific questions. The struc-tured survey and open-ended, qualitative inter-views covered similar topics; however, thequalitative interviews yield richer, narrativedata about how families are coping and whatthey are experiencing, which arguably is animportant basis for policy evaluation and

formulation.Under the auspices of MDRCs Next Gener-

ation Project, we have also used the qualitativeinterview data from the Cleveland and Phila-delphia sites of Urban Change to inform anal-yses of structured interview data from a set ofexperimental evaluations of the impacts ofwelfare-to-work programs. These experimentswere conducted to evaluate the effects on low-income families of programs designed primarilyto increase parental employment, althoughsome also had income supplementation compo-

nents. All of the experiments were started in theearly to mid-1990s under waivers of the federalrules governing Aid to Families with Depen-dent Children (AFDC) granted to individualstates. In each study, the heads of low-incomehouseholds in a given locationmost of themwelfare recipientswere randomly assigned toa program group or a control group. Programgroup members were eligible for the servicesand subject to the requirements of the programunder study, while the control group in mostsites remained subject to the eligibility require-

ments and benefits of the existing AFDC sys-tem. 2

These experimental data provide strong evi-dence of causal impacts, but often do not allowus to see within the black box to surmise howthe policy or program might have had its im-pact. Although the qualitative interview datawere not collected from the same participantsor in the same period, they were collected fromwomen who were subjected in a later period tomany of the same policies as the women inthese experiments. Thus, we used the qualita-

tive interview data to help us think about pro-cesses that might have been affected by suchpolicies, to nuance the conclusions emerging

344 WORLD DEVELOPMENT
http://-/?-http://-/?-http://-/?-


4/12

from the experiments, and to suggest hypothe-ses that could be tested.

3. COMBINING SURVEY AND

QUALITATIVE INTERVIEW DATA

Even though the survey and qualitative inter-view samples in the Urban Change study werenot linked, having complementary survey andqualitative interview data has been beneficial.The qualitative data have reminded us to con-sider the possibility of mismeasurement in pop-ulation-based structured interview studies dueto heterogeneity in the meanings people attri-bute to concepts, social desirability bias, theformat of the interview, and the social context

in which people are living their lives. The qual-itative data are useful for scratching beneaththe surface of prevalence estimates to suggestsome of the dynamic cognitive and behavioralprocesses that underlie static, point-in-timesurvey estimates. Below, we discuss severalexamples that illustrate this set of benefits.

(a) Measuring food insecurity

Polit, London, and Martinez (2000) com-bined data from the 1998 Urban Change survey

and the baseline qualitative interview studiesconducted in Cleveland and Philadelphia toexamine food security and hunger. Food secu-rity in the survey was measured with the USDepartment of Agricultures Household FoodSecurity Scale (HFSS). The HFSS consists of18 items that can be used to classify respon-dents into one of the four categories: foodsecure, food insecure without hunger, foodinsecure with moderate hunger, and food inse-cure with severe hunger (Carlson, Andrews, &Bickel, 1999). 3 The scale has been judged to

be reliable and valid for both individual andpopulation uses (Frongillo, 1999), and has beenadministered by the US Bureau of the CensussCurrent Population Survey since 1995. The wo-men in the qualitative study samples were notadministered the HFSS at baseline; however,they were queried about their food expendi-tures, use of Food Stamps, WIC, and otherpublicly-funded food programs, food depriva-tion, and the use of emergency food services.These data were used to classify women intoone of the above-mentioned food security cate-

gories. About half of the women in the surveyand qualitative interview samples, respectively,were classified as food secure.

Although the survey and qualitative inter-view samples were not linked and the strategiesfor measuring food security differed, havingboth data sources available provided importantinsights that allowed these researchers to raise

questions about the completeness of the HFSSas a measure of food security and highlightsome of the limitations of self-reported surveydata as measures of qualitative constructs suchas food security. Scales used in surveys oftenconsist of items allowing only fixed categoricalresponses and researchers often fail to appreci-ate the heterogenous meanings that peopleascribe to those categories. In-depth interviewspoint out some of this heterogeneity and sug-gest that analysts should have a more nuancedsense of how much confidence should be placed

in the survey responses and what the scale doesand does not measure.

By design, the HFSS measures whether fam-ilies are food secure, but it does not address theeffort that is expended in attaining food secu-rity. Carlson et al. (1999, p. 512S) note thatthe HFSS was developed as a unidimensionalscale using factor analysis and that with onemajor exception nearly all the items fit. Theone general type of item that did not fit themodel was indicators of coping strategies thata food-insecure or at-risk household might use

to improve its food supply from emergencysources, such as obtaining food from a foodbank or borrowing money for food. Suchcoping items were explicitly excluded fromthe scale because they did not fit the conceptualand empirical model that guided the develop-ment of the scale and were viewed by the scaledevelopers as representing a domain that wasdistinct from food security.

Most of the mothers in the qualitative inter-view samples who were classified as food secureexpended considerable energy in piecing to-

gether numerous strategies to make sure thatthere was adequate food for themselves andtheir children. Women who were food securetalked a lot about managing their paperworkwith the welfare office to maintain their FoodStamps, shopping carefully by sometimespurchasing day-old bread and reduced-pricemeats, and routinely relying on friends or rela-tives for meals or loans that enabled them tofeed their families. The qualitative data sug-gests that the label food secure masks con-siderable variation in the efforts that families

expend to achieve food security and in the pre-cariousness of their ability to maintain thisstatus.

http://-/?-http://-/?-http://-/?-


5/12

Researchers studying the impact of increasedmaternal employment resulting from welfarereform, especially when such employment doesnot appreciably increase income, should beconcerned about the effort required to attain

any level of food security, as well as the levelof food security itself. Additional survey ques-tions might capture womens efforts to feedtheir families but, as always, space on surveysis limited. At a minimum, survey analystsshould recognize that the level of food securityas indicated by this unidimensional scale is anincomplete measure that does not provide anindication of the effort it takes to put food onthe table, the opportunity costs associated withthat effort, or the precariousness of the status.We note that this is a traditional way in which

in-depth qualitative interviews can inform thedevelopment of quantitative survey questionsand the design of survey questionnaires.

The ethnographic data also revealed other is-sues of relevance to the analysis and interpreta-tion of the HFSS categories. In keeping withthe conceptual definition of food security ashaving adequate access to food withoutresorting to emergency food supplies (Ander-son, 1990), all of the women in the qualitativestudies who used food banks were classifiedby the coders as food insecure. However, it is

possible that these women might have classifiedthemselves as food secure, partially as a resultof their strategic and planned use of the foodbank resources available in their communities.This suggests that there might be some mis-classification in the HFSS resulting from impli-cit normative assumptions by researchers aboutwhat the use of food banks means. Suchassumptions may not map onto the normativeviews of low-income mothers who regularlyrely on food banks to achieve a desired levelof food security for their families.

Polit et al. (2000) noted that there were hintsin the qualitative data that the women wereproud of their ability to feed their children.They might not always be able to pay their util-ity, rent, and other bills, but they made surethere was adequate food on the table. Therewere also hints in the data that some womenfeared that if their children were not adequatelyfed, the children would be taken away fromthem. If there is pride associated with maintain-ing adequate food, and shame or fear in notbeing able to do so, it is possible that these con-

cerns would lead some women to avoid endors-ing survey items that are indicative of foodinsecurity. These biases would plague not only

the HFSS, but also any additional survey andqualitative interview efforts to monitor the pro-cess by which poor women attain whateverfood security they have.

(b) Contextualizing assessments of satisfactionwith living standards

In a second example that continues todevelop the themes from above, having bothquantitative and qualitative data on womensassessments of their standard of living providednuance to our understanding of what mightotherwise have been a perplexingly unclear sur-vey finding. Although half of the women in theUrban Change survey sample were living belowthe federal poverty threshold in 2001 and four

out of five were near poor (i.e., within 185%of the poverty threshold), three out of fourrespondents reported that they were very satis-fied or satisfied with their standard of living(Brock et al., 2002).

We asked women in the qualitative study inCleveland to rate their standard of living anddiscuss why they felt as they did about it.Although the samples were not linked, whatwe learned from the analysis of the qualitativeinterview data provided insights that helpedus make better sense of the survey estimate.

The qualitative interview data indicated thatwomen sometimes framed their relatively posi-tive assessments of their standard of living inrelation to their perception that many otherswere worse off than they were, or that they werebetter off than they had been previously. Forexample, several women said that they wouldnot use food banks because they are only forpeople who really need it. By framing theirsituations in such terms, women could maintaintheir self-esteem and optimism while resistingdemoralization and the sense that they were

somehow failing. This represents, we believe,a sort of cognitive coping strategy that mayhave affected how some of these women re-sponded to the survey questions.

Sometimes, when womens assessments oftheir standard of living were negative, it was be-cause financial constraints made it difficult forthem to fulfill important social role obligations,such as being a good mother and provider. Forexample, Cindy 4, a white mother of two chil-dren lived with many housing problems in aneighborhood she considered unsafe, experi-

enced unmet dental care needs and utilityshut-offs, and had to skimp on her own foodso that her children could eat. Cindy described

http://-/?-http://-/?-http://-/?-


6/12

herself as being unhappy about her standardof living because it made her feel bad as amom. She said:

When I got my income tax back, I went out and Ibought Kim, my oldest girl, clothes. She had not

had new clothes in over a year and a half, or shoes.I mean, you know, and that hurts me as a mom . . .Ithurts me that they would have to wait for a windfalllike that, just to get what would be considered basicday-to-day clothes. I mean, my idea of basic day-to-day clothes is a pair of jeans for each day and stufflike. . .I mean, Im not talking about going out toLondon for the weekend or something likethat. . .And maybe able to go out. . .like being ableto let them do the things they want to do, simplethings like going roller skating, have extra moneyto do that sort of stuff. When you dont have it,you cant, you know, it hurts.

The narrative accounts we obtained from wo-men living in some of the most disadvantagedneighborhoods in Cleveland provided contextfor the generally high estimates of satisfactionwith living standards we obtained from respon-dents in both the survey and qualitative inter-view components. In these neighborhoods ofconcentrated poverty, the reference group thatwomen use for making their assessments oftheir own circumstances include families whothey perceive to be worse off than they are.

Additionally, poor womens coping strategiesmay include denial and other cognitive strate-gies that allow them to maintain their dignityand sustain their considerable efforts to meettheir families needs. Researchers using suchsubjective appraisals must be mindful of theways that the context of respondents livesshapes how they respond to questions in bothsurvey and qualitative interview studies.

(c) Under-estimating ongoing domestic violence

The notion that the context of peoples livesshapes how they respond to questions, whetherthey are asked in structured survey question-naires or open-ended interviews, was broughthome to us forcefully in our work on domesticviolence. In this instance, we see the potentialfor the under-estimation of domestic violence,which can be pieced together from two differentresearch reports. The first study reported base-line survey and baseline qualitative interviewfindings from the four Urban Change sites (Po-lit, London, & Martinez, 2001). The qualitative

interview data were used to illustrate the typesof abuse women faced, but not to raise ques-tions about the survey estimates. The second

study reported longitudinal, qualitative datafrom the Cleveland site of the Urban Changeproject (Scott, London, & Myers, 2002a, Chap-ter 14).

It is well documented that poor women are at

disproportionate risk for domestic violence,and that many women on welfare have a his-tory of abuse (Raphael & Tolman, 1997; Scott,London, & Myers, 2002b; Tolman & Raphael,2000). The lifetime prevalence rate for welfare-reliant women is estimated to be 5075%, witha past-year prevalence of 1520%. Polit et al.(2001) reported that 8.8% of women in theUrban Change sample were hit, slapped, orkicked in the year prior to the 1998 survey.Results from our longitudinal ethnographicstudy suggest that this may under-estimate

the extent of recent or ongoing violence in thispopulation.

In our qualitative interview sample of 38 wo-men in Cleveland, approximately 60% of thewomen disclosed a history of domestic violence(Scott et al., 2002b). However, three womenwho were being abused during the course ofthe study did not disclose the abuse to theinterviewer until after the abuse had ended.Despite very high levels of rapport with theinterviewers, these three women respondedno to all questions on domestic violence on

a self-administered questionnaire, which wasthe same one that was used in the UrbanChange survey, and to open-ended questionsin the qualitative interview that took placethe year prior to the qualitative interview inwhich the woman disclosed the abuse (i.e.,these women reported no abuse in the sec-ond-year qualitative interview and disclosed itin the third-year qualitative interview). In eachcase, when they disclosed that they had beenabused, the women indicated it had been goingon at the time of the prior interview and some-

times throughout the entire course of thestudy. We illustrate this with the case of Mariaabout whom we have written elsewhere (seeScott et al., 2002a).

Maria had been physically abused by herboyfriend, the father of her two youngest chil-dren, for eight years, until February 2000 whenhe moved out of the house. Maria said she wasbeaten at least once a month during that time,sometimes resulting in trips to the hospital forstitches. The interviewer perceived herself tohave good rapport with Maria, and this was

demonstrated in the many intimate detailsMaria had shared about her life with this inter-viewer. The interviewer reported being shocked



7/12

when the abuse was disclosed. Maria had hadother opportunities in prior interviews to talkabout the abuse, but all of them occurred whileshe was still living with the batterer.

There may be a number of reasons why Mar-

ia chose not to disclose the abuse until it hadstopped, despite the high level of rapport be-tween her and the interviewer. She may havebeen afraid her boyfriend would retaliate if hefound out she had disclosed the abuse. Shemay have felt ashamed of the fact that shewas being beaten, and somehow that shamewas lessened when it had ended. Finally, shemay have been surviving the abuse by denyingeither the fact of the violence or the extent ofthe violence. Denial becomes much more diffi-cult if one acknowledges the violence to some-

one else.Our experience with having three cases of

domestic violence disclosed two years into astudy involving extensive contact betweeninterviewers and respondents surprised us. Wethink these data raise important methodologi-cal and policy concerns. Given that most stud-ies cited in the literature on welfare reform anddomestic violence involve single contacts, or atleast significantly less contact with subjectsthan we had in the Urban Change study, andgiven that most welfare caseworkers asking

questions to identify cases of domestic vio-lence will likely meet with recipients for muchshorter periods, we have reason to be very con-cerned about whether we are capturing the ex-tent of the problem either in our research or inour attempts to provide public assistance. Thisis particularly true for those women who arecurrently being abused. Those are the womenabout whom we should be most worried andfor whom we should seek the most flexiblepolicies. If the shame, denial and fear batteredwomen typically experience keep some women

from disclosing an ongoing history of domesticviolence to a trusted interviewer with whomthey have been talking intimately for twoyears, why do we think this is going to be easyto identify in a welfare office in a half-hourmeeting when uncertain benefits might beat stake? This poses an extraordinary conun-drum for policy-makers. It also suggests thatresearchers in both developed and developingcountry settings who are studying domesticviolence need to build trust through long-termcontacts in order to monitor how changes in

the social contexts of womens lives affect theirwillingness to report that they have experi-enced violence.

4. COMBINING EXPERIMENTAL ANDQUALITATIVE INTERVIEW DATA

In a recent paper, Gibson and Duncan (2005)argue that there are potentially large benefits to

be gained from efforts to combine qualitativeand experimental evidence. Although we didnot have in place the two design features thatthey identify as contributing to the synergy theyexperienced in the New Hope DemonstrationProject (i.e., an embedded and linked, ran-domly drawn sample for the ethnography andhaving the same analysts conduct both qualita-tive and quantitative analyses), we have alsohad experiences in which this combination ofmethodologies was beneficial. In this secondsection of the paper, we turn our attention to

two examples in which we fruitfully combinedqualitative and experimental data analyses inMDRCs Next Generation Project. We beginwith a discussion of the combination of qualita-tive and experimental findings in relation toyoung childrens well-being in the context ofmaternal welfare-to-work transitions, and thenwe turn our attention to the same issue withrespect to the well-being of adolescents.

(a) Young childrens well-being

Morris, Scott, and London (2005) attemptedto further our understanding of how welfareand work policies affect young children by com-paring, contrasting, and synthesizing the resultsfrom two separate Next Generation reports.They sought to interrogate and integrate theconclusions drawn from these studies in orderto better understand the mechanisms throughwhich welfare reform policies targeted atadults, most of whom are mothers, impactyoung childrens lives. One of the initial studiessynthesized findings from eight experiments

(Morris et al., 2001) and the other examinedlongitudinal, qualitative interview data fromCleveland and Philadelphia (Scott, Edin, Lon-don, & Kissane, 2004).

Findings from the experimental studiesrevealed that impacts on childrens well-beingdiffered depending on the policy approachand subsequent changes in parents economicoutcomes. Programs that increased parentsemployment and earnings benefited young chil-dren; they exhibited better school achievement,reduced behavior problems, increased positive

social behavior, and improved health. By con-trast, programs that simply moved mothersinto employment (without increasing earnings)



8/12

did little to benefit young children. Becauseexperimental impacts are derived from the dif-ference in functioning between children andparents in the program and control groups,these findings may reflect the fact that receiving

welfare (as more parents in the control groupare doing) and working (as more parents inthe program group are doing) are equallystressful for parents. The advantages thatemployment may bring to parents may be out-weighed by the stresses of balancing work andfamily.

The majority of respondents in the qualita-tive interview sample, which included womenfrom some of the most disadvantaged neigh-borhoods in Cleveland and Philadelphia, expe-rienced neither stable work nor sustained

increases in income in the first year of the study.They struggled to make ends meet with incomelevels hovering around the federal povertylevel. Juggling stressful jobs, irregular or part-time hours, and often temporary employ-ment, mothers worried a great deal about theirchildrens well-being. 5 Exhaustion, disruptedroutines, and increased absence from the homedue to employment meant that some mothersreported their children were doing worse,exhibiting diminished school performance andincreased problem behavior.

Consistent with the experimental results, thequalitative interviews also suggested that whenwomen moved from welfare to work and theirincomes increased, they felt their children andfamilies benefited. In addition to increasedemployment and income, two other factors alsoappeared to be critical to these families success:(1) they had stable employment and (2) they felttheir child care arrangements were reliable andtrustworthy, which most of the women definedas kin care. Because these two factors tended tocoincide with work-related increases in income

among the women in our sample, it was difficultto disentangle the relative contributions ofmaternal employment, the income increase thataccompanied the transition to employment,and employment and child care stability tothe improvements these women said wereoccurring in their families and their childrenslives.

As a consequence of their examination of theintersections of the experimental and qualita-tive results, Morris et al. (2005) argued thatthese two lines of research helped them develop

a more-nuanced understanding about how fam-ilies and young children fare as single parentsmake the transition to employment and the

conditions that aid families in coping with thistransition. The qualitative data clearly suggestthat when considering employment, income,and family well-being, researchers must con-sider a host of interacting and possibly counter-

vailing factors. Income increases must besubstantial and not offset by work expendi-tures, employment and income must be stable,the psychosocial and financial benefits ofmaternal employment for children must notbe outweighed by maternal exhaustion and ab-sence, and children must be in reliable care thatis trusted by their mothers. Proponents ofchanges in welfare policy had suggested thatone way maternal employment may benefitchildren is that it would increase the regularityof family routines. These findings suggest that

employment itself does not necessarily lead tosuch regularity. But employment increases thatare accompanied by increases in income,perhaps because they are accompanied byincreases in stable employment and child care,are more likely to result in such benefits forfamily life, and, in turn, childrens develop-ment.

The qualitative findings also point to the roleof employment stability as an important com-ponent of the benefits of increased employmentand income for young children. Interestingly,

employment stability may be responsible forsome of the benefits of the experimental pro-grams that increase both employment andincome. Experimental findings suggest thatearnings supplementation programs increaseemployment stability as they increase the pro-portion of families working. While some ofthese programs increased the number of jobsor spells of employment experienced by pro-gram group members (indicating a greaternumber of employment transitions), they allincreased the length of employment spells. It

appears that at least some of the increase inemployment generated by these programs wasin stable employment. As indicated by the qual-itative research, this may have added somestability to these families lives and led to someof the benefits for children.

The qualitative findings also suggest thatstable and reliable child care is a key interven-ing mechanism in creating the benefits ofincreased employment and income (see alsoScott, London, & Hurst, 2005). Interestingly,the quantitative studies also find evidence that

programs that increase employment and in-come may increase formal child care andparticipation in after-school programs. More

http://-/?-http://-/?-http://-/?-


9/12

specifically, in three of the four earnings sup-plement programs, mothers in the programgroups were more likely to enroll their chil-dren in formal child care or after-school pro-grams and extracurricular activities than were

mothers in the control group. These findingsstand in contrast to those regarding impactsof these earnings supplementation programson measures of parenting behavior. Theseoutcomes were not much affected by theseprograms.

Together, these findings point to the benefitsto children when welfare-reliant parents in-crease both their employment andtheir income.While income in these experimental programsincreased as a result of income supplements,other mechanisms, such as increases in mini-

mum wages and providing supports for edu-cation might also have effects on income.Longitudinal, qualitative interview findingssuggest that increases in employment and in-come may benefit families in part because theyare accompanied by stability of employmentand child care arrangements. By contrast, few,and perhaps countervailing, effects were ob-served when parents increased their employ-ment but not their income, or when incomeincreases were not enough to move the familysubstantially above the poverty threshold.

(b) Adolescent well-being

A second example of the benefits derivedfrom having complementary qualitative andexperimental evidence emerged in the NextGeneration teams work on adolescent well-being (Gennetian et al ., 2002; Gennetianet al., 2004). Members of the research teamworked independently on analyses of the quan-titative and qualitative data, but regularly heldconference calls to discuss emerging findings.

Case studies, developed from the longitudinal,qualitative interview data from Cleveland andPhiladelphia, were sent as they were completedto the other team members who were workingon the experimental analyses.

Consistent with the emerging experimentalfindings, the case studies revealed instances ofadolescents who were faring poorly as theirmothers went to work. The case studies sug-gested that one mechanism by which adoles-cent school performance and other aspects ofwell-being might be compromised was that

adolescents were called upon to provide careto younger siblings. For example, Tina, a 35-year-old African American mother of six living

in Philadelphia, described the negative impactof her welfare-to-work transition on her oldestchild, Tamara. Tina said that because of herwork schedule, Tamara had to be responsiblefor waiting in the morning with the younger

children until the van that took them to theirday care center arrived. The van typically ranlate, which made Tamara 2030 minutes latefor school on a regular basis. Tina said:

Shes late every day for school, every day. And whatthe school says to me is. . .they gotta do what they,whats their policy. Shes gotta stay after school, doher detention. . .or shell lose her credit out of her,out of that morning class cause she didnt get thereon time. So, she feels sad and I feel bad because Igotta be at work at 7:00. She cant be at school by7:00 she cant. We all cant be at the same place

at the same time.

Although Tina felt tremendous guilt for impos-ing this child care burden on her oldest daugh-ter, she felt she had no choice. Tamaras timelyschool attendance meant less for the familysimmediate well-being than Tinas timely atten-dance at work. So, it was Tinas adolescentdaughter who bore the brunt of this burden inthe short term and, potentially, in the longerterm.

Other women told similar stories, which led

the group that was engaged in analyzing theexperimental data to hypothesize that the nega-tive effects they were observing might be morepronounced in those families where there wereyounger children who might need care providedby an adolescent sibling. Quantitative resultsindicated support for this hypothesis. Theexperimental analyses indicate that programsthat increased employment and had negativeeffects on adolescents school outcomes werethe same programs that increased adolescentshome responsibilities.

Most of the experimental studies did not col-lect detailed information about older childrensresponsibilities at home. However, three studiesdid collect information about child care respon-sibilities for the subset of families that had ele-mentary school-aged children. In these studies,adolescents were taking care of their siblings in1428% of the families, with the adolescent beingthe primary source of care in 719% of cases.The majority of adolescents who provided sib-ling care were female (5080%) and ranged inage from 14 to 18 years. Adolescents providedcare to younger siblings up to about 10 hoursper week. In the two programs that increasedmaternal employment and participation in



10/12

employment-related activities and negativelyaffected adolescents school outcomes, adoles-cents were more likely to be caring for elemen-tary school aged siblings, suggesting thatthey did take on some additional responsibility

at home. Adolescents in the third program,which did not adversely affect adolescent schooloutcomes, were no more likely than theircontrol group counterparts to care of youngersiblings. Taken together, the complementaryevidence from the experiments and the qualita-tive interview study provides a compellingcase for the role of increased home responsibil-ities and child care as an important mediatorin the relationship between adult welfare andwork policies and adolescent school outcomes.

5. DISCUSSION

As noted at the outset of this paper, there ap-pears to be increasing interest in mixed-meth-ods, qualitativequantitative research amongpoverty researchers in various contexts (Gibson& Duncan, 2005; Kanbur, 2003; see also theother papers in this special issue). There aremany approaches to doing this work and sub-stantial accumulated wisdom. Numerous recentanalyses address issues related to mixed-meth-

ods research (e.g., Jessor, Colby, & Shweder,1996), including the recently-published Hand-book of Mixed Methods in Social & BehavioralResearch (Tashakkori & Teddle, 2003). Whilea full discussion of approaches to mixed-meth-ods research is beyond the scope of the currentanalysis (e.g., we do not discuss the sequencingof methods, which is discussed in other papersin this special issue and in the Handbook refer-enced above), we believe our accumulating expe-rience suggests several important lessons aboutbest practices and directions for the future.

Consistent with the examples and argumentsput forward by Gibson and Duncan (2005), webelieve it is important that there be tighter inte-gration of quantitative and qualitative compo-nents of studies than we have been able toachieve in the Urban Change and Next Gener-ation projects. While our analyses have yieldedvaluable insights, as discussed above, the factthat there is no formal linkage between thequantitative and qualitative samples in any ofthe studies constrains full integration. We thinkthere are a number of points where linkage can(and arguably should) be made that will en-hance the ability of analysts to integrate thedata and findings.

First, there should be greater emphasis ondesigning and funding studies as mixed-meth-ods studies so that research questions andinstruments can be coordinated from the outsetand fielded in tandem. As Gibson and Duncan

(2005) note, funding constraints meant that thequalitative component of the New Hope evalu-ation was not fielded at the beginning of theexperiment, which compromised to some extentthe integration and analyses they were able toconduct.

Second, we believe that, whenever feasible,the samples should be embedded and formallylinked to allow for better integration of results.This applies both to nonexperimental survey re-search and to experiments. In combination withfielding mixed-methods studies from the outset,

such a linkage has benefits for analysis and en-sures that the time frame and location for bothtypes of data are the same. The two examplesdescribed above, which combine experimentaland qualitative data, combine data from differ-ent settings and time periods (i.e., differentpolicy contexts), which necessarily weakensthe formality of the integration. The UrbanChange survey and qualitative interviewswere collected in the same locations and peri-ods, from women drawn from the same popula-tion, but the samples were not linked, which

again compromises the extent of the integra-tion.

Third, we agree with Gibson and Duncan(2005) that having the same analysts workingon the analysis of the quantitative and qualita-tive data is beneficial and should be pursuedwhenever time, funds, expertise, and goodwillfor data sharing exist. Like the analysts onthe New Hope evaluation, Polit et al. (2000,2001) conducted both quantitative and qualita-tive analyses themselves and found the cross-fertilization across data sources generated new

ideas and insights that might not have other-wise arisen.

A fourth suggestion does not directly emergefrom our experiences, and is perhaps the mostnovel and potentially controversial of oursuggestions. However, we believe it followslogically from the considerations outlinedabove. We believe that there is utility in explor-ing and developing methods for conductingmixed-methods interviews, mixing closed-ended surveys with in-depth interviews.Although there are various issues that need to

be worked out and tested, we believe the inte-gration of funding, design, data collection,sampling, and analytic personnel holds the



11/12

most promise for maximizing the contributionsof mixed-methods research in the context ofpolicy evaluation.

Overall, our experience suggests that mixed-methods research and the integration of quanti-

tative and qualitative research methodologiesand data holds great promise for enhancingthe evaluation of policies and programs. We be-lieve that having high quality qualitative data isimportant even when high quality quantitativedata are collected, as in the case of the UrbanChange survey and the experimental evalua-tions. We also believe it is important forquantitative researchers to pay attention to var-iability as well as central tendencies in theirdata and impacts in their surveys, and to paymore attention than they do to the nature of

the scales that they use. The qualitative data re-mind us that there are different pathways tosimilar ends, as well as difference in experiencesand appraisals of seemingly similar circum-stances. Additionally, the qualitative data intheir own right, but also in relation to the quan-titative data, encourage researchers to considernonmonetary aspects of economic well-being,poverty, poverty appraisal, and policy evalua-tion more systematically.

Although our paper argues for the benefits ofmixed-methods research, we close with a cau-

tion. We must be careful not to raise expecta-tions too high. Doing mixed-methodsresearchdesigning the study, raising themoney for it, collecting the data, and analyzingitare difficult and require different kinds of

initiatives than those that are currently pur-sued, funded, and staffed. Training personnelwill require effort, and data collection and pro-cessing is differentially expensive and time-consuming. There may be questions that onemethod can answer well that the other cannot,and if resources are prioritized toward one setof questions (and method) over the other, it isnot clear that mixed-methods research wouldbe the best approach in that case. Moreover,inductive analyses of qualitative data have thepotential to yield valuable and unanticipated

insights. However, the time involved for suchanalyses may be at odds with both the deduc-tive, a priori framing of research questionsand the time frames for producing results thatcan drive much policy and program analysis.Recognizing the strengths and limitations ofthese methodologies, and making sure that theyare not overly compromised in our efforts tointegrate them will be important to preventoverselling the promise or underselling the pit-falls associated with undertaking mixed-meth-ods research.

NOTES

1. See Quint et al. (1999) for additional details aboutthe Urban Change study.

2. One of the experiments involved a Canadian welfarereform demonstration program and therefore had asomewhat different institutional structure. Additionaldetails about these experimental evaluations are avail-able in Morris, Huston, Duncan, Crosby, and Bos(2001) and Gennetian et al. (2002, 2004).

3. The 18 items in the HFSS ask respondents toindicate whether, as a result of inadequate financialresources, they experienced any of the following in theprior 12 months (Carlson et al., 1999, p. 513S). Theitems are arrayed in order of severity: worried foodwould run out; food bought did not last; unable to

afford balanced meals; child(ren) fed few, low-costfoods; adult(s) cut size or skipped meals; could not feedchild(ren) balanced meals; respondent ate less than felts/he should; adult(s) cut, skipped meals in 3+ months;child(ren) not eating enough; respondent hungry but didnot eat; respondent lost weight; cut size of child(ren)smeals; adult(s) not eat for whole day; child(ren) hungry;adult(s) not eat for whole day in 3+ months; child(ren)

skipped meal; child(ren) skipped meal in 3+ months;child(ren) not eat for whole day.

4. All names used in this paper are pseudonyms.

5. See also Scott, Edin, London, and Mazelis (2001,chap. 8) and London et al. (2004).

REFERENCES

Anderson, S. A. (1990). Core indicators of nutritionalstatus for difficult-to-sample populations. Journal ofNutrition, 120, 15591600.

Brock, T., Coulton, C., London, A., Polit, D., Richburg-Hayes, L., Scott, E., et al. (2002). Welfare reformin Cleveland: implementation, effects, and experiences



12/12

of poor families and neighborhoods. New York:MDRC.

Carlson, S. J., Andrews, M. S., & Bickel, G. W. (1999).Measuring food insecurity and hunger in the UnitedStates: development of a national benchmark andprevalence estimates. Journal of Nutrition, 129,

510S516S.Frongillo, E. A. (1999). Validation of measures of food

insecurity and hunger. Journal of Nutrition, 129,506S509S.

Gennetian, L. A., Duncan,G. J., Knox, V. W., Vargas, W.G., Clark-Kauffman, E., & London, A. S. (2002). Howwelfare and work policies for parents affect adolescents:A synthesis of the research. New York: MDRC.

Gennetian, L. A., Duncan, G. J., Knox, V. W., Vargas,W. G., Clark-Kauffman, E., & London, A. S. (2004).How welfare policies can affect adolescents. Journalof Research on Adolescents, 14, 399423.

Gibson, C., & Duncan, G. J. (2005). Qualitative/

quantitative synergies in a random-assignment pro-gram evaluation. In Thomas Weisner (Ed.), Discov-ering successful pathways in childrens development:New methods in the study of childhood and family life(pp. 283315). Chicago: University of Chicago Press.

Jessor, R., Colby, A., & Shweder, R. A. (Eds.) (1996).Ethnography and human development: Context andmeaning in social inquiry. Chicago: University ofChicago Press.

Kanbur, R. (2003). Q-squared: Qualitative and quantita-tive methods of poverty appraisal. New Delhi: Per-manent Black.

London, A. S., Scott, E. K., Edin, K., & Hunter, V.(2004). Welfare reform, work-family tradeoffs, and

child well-being. Family Relations, 53(2), 148158.Morris, P. A., Huston, A. C., Duncan, G. J., Crosby, D.

A., & Bos, J. M. (2001). How welfare and workpolicies affect children: A synthesis of research. NewYork: MDRC.

Morris, P. A., Scott, E. K., & London, A. S. (2005).Effects on children of parents transitions fromwelfare to employment: integrating quantitative andqualitative research. In Jill Duerr Berrick, & BruceFuller (Eds.), Good parents or good workers? How

policy shapes families daily lives (pp. 87116). NewYork: Palgrave Macmillon.

Polit, D. F., London, A. S., & Martinez, J. M. (2000).Food security and hunger in poor, mother-headed

families in four US cities. New York: MDRC.

Polit, D. F., London, A. S., & Martinez, J. M. (2001).The health of poor urban women: Findings from the

project on devolution and urban change. New York:MDRC.

Quint, J., Edin, K., Buck, M. L., Fink, B., Padilla, Y. C.,Simmons-Hewitt, O., et al. (1999). Big cities and

welfare reform: Early implementation and ethno-graphic findings from the project on devolution andurban change. New York: MDRC.

Raphael, J., & Tolman, R. M. (1997). Trapped bypoverty/Trapped by abuse. Ann Arbor: University ofMichigan, Research Development Center on Pov-erty, Risk, and Mental Health.

Scott, E. K., Edin, K., London, A. S., & Kissane, R. J.(2004). Unstable work, unstable income: implicationsfor family well-being in the era of time-limitedwelfare. Journal of Poverty, 8(1), 6188.

Scott, E. K., Edin, K., London, A. S., & Mazelis, J. M.(2001). My children come first: welfare-reliant

womens post-TANF views of work-family trade-offs and marriage. In G. J. Duncan, & P. L. Chase-Lansdale (Eds.), For better and for worse: Welfarereform and the well-being of children and families(pp. 132153). New York: The Russell Sage Foun-dation.

Scott, E. K., London, A. S., & Hurst, A. (2005). Out oftheir hands: patching together care for children whenparents move from welfare to work. Journal ofMarriage and the Family, 67, 369385.

Scott, E. K., London, A. S., & Myers, N. A. (2002a).Living with violence: womens reliance on abusivemen in their transitions from welfare to work. In N.Gerstel, D. Clawson, & R. Zussman (Eds.), Families

at work: Expanding the bounds (pp. 302316). Nash-ville: Vanderbilt University Press.

Scott, E. K., London, A. S., & Myers, N. A. (2002b).Dangerous dependencies: domestic violence in thecontext of welfare reform. Gender & Society, 16(6),878897.

Tashakkori, A., & Teddle, C. (Eds.) (2003). Handbook ofmixed methods in social & behavioral research.Thousand Oaks: Sage Publications.

Tolman, R. M., & Raphael, J. (2000). A review ofresearch on welfare and domestic violence. Journal ofSocial Issues, 56(4), 655681.


schwartz quantitative and qualitative data

Documents