NATIONAL CENTER FOR EDUCATION STATISTICS

Technical Report September 1996

National Homehold Education suNey

Use of CognitiveLaboratoriesand Recorded Interviewsin the National HouseholdEducation Survey

NationalHouseholdE d u c a t i o ns urvey

Mary Jo NolinWestat, Inc.

Kathryn ChandlerNational Center for Education Statistics

U.S. Department of EducationOffice of Educational Research and Improvement NCES 96-332

For sale by the U.S. Government Printing OfficeSuperintendent of Documents, Mad Stop: SSOP, Washington, DC 20402-9328

ISBN 0-16 -048830-3

i

U.S. Department of EducationRichard W. RileySecretary

Office of Educational Research and ImprovementSharon P. RobinsonAssistant Secretary

National Center for Education StatisticsPascal D. Forgione, Jr.Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting,analyzing, and reporting data related to education in the United States and other nations. Itfulfills a congressional mandate to collect, collate, analyze, and repott full and completestatistics on the condition of education in the United States; conduct and publish reports andspecialized analyses of the meaning and significance of such statistics; assist state and localeducation agencies in improving their statistical systems; and review and report on educationactivities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent,reliable, complete, and accurate indicators of education status and trends; and report timely,useful, and high quality data to the U.S. Department of Education, the Congress, the states,other education policy makers, practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that isappropriate to a variety of audiences, You, as our customer, are the best judge of our successin communicating information effectively. If you have any comments or suggestions about thisor any other NCES product or report, we would like to hear from you. Please direct yourcomments to:

National Center for Education StatisticsOffice of Educational Research and ImprovementU.S. Department of Education555 New Jersey Avenue NWWashington, DC 20208-5574

September 1996

Suggested Citation

U.S. Department of Education. National Center for Education Statistics. Use of Cognitive Laboratories and Recorded/nterviews in the National Household Education Survey, NCES 96-332, by Mary Jo Nolin and Kathryn Chandler.Washington, DC: 1996.

Contact:Kathtyn Chandler(202) 219-1767

For single copies, call:National Education Data Resource Cer!ter(703) 845-3151

Table of Contents

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Overview of Data Quality intheNHES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Overview ofCognitive Laboratory Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Cognitive Laboratory Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Cognitive Laboratory Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Individual Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Focus Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Analysis of Cognitive Laboratory Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Example s o f Cogn i t i ve Labo ra to ry Resea rch Conduc t ed forthe NHES . . . . . . . . . . . . . . . . . . . . 8Exploring the Feasibility of a Research Topic and the Appropriateness ofRespondents: School Safety and Discipline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Testing Survey Instruments.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Testing the NHES:91 Early Childhood Questionnaire . . . . . . . . . . . . . . . . . 11Testing the NHES:93 School Readiness and School Safety and DisciplineQuestionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Evaluating Respondent Comprehension During Data CollectionThrough Taped Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Content Analysis of Taped Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Interviewer-Respondent Behavior Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Appendix A: Moderator’s Topic Guide for Parent Groups . . . . . . . . . . . . . . . . . . . . . . . . . 20

Appendix B: Moderator’s Topic Guide for Adolescent Groups . . . . . . . . . . . . . . . . . . . . . 27

Appendix C: Form for Behavior Coding of Interviewer-Respondent Interactionfrom Taped Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...35

. . .111

Background

The National Household Education Survey(NHES) is a data col lect ion system of theNational Center for Education Statistics (NCES),which hasasits legislative mission the collectionand publication of data on the condition ofeduca t ion i n t he Nation. T h e NHES isspecifically designed to support this mission byproviding information on those educational issuesthat are best addressed by contacting householdsrather than schools or other educat ionalinstitutions. The NHES provides descriptive dataon the educational activities of the U.S.population and offers policymakers, researchers,and educators a variety of statistics on thecondition of education in the United States.

The NHES is a telephone survey of thenoninstitutionalized civilian population of theUnited States. Households are selected for thesurvey using random digit dialing (RDD)methods, and data are collected using computer-assisted telephone interviewing (CATI)procedures. Approximately 60,000 householdsare screened for each administration, andindividuals within households who meetpredetermined criteria are sampled for moredetailed or extended interviews. The data areweighted to permit estimates of the entirepopulation. The NHES survey for a given yeartypically consists of a Screener, which collectshousehold composition and demographic data,and extended interviews on two substantivecomponents addressing education-related topics.

The primary purpose of the NHES is to conductrepeated measurements of the same phenomenaat different points in time, although one-timesurveys on topics of interest to the Departmentof Education may a l s o b e conducted.Throughout its history, the NHES has collecteddata in ways that permit estimates to be trackedacross time. This includes repeating topicalcomponents on a rotating basis in order to

provide comparative data across survey years. Inaddition, each administration of the NHES hasbenefited from experiences with previous cycles,resulting in enhancements to the survey proceduresand content. Thus, while the survey affords theopportunity for tracking phenomena across time, it isalso dynamic in addressing new issues and includingconceptual and methodological refinements.

A new design feature of the NHES programimplemented in the NHES:96 is the collection ofdemographic and educational information onmembers of all screened households, rather than justthose households potentially eligible for a topicalcomponent. In addition, this expanded screeningfeature includes a brief set of questions on an issueof interest to education program administrators orpolicymakers. The total Screener sample size issufficient to produce state estimates of householdcharacteristics for the NHES :96.

Full-scale implementations of the NHES have beenconducted in 1991, 1993, 1995, and 1996. Topicsaddressed by the NHES:91 were early childhoodeducation and adult education. The NHES:93collected information about school readiness andschool safety and discipline. The 199 I componentswere repeated for the NHES :95, addressing earlychildhood program participation and adult education.Both components underwent substantial redesign toincorporate new issues and deve lop newmeasurement approaches. In the NHES:96, thetopical components are parent/family involvement ineducation and civic involvement. The NHES:96expanded screening feature includes a set ofquestions on public library use.

In addition to its topical components, the NHESsystem has also included a number of methodologicalinvestigations. These have resulted in technicalreports and working papers covering diverse topicssuch as t e l ephone undercoverage bias, proxyreporting, and sampling methods. This series oftechnical reports and working papers, which includesthis report, provides valuable information on ways ofimproving the NHES.

This report describes cognitive laboratoryresearch and analysis of taped interviews in theNHES. The description of these data qualityactivities is illustrated with selected examplesfrom survey components in the NHES:91 and theNHES:93. The report is intended to give anoverview of these methods as they are used in amajor education data collection, not to describethe results of the activities in detail.

Overview of Data Quality Activities in theNHES

Many social scientists rely on data from large-scale surveys to conduct their research, and it isassumed that the data are accurate and reliable.However, the quality of survey data is affectedby both sampling and nonsampling error.Substantial research has been conducted tomeasure and to find ways of reducing samplingerror, error that results from surveying only aportion of the inference population. Otherresearch has been conducted to find ways ofreducing nonsampling error, error that arisesfi-om behavior of the respondent or theinterviewer or both. In order to reduce this typeof error, researchers expend considerable effortto 1) design questions that are clear and easilycomprehensible to the respondent, 2) reducenonresponse among sampled persons, and 3)train interviewers to ensure consistency in theway that questions are asked (Groves 1989, 1-6).

Data quality is a central issue for the NHES, andcare is taken at each stage of the survey designand data collection process to secure high qualitydata. Several methods have been used in theNHES to reduce nonsampling error by increasingthe comprehensibility of survey questions andmaximizing interviewer consistency. Eachmethod elicits different information and suggestsspecific means of improving the survey (Presserand Blair 1994). For instance, for some surveytopics, cognitive research explores the salience ofthose topics with participants drawn from theintended population of interest. Then, drafts

of NHES questionnaire items are presented forreview and comment to Technical Review Panelscomposed of both methodological experts andexperts in the content areas from which the surveytopics have been drawn. Cognitive researchconsisting of intensive interviews and focus groupsis conducted after survey items are developed, andchanges are made to the instruments based on thefindings from this research. Following theconclusion of cognitive laboratory research, theinstruments are programmed for computeradministration, and one or more pretests areconducted using live, computer-assisted telephoneinterviews (CATI). These efforts result inadjustments to the interview instruments. Pretestresults come from both the actual responses and fromwhat is learned about respondent comprehension andrespondent-interviewer interaction through extensivemonitoring by project staff and in-depth debriefingsof interviewers.

During data collection, interviews are regularlymonitored by project staff and Telephone ResearchCenter staff, and interviewers receive quick andspecific feedback on their performance. In addition,a small number of interviews have been recorded,and staff have analyzed the observed interviewer-behavior interaction. These activities are used inconjunction with the results of a reinterview of asubsample of the respondents and a comparison ofthe NHES findings with other surveys to investigateand offer probable explanations for any unexpectedresults and to improve future administrations of thesurvey (Brick and West 1992; Brick, Rizzo, andWernimont forthcoming; Brick, Collins, Nolin, andDavies forthcoming).

Two data quality activities of the NHES, cognitivelaboratory research and analysis of recordedinterviews, are the focus of this report. Cognitivelaboratory research plays a key role during thedesign of each NHES survey. This research utilizesthe methods of cognitive psychology to increaseunderstanding of the ways that respondentscomprehend survey instructions and questions, recallrequested information, and respond to the influenceof word and question order. It has been used in the

NHES to explore the feasibility of potentialsurvey topics, to assess the reliability of classesof respondents, and to test questionnaires inorder to reduce measurement error. Maximizingthe clarity and conciseness of the questions is aparticular concern for a survey like the NHES, inwhich a random sample of persons is surveyedwithout any previous introduction to the study.Respondents are typically unfamiliar withsurveys, some have relatively low educationalattainment, and all possess perceptions unique totheir own social experience. An uilderstandirigof respondents’ general perceptions of the surveytopics and the salience of the issues beingexplored, as well as knowledge about thestrategies that respondents are likely to use toanswer survey questions, has the potential ofreducing measurement error. So also doescreating an interview context in whichrespondents are likely to respond accurately interms of their own experience (Schwartz 1995).

The analysis of recorded interviews conductedduring the NHES:91 and the NHES:93 hasprovided a useful assessment of data quality.During data collection for those surveys, alimited number of interviews were tape-recordedand later analyzed to systematically assess theinterview process and to evaluate respondentcomprehension during the interview. A differentmethod of analysis was used each year. For theNHES:91, tapes were analyzed for content; forthe NHES :93, each interviewer-respondentinteraction was coded.

Following a brief account of the emergence ofcognitive laboratory research as a tool for surveyresearchers and a description of cognitivel a b o r a t o r y m e t h o d s and procedures, the

application of these procedures in the NHES:91and NHES:93 is described. An account of theassessment of interviews recorded during datacollection in the NHES:91 and NHES:93 is thenpresented.

Overview of Cognitive LaboratoryResearch

Cognitive laboratory research, one of the manyefforts to reduce nonsampling error in surveys, drawson the theories and methods of cognitive psychology.Collaboration between survey researchers andcognitive psychologists began in 1978. To a greatextent this collaboration was sponsored and fundedby government agencies in the United States, theUnited Kingdom, and West Germany in order toimprove the quality of the survey data that served asthe bas i s fo r po l i cy decisions. Cognitivepsychologists were motivated to join in theinterdisciplinary effort because they were providedwith the opportunity to study cognitive processes ina natural setting rather than in the artificial setting ofthe laboratory (Jobe and Mingay 1991). Examplesof this research (as cited by Jobe and Mingay)include research on autobiographical memory,cognitive interviewing techniques, experientialmemory, the influence of response alternatives onresponses, memory for dates and events, questioncomprehension, question ordering and wording, andresponse order. Most of the research was publishedin the 1980s. According to Lessler, Tourangeau, andSalter (1989), despite government funding of earlycognitive research, cognitive laboratory procedureswere underutilized in survey research for Federalagencies. Their own program for investigating thecognitive aspects of survey methodology at theNational Center for Health Statistics, described inLessler, Bercini, e t al. (1986) a n d Lessler,Tourangeau, and Salter ( 1989), was an exception. Inthis decade, however, more Federal studies haveincorporated cognitive laboratory research (see, forexample, Bates and DeMaio 1990; Campenelli et al.199 1; Cantor and Edwards 1992; Cantor et al.,1995; Dippo and Norwood 1992; Jenkins, Ciochetto,and Davis 1992; Jenkins 1992). Cognitive laboratoryresearch in the NHES has been part of this effort.

In the next sections, cognitive laboratory methods arediscussed first, followed by a description of the

procedures that are used in combination withthose methods, and then by a brief account ofhow data from cognitive laboratory research areanal yzed.

Cognitive Laboratory Methods

Forsythe and Lessler (1991) provide a valuableclassification of cognitive laboratory methodsfrom which this summary is drawn. Theydifferentiate the methods used in cognitivelaboratory research along two dimensions: thetiming of the cognitive tasks, and the level towhich the respondent must attend to his or hercognitive processes in order to provide theinformation requested. Some methods are calledconcurrent methods because the cognitive tasksof providing information and articulatingmeaning and/or recall processes are performed atthe same time. The respondent’s attention isspecifically directed to the response process atthe same time he or she is required to retrieveand provide information. The respondent mustnot only respond to the question, but also talkabout the mental steps involved in retrieving andsort ing the requested information. Somecognitive laboratory methods are termed delayedbecause they involve cognitive tasks that ther e s p o n d e n t p e r f o r m s after the process ofretrieving and providing information. That is,the respondent first provides the information andthen, shortly afterwards or when a somewhatlonger period of time has elapsed, the respondentreports on his or her cognitive processes.Selection of a concurrent or delayed method isguided by the specific goals of the research.

Two concurrent methods, concurrent think-alouds and concurrent probing, are frequentlyused in cognitive laboratory research. Theconcurrent think-aloud method requires therespondent to verbalize his or her thoughtsspontaneously while answering the interviewquestions. This method is based on protocolanalysis (see Ericsson and Simon 1980 andEricsson and Irwin 1984) in which participantsare asked to think aloud as they solve a problem.When this approach is used in cognitive

laboratories to test a questionnaire, respondents areasked to think aloud as they answer each question inorder to reveal the strategies they use to recall theinformation. They talk through the steps they taketo produce an answer while they are producing theanswer. For example, when asked a question thatrequires recall over an extended period, a respondentmight think in terms of months or seasons, and thatwould become apparent as he or she spoke whileretrieving the information. (For example, arespondent might say, “Let’s see, in June we went tothe beach, and I wasn ‘t taking courses then. 1 musthave started in July.”) This activity is not an easyone for many respondents because the thoughts thatthey are asked to verbalize are usually not expressedaloud. Typically, we do not think about “how wethink” when generating responses to questions, andsome respondents do not articulate their thoughtseasily, even when they are attending to them. Avariant on the think-aloud method, in which theresearcher takes a more directive role, is theconcurrent probe. Immediately following aresponse, the participant is asked about some aspectof the question or response in order to elicit specificand detailed information on response strategy.

Cognitive laboratories a l s o u s e delayed, o rretrospective, methods. Delayed methods tapinformation about re sponden t knowledge,com prehens ion, and recall, as well as generalperceptions about survey items that may affectmotivation, particularly those that are formed overtime (Forsythe and Lessler 1991). These methodscall for the administration of the survey underconditions similar to those encountered during theactual data collection. Then, after all questions areanswered, information about the survey and theresponse process is elicited. The researcher may useeither nondirective or directive retrospective probesto draw out this information. F o r example, aresearcher may refer to a particular answer and askthe respondent to think-aloud retrospectively,explain what he or she was thinking whenresponding. Or, in a more directive probe, theresearcher may refer to a particular term that therespondent used and ask for the respondent’sdefinition of that term in more detail. The elapsedtime between administration of the interview and the

4

procedures to measure cognitive processes variesfrom immediately following administration of thesurvey to a few days. In the former instance,information requested of the respondent residesin short-term memory and is generally easilyretrievable. In the latter case, the time intervalrequires the respondent to reconstruct what he orshe was thinking at the time the question wasasked. It also permits access to opinionsregarding the survey that have been formed overthe time between asking the question andprobing for more information.

Both concurrent and delayed methods are used incognitive laboratory research for the NHES.Concurrent methods are most often employed intesting questionnaire items. Delayed methodsare most commonly used in the early stages ofthe design of an NHES survey to exploreresearch topics and determine the appropriaterespondents for those topics. Delayed methodsmay also be used to test questionnaire items.

Cognitive Laboratory Procedures

Concurrent and delayed methods can beemployed with different cognitive laboratoryprocedures, creating an array of tools for theresearcher (table 1). The procedures commonlyused are individual, in-depth interviews andfocus groups. In the NHES, focus groups havebeen used during the earliest stages of surveydesign to assess the feasibility of potential

research topics and the suitability of a class orclasses of respondents, given the topic. In-depthface-to-face interviews and focus groups have beenused to evaluate the clarity and efficacy ofquestionnaire items and to improve and test versionsof the instruments. The following sections describeinterviews and focus groups and the specificcognitive laboratory methods typically used withthese procedures.

Individual Interviews

Individual interviews are an important part ofcognitive laboratory research. During cognitiveinterviews, the researcher is able to interact with oneparticipant at a time, and give his or her fullattention to the information offered by theparticipant. This procedure affords the researcher theflexibility to adapt the methodology to the participantand to pursue valuable and perhaps unanticipatedlines of inquiry. Both concurrent and delayedcognitive laboratory methods can be used withindividual interviews.

Cognitive laboratory activities to test NHES surveyitems usually include concurrent think-aloudinterviews in which respondents verbalize theirresponse strategies as they answer each question.The absence of intervention by the interviewerpermits unexpected information to emerge.However, the concurrent think-aloud method caninfluence the responses that are obtained. Becausethe respondent is attending to the process ofretrieving information and deciding upon response

Table 1--- Cognitive laboratory techniques used in the NHES

Methods

Procedures Concurrent Delayed

Individual interviews Concurrent think-aloud Retrospective think-aloudConcurrent probing Retrospective probing

Group discussion Focus group

5

strategies, he or she may be more carefil insearching for accurate information than a surveyrespondent under typical interviewing conditions.

Although respondents are instructed, in the think-aloud method prior to administration of thequestionnaire, the procedure is awkward forsome respondents because they must considerboth the response and the strategy for obtainingthe response. Some respondents are simplyunable to report on their response strategieswhile answering questions. In addition, becausesome respondents focus intently on generalizingan out-loud response to a question, they mayhave difficulty focusing on the subsequentquestions, which can reduce the accuracy of theirself-reporting.

In these cases, another concurrent method, theconcurrent probe, may be used. During theinterview, the respondent answers questionswithout having to attend to retrieval processes atthe same time. However, just after a response isgiven, the interviewer probes to ascertain detailsof the response process including relevantinformation about knowledge and motivation torespond.

Delayed cognitive methods may also be used inconjunction with individual interviews. In theseinterviews, after the respondent has answered allquestions, he or she is debriefed about his or herresponses. Usually certain questions or groupsof questions are targeted for debriefing. Delayedmethods used with interviews can benondirective, as in the retrospective think-aloudtechnique in which the respondent is simplyasked to recall and state what were his or herthoughts when responding to particular questions.Retrospective probing is more structured;information about specific aspects of content andtime domains is elicited by the researcher.Questioris about the meaning of terms, thetechniques used to r eca l l information, orreference periods for time-related questionsmight be asked. The respondent’s assessment ofthe accuracy of the original responses can alsobe obtained.

Focus Groups

Focus groups are a cognitive laboratory method thatcan generate valuable insight on respondentcomprehension, recall, and motivation to answersurvey items. In the NHES, they are a means ofexploring potential survey topics, establishing theappropriate respondents to the surveys, andevaluating drafts of interviews. First used in socialscience research over 30 years ago, focus groups aresmall groups of 6 to 10 individuals who are broughttogether to react to a stimulus situation or discuss aparticular issue or set of issues (see Merton, Fiske,and Kendall 1956 for a discussion of the early use offocus groups). Focus group participants are selectedaccording to criteria determined by the research topicand the research goals. Groups are often composedof participants who share many sociodemographiccharacteristics because interacting with others whoare similar can encourage self-disclosure (Merton,Fiske, and Kendall 1956, 137-1 38). Alternatively,especially when the number of groups to beconducted on a topic is limited, conducting groupshomogeneous on several variables (e.g., older whitemales, younger white males, older black males, olderwhite females, etc.) may not be feasible. In suchinstances, diversity on sociodemographic variables isaccepted, and homogeneity with regard to theparticular research topic can be sought. Forexample, for a study in which it is anticipated thatexperience will vary by the grade of the student, onemight want to include youth from a narrow graderange but who attend public and private schools, whoare black, white, and Hispanic, and who live inurban and suburban areas.

Protocols designed with the stage of development ofthe survey and the particular research purpose inmind are created to guide focus group discussions.They consist of unstructured, semistructured, orstructured questions and probes. The intent is tocapture a range of responses from participants, toelicit information that is as specific as possible, toencourage participants to contribute perceptions thatare both cognitively and emotionally meaningful, andto learn about the personal context from which theirresponses are drawn (Merton, Fiske, and Kendall1956). For example, protocols for focus groupsconducted for exploratory research will contain open-ended questions to encourage the respondents to

6

voice their own opinions and perceptions asu n c o n s t r a i n e d a s p o s s i b l e b y t h econceptualizations of the researchers. Protocolsfor focus groups conducted for the purpose oftesting survey instruments may use moredirective techniques specified by semistructuredor structured questions. Typically, participantsin these focus groups are prepared for thediscussion by having the survey administered tothem beforehand. The protocol is constructed todecompose responses and explore comprehensionand time references. The focus group discussionencompasses thoughts participants have had inthe period intervening between the interview anddiscussion, and elicits attitudes about the surveythat may have developed over time.

Focus groups are led by trained moderators whodirect the discussion but allow the respondents tovoice their own opinions and perceptions aboutthe survey. This method affords participants thebenefit of group interaction. In exploratoryresearch, group discussion can yield constructiveinformation about respondents’ generalknowledge of survey topics and related issuesand can help researchers to use familiar termswhen designing questionnaire items. In groupsconducted to test questionnaires, comments bysome group members may help others recallthoughts and perceptions that occurred at thetime they responded to the survey or encouragethem to offer opinions about the survey. Becauseseveral respondents take part in the discussionsimultaneously, focus groups offer researchers alabor- and time-efficient method of gatheringqualitative data to improve survey content andreduce measurement error.

Focus groups have some limitations, however.Because several participants are involved in thetask, the researcher must pay close attention togroup process as well as to the researchobjective. Some participants may attempt todominate the group, subvert the task, fulfillpersonal goals, or otherwise adhere to their ownagenda. Alternatively, other participants may be

swayed by group opinion or may be reluctant todivulge information in front of others that theywould reveal to a researcher in an individualinterview. Focus group participants must be assuredof the confidentiality of their opinions, which are tobe used for research purposes only. Focus groupdata, whether in the form of notes or video- oraudiotapes of the proceedings, must be treated withextreme care by researchers to protect thatconfidentiality. Also, focus groups, like the othermethods discussed here, do not yield findings thatare generalizable to a larger population, so theirresults must always be interpreted with caution.

Analysis of Cognitive Laboratory Data

Cognitive laboratory activities are typicallyaudiotaped or videotaped to provide a record of theresearch. Findings are drawn from content analysisof the tapes, or, in some instances, of transcriptionsof the tapes. However, analysis of transcriptionscannot take the place of listening to or viewing thetapes or observing the sessions because the contextof a person’s remarks (tone of voice, posture, etc. ) isalso useful information. In addition to the tapes,data may be generated by notes taken by an observerduring a focus group and notes recorded by theinterviewer or focus group moderator immediatelyfollowing the activity.

The specific analytical procedure used by theresearcher depends upon the research goal. Datamay be precisely coded, and words and phrasescompared and contrasted in research conducted togenerate conceptual categories, or the data may besubjected to a more general analytical search form a j o r points. Data collected to assesscomprehension, recall, and the sensitivity ofquestionnaire items can be examined for the range ofanswers, specificity of ideas, and patterns of response(Krueger 1988, 106-121). Although findings areoflen based upon responses that emerge repeatedlythroughout the research, valuable insights can begenerated by one comment made by a single researchparticipant. Therefore, a search for key comments,not only patterns, is part of the analysis of cognitivelaboratory data.

In the NHES, individual interviews areaudiotaped, and focus groups are both audio- andvideotaped. An observer also sits behind a one-way mirror and takes notes during the focusgroup. The interviewer or focus groupmoderator records his or her notes on theresearch immediately following the activity.Answers to the general questions that guide theresearch and are embedded in the protocol aredrawn from reviewing the tapes and the notes ofthe observer and the interviewer or moderator.Recommendations are based on these findings.

Examples of Cognitive LaboratoryResearch Conducted for the NHES

Cognitive laboratory research has been conductedin the design phase of each NHES datacollection, and the role of this research in thedevelopment of each survey has increased overtime. The examples presented here are notintended to be comprehensive with regard to theissues explored in the cognitive laboratories, theprocedures used during the design of each surveycomponent, or the findings that emerged fromthe research. However, they illustrate ways thatcognitive laboratory methods and procedureshave been used to investigate various issues fordifferent topical components with beneficialresults for the overall data quality of the NHES.

Exploring the Feasibility of a Research Topicand the Appropriateness of Respondents:School Safety and Discipline

Cognitive laboratory research was undertaken toexamine the feasibility of school safety anddiscipline as a topic for the NHES:93. Interestin School Safety and Discipline as a potentialtopic for the NHES intensified when this topicbecame one of the President’s and Governors’National Education Goals. While some data onstudent drug use, victimization, and teachers’perceptions of school safety were published(National Education Goals Panel 1991, 1992),there was a need for more data, not only onstudent victimization, but also on overall

perception of the “learning environment” at school.In addition, NCES was interested in placing parents’and students’ reports in context by gathering data onparent approval or disapproval of alcohol use bytheir children, the extent of parent support of schoold i sc ip l ine policy, and certain school andneighborhood characteristics. Substantive expertsagreed with the need for more data but thought thatobstacles to collecting it might exist. There wereconcerns that parents may not pay much attention toissues related to safety at school, that students maynot be reliable respondents because of their age, andthat concepts used by educational policymakers andresearchers would not be readily understood byparents and students.

Because of the open-ended format and the potentialfor new information to emerge from groupinteraction, focus groups were chosen as theappropriate methodology to explore the salience ofthe issues and parent and student comprehension ofthe concepts related to school safety and discipline.Professionals had identified areas related to safetyand discipline in school for which data were needed.Small groups of 8 to 10 parents or students werebrought together to discuss aspects of the topic andto test the ability of potential respondents to providethe data. The group format encouraged a range ofopinions, with the comments of some members ofthe groups serving as stimuli for others.

Participants. During May and June of 1991, sevenfocus groups were conducted to test the feasibility ofa School Safety and Discipline topical component forthe NHES. All of the groups were conducted nearthe Washington, D C o r Baltimore, Maryland,metropolitan areas. Participants were recruitedthrough flyers posted in public places such as officebuildings and grocery stores, and through anadvertisement placed in a local newspaper inGaithersburg, Maryland, a suburb approximately 35minutes outside of Washington, D.C. Participantswith experience in inner-city schools were recruitedthrough a market research firm in Baltimore,Maryland. Volunteers were administered a briefscreener so that participants could be selectedaccording to specific characteristics that the includedgrade of student, type of school, race/ethnicity, and

parents’ occupation. Four of the groups werecomposed of parents; one group of parents ofstudents in grades 1 through 6, two of parents ofstudents in grades 7 through 9, and one ofparents of students in grades 10 through 12.Groups contained both mothers and fathers, butnot two parents from the same family. Inaddition, three groups of students wereconducted, two composed of students in grades7 through 9, and one of students in grades 10through 12. Five of the groups (three parent andtwo student groups) were conducted inGaithersburg, and two (one parent and onestudent group) were conducted in Baltimore.The prior written consent of a parent wasobtained for each student who participated in afocus group, and all participants, parents andstudents, gave written consent before thediscussion began. Each of the participants waspaid an honorarium.

The groups were racially and ethnically diverse,and respondents were f rom d i f fe ren tsocioeconomic levels as measured by occupation.Twenty-five of the parent participants and 18 ofthe youth participants were white, and 10 of theparents and 6 of the youth were black orHispanic. Approximately half of the parentparticipants were in relatively low-wageoccupations that did not require postsecondaryeducation, for example, automobile salesman ormaintenance worker. About half of the studentparticipants came from households in which thehighest occupation was also a relatively low-wage one. Mothers and fathers (from differentfamilies) participated, as did male and femalestudents.

Protocols. Moderator’s topic guides weredeveloped to guide the focus group discussions(see Appendix A and Appendix B). They wereorganized around four broad topics: safety in theschools, alcohol and other drug use, alcohol anddrug education, and school discipline policy.

The protocols contained both structured andunstructured questions and probes designed toelicit in-depth and personal discussion of each ofthe topics.

Findings. The focus groups demonstrated thatschool safety and discipline was a feasible topic forthe NHES----one of great interest to both parents andstudents. Most parents held understandings of thekey concepts consistent with those of theprofessionals who had been consulted in the earlieststage of the design, There were few differencesbetween the suburban and urban groups with regardto level of awareness and concern about unsafeincidents at school, although urban parents andstudents regarded lack of safety at school as moreprevalent and inevitable than did the others.

One of the points that the focus group researchaddressed was who would be appropriate respondentsfor this topical component. It was believed thatsafety and discipline could be of concern to parentsand students at lower as well as higher grade levels;however, it had not been determined at what gradelevel parents began to focus on those issues. Asintended, during the course of the focus groupdiscussion, the meanings that the participantsattached to safety and discipline in school emergedrelatively unconstrained by the perceptions ofresearchers. The focus group composed of parentsof elementary school students revealed a range ofconcern about safety issues linked to the age of thechild. Parents of younger elementary schoolstudents, children in grades 1 through 3, tended tothink about safety in terms of freedom from outsideintruders who could gain access to the school andpotentially harm their children. These parents wereless concerned with serious disruption in theclassroom, fights between students, and weapons anddrugs in school. Parents with older elementaryschool children begin to anticipate the occurrence ofunsafe conditions in schools and serious lack ofdiscipline in the classrooms. As children entered the4th, 5th, and 6th grades, the issues related todiscipline, fighting, and even weapons in schoolbecome more salient for parents. Parents whosechildren were to enter middle school in 6th gradeexpressed more anxiety than parents of children whowould remain in an elementary school through 8thgrade, and parents of students in junior high (7th-9thgrade) and high school (9th - 12th grade) saw thesafety of their children from harm or threats of harmby other students as being of central concern.

The appropriate parent respondents for thissurvey would be those who were immediatelyconcerned about safety and discipline at schoolas it was identified in the National Goals.Discussions in the focus groups established thatparents of students in grades 6 through 12 werethose parents; they would be the best respondentsfor a broad range of questions on safety and onsuch issues as drug availability and use.However, the emerging concerns of parents ofyounger students argued for including them inthe survey population also. Their commentsshowed that they were beginning to look aheadto their children’s middle or junior high schoolyears. The focus group findings led to thedecision to include parents of students in grades3 through 5 as respondents for this surveycomponent. Based on the findings that somesubjects (e.g., alcohol and drugs or weapons atschool) were not salient for this population, itwas decided to administer to them a subset ofquestions focused on the safety issues mostrelevant to the parents of younger students, forexample, classroom disruption and bullying.

It was hypothesized that parents do notnecessarily have full information fi-om theirchildren or fi-om other sources about disruptionand violence in school, the availability of alcoholor other drugs, the school discipline policy, orthe alcohol/other drug education at their child’sschool, and the focus group discussionssupported this conjecture. Some parents reportedconsulting with a child after volunteering for thecognitive laboratory research, and therebylearning that some of their ideas about safety atschool were inaccurate. Data from the studentsthemselves would be potentially more accurateon some subjects; however, there was concernabout the reliability of students, particularlyyounger students, as respondents.

Focus groups with students tested their ability toreport on issues related to school safety anddiscipline. Students in high school were, as

I

anticipated, knowledgeable about the full range ofissues related to school safety and discipline,including alcohol and drugs and weapons at school.The focus groups also revealed that students ingrades 7 and 8 were knowledgeable, and at least asconcerned with their personal safety as were olderstudents. In fact, the discussions with these studentshighlighted the problem of older students bullyingyounger students, a topic that was explicitly includedin the survey because of this finding. None of thestudents who participated in the focus groups haddifficulty understanding the concepts and termslinked to the topics of interest related to schoolsafety. Although this round of cognitive laboratoryresearch did not include 6th grade students, theretrospective reports of the 7th and 8th graders abouttheir experiences as 6th graders and their apparentreliability as respondents suggested that students ingrade 6 would also be appropriate respondents forthis survey component. The other focus groups hadrevealed that this was the age that their parents’awareness of safety issues sharpened. Accordingly,the decision was made to include 6th through 12thgrade students as survey respondents for thecomponent on safety and discipline in school.

The focus groups also confirmed substantialdifferences in the perspectives of “parents andstudents about their school, and in the informationprovided by parents and by students for this surveycomponent. Youth were more informed about allaspects of safety and discipline at school than weretheir parents. Some youth commented that they didnot like to worry their parents and so kept reports ofunsafe incidents from them. As a result of thisfinding, the survey component was designed withidentical wording of most of the questions aboutschool safety so that analysts could compare parents’and children’s responses. In addition, the focusgroup discussions suggested that both students andtheir parents judge the safety of their schools relativeto the safety of their neighborhoods, so items wereincluded measuring neighborhood safety that wouldprovide a useful context for assessments of safety atschool.

10

Testing Survey Instruments

Cognitive laboratory research has also beenemployed extensively in the des ign o finstruments for the NHES. For example, it wasused to test the survey instruments for the EarlyChildhood Education component of theNHES:91, and for both the School Readiness andthe School Safety and Discipline components ofthe NHES :93. The purpose of the research wasto reduce the incidence of measurement error inthe NHES by examining whether the participants1) understood the questionnaire items; 2) had theknowledge necessmy to answer the questions; 3)were able to recall the information when asked;and 4) would be willing to answer all thequestions, especially those that might elicitsensitive information. Individual interviews andfocus groups were included in the cognitivelaboratory research in each test to take advantageof the benefits of the two procedures. Interviewsallowed the researchers to explore selected issuesto some depth with one respondent at a time.Focus groups were intended to capture rangemore than depth. Guided by open-endedresearch questions contained in the protocol, theyoften produced valuable and unanticipatedfindings due to the interaction that took placeamong respondents. Some interviews usedconcur ren t methods, both think-alouds andconcurrent probing. Other interviews and thefocus group discussions used retrospectiveprobes.

Testing the NHES:91 Early ChildhoodQuestionnaire

The Early Childhood component was designed toobtain information from parents about the careand education of their children age 3 through 8.It required parents to report on carearrangements and schooling over the course ofseveral prior years, as well as on currentarrangements. It also measured activities thatparents might do with their children at home that

would promote learning. There was concern aboutthe ability of parents to recall details years later, andtheir willingness to report on their children and theirfamily activities.

Two rounds of cognitive laboratory research wereconducted to test the Early Childhood Educationquestionnaire for the NHES:91. The first roundtested an initial version of the questionnaire, and thesubsequent round tested the revised version. Eachround consisted of six cognitive interviews and twofocus groups. Prior to the focus groups, eachparticipant was administered the interview over thetelephone. Half of the interviews and one of thefocus groups in each round were conducted withparents of preschool children; the other half of theactivities were conducted with parents of children inkindergarten through grade 3. Respondents wererecruited through day care providers in the Rockville,Maryland area, and through the personal networks ofWestat employees in Gaithersburg and Frederick,Maryland. Each participant was paid an honorarium.

One of the primary purposes of the cognitivelaboratory research was to test respondentcomprehension of the terms used in the survey. TheEarly Childhood Education component of theNHES:91 focused on nonparental care arrangementsand participation in preprimary programs as well asprimary school programs. It was important to knowhow parents understood nonparental care, defined inthe interview as a “child care arrangement in whichcare is provided by someone other than you or yourspouse or partner.” Following administration of thesurvey, participants in the first round of cognitivelaboratory research were asked about their definitionsof nonparental child care. Specific probes were usedto determine whether respondents included certainarrangements, for example, care by a grandparentliving in the home or a day camp that lasted for ashort period in the summer. Other probes exploredwhether a parent distinguished between a nurseryschool and a day care with an educational componentwhen asked about preschool.

11

In the initial version of the questionnaire, thesection asking about child care arrangementsbegan with a rather lengthy definition ofnonparental care that named various types ofarrangements. The cognitive laboratory researchrevealed that respondents did not attend to thedetails of the definition, and were not alwaysreporting care arrangements accurately.Following the definition was a general queryabout any kind of nonparental care. Thisresearch indicated that it is more appropriate toask first about specific types of nonparental careor early childhood programs rather than asking ageneral question. The findings suggested that ashorter definition would more appropriately cuethe respondent. Because participants in thecognitive laboratory research did not alwayscount care by a relative if there was no chargefor that care, a specification was added to theintroductory definition indicating that theyshould report any regular, nonparental care givenby relatives or nonrelatives whether or not therewas a charge for the service. The second roundof activities tested the new definition and thefeasibility of collecting information on each typeof arrangement separately. The more specificquestions prompted respondent recall and cuedrespondents as to the types of arrangement inwhich the study was interested. Although thisinvolved asking more questions, the cognitiveburden on the respondents was reduced becausethey did not have to keep multiple criteria inmind while answering.

The cognitive laboratory research also testedwhether terms that were used in the initialquestionnaire, such as preschool, nursery school,prekindergarten, and day care center, might holddifferent meanings for respondents. The findingsindicated that parents did indeed have differentdefinitions, particularly for preschool. In theoriginal version of the questionnaire, somerespondents gave information about their child’sday care center when asked about day care, andthen gave the same information again whenasked about preschool. The subsequent version

of the questionnaire did not use the term preschool.Instead, respondents were asked about day carecenters, including whether the program had aneducational component, and then about nurseryschool, prekindergarten, or Head Start programsgrouped together. When asked about the latter, theywere instructed to answer only for programs forwhich they had not already provided information inorder to reduce the incidence of double counting.

The questionnaire also asked parents to speci~ thetype of nonparental care their child had receivedeach year between birth and age 5. Participants hadlittle difficulty accurately counting the number ofcare arrangements in which their child was currentlyparticipating. 4 They were also able to recall pastarrangements for their child; however, they found itdifficult to specifi the arrangements on a year-by-year basis, particularly if they had had severalarrangements over the years. Participants in thecognitive laboratory research explored ways to makerecall over that period of time easier. Theydiscussed whether it was easier to think in terms ofthe ages of their children, months, seasons, or schoolyears and summers. However, it appeared from thediscussions that parents of the older children in thesurvey would not be able to report the number ofarrangements accurately, even with suggestions thatmight cue them. Consequently, the revisedquestionnaire contained only one question askingparents of primary school children whether the childhad received nonparental care, either home based orcenter based, from birth to age 5.

Focus group participants were also asked to commenton the sensitivity of the items and their willingnessto answer questions about their children’s carearrangements and family activities. They indicateda willingness to answer questions on these topics;however, it was apparent that parents were inclinedto answer some questions in a manner that would puttheir parenting in a favorable light, particularlyquestions about activities with their children. Parentswant to be thought of as competent and caring, andthey admitted in their discussions that they mighthave a tendency to exaggerate time spent with a

12

child and the types of activities in which theyengaged by estimating on the high side. Forexample, although the question referred to thepast week, some parents said that they wouldprobably respond about a “typical” week, if thepast week had not included many activities.This finding emphasized the need to both wordand order questions carefidly to minimizeoverreporting.

Testing the NHES:93 School Readiness andSchool Safety and Discipline Questionnaires

Cognitive laboratory research also played asignificant part in testing questionnaires for theSchool Readiness and School Safety andDiscipline components of the NHES:93. For theSchool Readiness component, nine individualinterviews and three focus groups containingbetween six and nine participants each, wereconducted. Approximately one-third of theinterviews used the think-aloud procedure, one-third used concurrent probes, and one-third usedretrospective probes. Separate focus groups wereheld with parents of children not yet inkindergarten, parents of children in kindergarten,and parents of children in primary grades. Forthe School Safety and Discipline component, 11interviews, 5 with parents and 6 with students,were completed, and 4 focus groups, 2 withparents and 2 with students, were conducted.Parent and student focus groups were structuredby grade of child, either 6th through 8th grade,or 9th through 12th grade. Written consent foreach student’s participation was obtained inadvance from his or her parent. Parentparticipants and student participants also signedconsent forms before the discussion began. Allparticipants were paid an honorarium.

Participants were recruited through flyers postedin businesses, such as grocery stores and officebuildings, and in apartment and housingcomplexes with ethnically diverse residentslocated in the Rockville, Maryland, area andthrough the personal networks of Westatemployees in Frederick and Gaithersburg,Maryland. Recruitment criteria included

diversity on racial/ethnic, gender, and socioeconomicstatus as measured by highest occupation in thefamily. An additional criterion for the SchoolReadiness research was having representation of arange of participation in types of nonparental care, aswell as in parental care. For the School Safety andDiscipline component, limits were placed on thenumber of students or parents of students attendingprivate schools who could take part in the research,and parents who had ever worked for a schoolsystem or held office in a parent-teacher associationwere excluded, as were students or parents ofstudents attending alternative schools because ofbehavior problems.

The results of the cognitive laboratory researchconducted with youth participants to test the SchoolSafety and Discipline questionnaire illustrate fhrtherthe utility of including these methods in the designof a survey. Particular care was taken with thewording of the items for this survey componentbecause both parents and students in grades 6through 12 would be answering many of the samequestions. Questions were crafted to appropriatelycue each population and to be easily comprehensibleto youth as young as 11 years old. Nevertheless, thecognitive laboratory research revealed some lack ofclarity. For example, the test questionnaire containedquestions about knowledge of and victimization bybullying. Youth appeared to have no difficultyunderstanding this question and responded in theinterviews with correct definitions of bullying.However, a more extended discussion that took placein the focus group of younger students revealed thatthey had a correct, but narrow, perception. They didnot think a person was being bullied unless he or shewas under immediate physical threat. They failed tocount cases of extortion of lunch money, forexample, in which the intimidation was pervasive butsubtle. As a result of this finding, the final versionof the questionnaire included a definition of bullyingto cue the respondents to the broader meaning of theterm.

Cognitive laboratory research also providedreassurance that some terms, thought to be toosophisticated perhaps, were likely to be readilyunderstood by respondents. For example, youth were

13

asked if they would think that any fight that tookplace between some white students and someblack students was racially motivated. Theparticipants in this research indicated theyunderstood the term “racially motivated” as itwas intended. The students asserted that theywould know the cause of fights at their schools,and only incidents caused by specific animositybetween racial groups, not just any incident thatinvolved both blacks and whites, would belabeled racially motivated.

The individual interviews were particularlyvaluable in assessing respondent comprehensionand guiding revision of the School Safety andDiscipline instrument. Youth had difficulty withthe think-aloud procedure but were able to recalltheir thought processes in response to bothconcurrent and retrospective probes. Often,inconsistencies in their responses revealed a jackof comprehension, and the ability to exploremeanings at length with one participant at a timewas constructive. In some cases, commentsmade only a single time in the research had animpact on the redesign of the questionnaire, andseveral items were reworded to improverespondent comprehension. F o r example,because one youth interpreted get the followingthings on school property (referring to tobacco,alcohol, and other drugs) as carrying them ontothe school grounds rather than obtaining them onschool property, the wording was changed to getthe following things at school or on the schoolgrounds. Not only were respondents’ specificconfusions and suggestions about improvingclarity helpful, but also the research yieldedsubtle intimations about adolescent, andparticularly preadolescent, cognitive abilities thatinspired changes in item wording. For example,because a youth thought under the injluence ofalcohol meant under the age to drink legally,separate words, not just whole phrases wereexamined for potential confhsion for respondents.Because of the findings from the interviews, forexample, the survey asked about students whowere drunk or showed the eflects of alcohol atschool.

Findings from both the individual interviews and thefocus groups revealed that the youth had difficultywith questions that required them to think about theirschool experiences in the abstract. For example, theyhad difficulty evaluating whether the school waspreparing them well for further study or for a futurejob, or whether most of the students at the schoolwere well behaved. On the other hand, they hadlittle difficulty with questions that asked aboutpersonal experiences, such as whether they enjoyschool or if they are challenged by their classes.This preference for the personal was reflected in thedivergent meanings assigned to the term classmate.Some thought about the class leaders and some aboutthe troublemakers, while others thought of their ownfriends. However, when asked about yourfiiends atschool, the youth participants described a clearerframe of reference. They reported that they knewexactly about whom they were thinking. Thisfinding led to rewording questions, whereverappropriate, to personalize the reference for therespondent. For instance, instead of asking aboutpressure from other students at school to use tobacco,alcohol, or other drugs, the question asks if theyouth’s friends at school think it is all right to smokecigarettes, etc. And rather than asking how easy ordifficult it would be for students to obtain thosesubstances at the school, the question was worded toask how easy or hard it would be for the youthrespondent to obtain the substances if he or shewanted to. As a result of the cognitive laboratoryresearch, the data obtained from youth are likely tobe much more accurate.

Evaluating RespondentComprehension During DataCollection Through Taped Interviews

An additional data quality activity that has beenconducted in the NHES is recording and analyzing asmall number of interviews taped during datacollection. This activity provides another check onquestionnaire wording in terms of respondentcomprehension and the ability of respondents toprovide accurate responses. The research was

14

conducted during t h e NHES:91 a n d t h eNHES :93. Different analysis techniques wereused for each survey. For the NHES:91, contentanalysis of the transcripts of the taped interviewswas conducted. For the NHES :93, a differentmethodology based on coding interviewer-respondent interaction on an item-by-item basiswas used.

Content Analysis of Taped Interviews

During data collection for the NHES:91, a smallnumber of extended interviews were recordedand subjected to a systematic content analysis,although not coded item by item. The analysisof the taped interviews took a linguistic andholistic approach to the interview process. Thecumulative effects of the telephone interviewprocess, and of the wording of and responses toearly interview i tems on respondents’comprehension of later terms were considered.The analysis was conducted by staff who werenot involved in the design of the surveyinstruments and who had not participated in theearlier cognitive laboratory research. Thepurpose of the research was to assess respondentcomprehension during actual te lephoneinterviewing.

Taping was done during the last 2 weeks of datacollection, and only cases that had never beencoded a refusal (i.e., to the Screener or toanother extended interview in the household)were eligible. Respondents were asked forpermission to record their interview after theyhad completed the screening questions, butbefore the extended interview began. If anyreluctance on the part of the respondent wasdetected, the interview was not taped. Thus,participation was voluntary, and this should benoted when interpreting results. Twenty-oneinterviews (11 for the Early Childhood Educationcomponent and 10 for the Adult Educationcomponent) were taped and analyzed. Two

additional interviews were taped but not analyzedbecause the extended interview was not completed.

Examples from the taped Adult Education interviewsdemonstrate the contribution of this research toimproving data quality in future surveys. In theAdult Education survey, the respondents were read adefinition of adult education activities. Some of thetaped interviews revealed that respondents recalledparticipation in adult education activities late in theinterview, including participation in activities thatshould have been iilcluded in earlier responses.Respondents seemed to hold clear definitions of whatconstituted adult education, but those definitionswere personal and varied widely. (Had cognitivelaboratory research been conducted on the AdultEducation component during the design of theNHES:91, this might have been learned earl ier.)This finding indicated that cognitive research duringthe design of fiture questionnaires should exploreways of cuing respondents more accurately to thedefinition used in the survey, as well as exploring thefeasibility of cueing them earlier in the interview.

The issue of sidetracking, in which respondents arestill considering a previous question or their responseto a previous question when the next question isasked, also emerged from analysis of the tapedinterviews as an important data quality issue. Forinstance, respondents were read a list of types ofeducational activities that included full-time, degree-seeking education, part-time, degree-seekingactivities, and all other types of educationalactivities. These types were intended to cue them totheir possible participation but were not intended tobe mutually exclusive or exhaustive. In some of theinterviews, respondents were still pondering one typeof activity as the interviewer read another, resultingin an increased potential for inaccurate responses.This suggested that future design activities shouldtest shorter and more distinct cues for the differenttypes of activities. Also, interviewer training shouldemphasize allowing respondents ample time toabsorb the meaning of an item.

15

Interviewer-respondent interaction during theinterview was also considered. The AdultEducation respondents were asked questionsabout full-time, degree-seeking education, part-time, degree-seeking activities, and all othertypes of educational activities. They were askedabout the number of activities, names of courses,and names of providers. It was often necessaryfor interviewers to cku-ifi for respondentswhether some activities constituted a separatecourse or activity. For example, one respondentwas confised about whether private instructiontaken to help pass a college course was part ofthe course or should be counted as a separateactivity. The need for interviewer clarificationpointed to items that could be improved forfuture administrations. Issues such as providingmore versus providing fewer cues and theplacement of cues in the interview would beappropriate to consider in cognitive laboratoryactivities for fiture surveys.

Similarly, the interview required the repetition ofa series of questions about each course (for up tofour courses) taken during the past 12 months.In some interviews it was quite clear that allcourses were taken for the same reason, and afew of the interviewers mentioned that reasonfirst so the respondent did not have to hear thesame lengthy list repeated again and again. Thesubsequent administration of this component inthe NHES :95 included methods to reduce therepetition in the survey.

Interviewer-Respondent Behavior Coding

The analysis strategy used for the NHES:91taped interviews led to valuable insights thatinfluenced the design of subsequent surveys.However, building on recent research on codingrespondent and interviewer behavior (Oksenberg,Cannell, a n d Kalton 1991), p r o j e c t s t a f femployed another way of coding taped interviewdata for the NHES:93. A structured codingscheme was developed for each item in thesurvey that would point to areas of the SchoolReadiness and School Safety and Disciplinequestionnaires in which deviations from thequestion wording occurred or for whichinterviewer clarification was required. This

method provided a quantifiable indication of howwell the questionnaires allowed the interviewers tofollow prescribed procedures and evoked from therespondents an answer that could be coded using theestablished response categories. However, like thecontent analysis conducted in 1991, it did notprovide reasons for deviations from the questionnairestructure.

Seventy interviews, 25 School Readiness interviews,25 School Safety and Discipline parent interviews,and 20 School Safety and Discipline youthinterviews, were taped during data collection andana lyzed by two spec ia l ly t r a ined coders.Interviewer behavior was rated according to thefollowing five codes: interviewer read the questionexactly as worded, interviewer read the question withonly a minor wording change, interviewer read thequestion with a major wording change, interviewerclarified the question for the respondent, orinterviewer displayed some affect. Respondentbehavior was coded using six categories: respondentgave a “correct” response (i.e, one that fit a precededresponse alternative), respondent interrupted theinterviewer before the question was completed,respondent clarif ied the question, respondentqualified the answer with respect to accuracy,respondent did not provide an adequate answer, orrespondent expressed sensitivity to the question. (SeeAppendix C for an example of a coding form).Fourteen interviews were coded by both of the twocoders. The greatest discrepancy was between “exactwording” a n d “minor wording change. ” Forexample, one coder coded pausing as a minorwording change, and the other did not. When the“exact wording” a n d “minor wording change”categories were collapsed, overall inter-raterreliability was high, ranging from 78 percent to 84percent over the three types of interviews.

The analysis produced the number of times and thepercentage that each code was used for each of thethree interview questionnaires. It also permittedevaluation of each question according to the codesassigned across all of the interviews of that type.This allowed identification of specific questions thathad been subject to major wording changes by theinterviewer, or that had caused the respondent to askfor clarification or to produce a response that wasnot initially codable.

16

Valuable information was obtained from thisanalysis. For example, the introductions at thebeginning of sections of the questionnaires,intended to cue respondents to a change in topic,were coded as having the most minor wordingchanges. This may be due to interviewers whoare responding to the conversational demands ofthe interview process by trying to place the newtopic in the context of what has gone before.The analysis also highlighted the difficulty thatrespondents have with series of questions thathave a Likert response scale, such as stronglyagree, agree, disagree, and strongly disagree.The information gleaned from the analysis oftaped interviews will be useful in the design ofnew NHES surveys, both in a general sense, andwhen the specific topics are included in a futuredata collection.

Summary

The National Household Education Survey hasincorporated cognitive laboratory research andanalysis of taped interviews as integral parts ofits design process. Individual interviews, usingconcurrent and delayed cognitive laboratorymethods, and focus groups guided by lengthyprotocols contributed to the assessment of thefeasibility of topical components of the surveyand to testing the questionnaires to reducemeasurement error. Taping and analysis of liveinterviews measured interviewer and respondentbehavior during data collection and contributedto the understanding of words and items thatmay cause difficulty. The findings will guidecognitive laboratory research for future surveyadministrations. The NHES will continue toexplore new ways to utilize cognitive laboratoryresearch and other data quality activities and toadapt the work of other researchers to enhancecoding schemes used in the analysis of theNHES data quality.

17

References

Bates, N., and DeMaio, T.J. (1990). Report on Cognitive Research on the Public and Private SchoolQuestionnaire for the Schools and Staf$ng Survey (SASS-3A and SASS-3B). Internal CensusBureau Memorandum. July 11, 1990.

Brick, J. M., and West, J. (1992). Reinterview Program for the 1991 National Household EducationSurvey. Proceedings of the Section on Survey Research Methodology, 387-392. Alexandria, VA:American Statistical Association.

Brick, J. M., Collins, M., Nolin, M. J., and Davies, E. (Forthcoming). Selected Data Quali~ Activities inthe NHES:93. Washington, DC: U.S. Department of Education, National Center for EducationStatistics.

Brick, J. M., Rizzo, L., and Wernimont, J. (Forthcoming). Reinterview Results for the School Readinessand School Safety and Discipline Components of the 1993 National Household Education Survey(NCES 96-339). Washington, DC: U.S. Department of Education, National Center for EducationStatistics.

Campanelli, P., Rothgeb, J., Esposito, J., and Polivka, A. (1991). Methodologies for Evaluating SurveyQuestions: An Illustration from a CPS CATI/RDD Test. Paper presented at the 1991 AnnualMeeting of the American Association for Public Opinion Research, Phoenix.

Cantor, D., and Edwards, C. (1992). Testing an Alternative Household Rostering Procedure for the Surveyof Income and Program Participation (SIPP Working Paper 92-03). Washington, DC: U.S. Bureauof Census.

Cantor, D., Kerwin, J., Sheridan, S., Powell, T. and Banthen, J. (1995). Validat ing Information wi thCognitive Interviews to Measure Respondent Knowledge: Developing Questions to Report HealthCare Plans. Paper presented at the International Conference on Survey Measurement and ProcessQuality, April 1995, Bristol, England.

Dippo, C. S., and Norwood, J.L. (1992). A Review of Research at the Bureau of Labor Statistics. In J.Tanur (cd.), Questions About Questions: Inquiries into the Cognitive Bases of Surveys, 271-290.New York: Russell Sage Foundation.

Ericsson, K.A. and Simon, H.A. (1980). Verbal Reports as Data, Psychological Review, 87:215-51.

Ericsson, K. A., and Irwin, H.A. (1984). Protocol Analysis: Verbal Reports As Data. Cambridge, MA:MIT Press.

Forsythe, B. H., and Lessler, J.T. (1991). Cognitive Laboratory Methods: A Taxonomy. In P.P. Biemer,R.M. Groves, L.E. Lyberg, N.A. Mathiowitz, and S. Sudman (eds.), Measurement Errors inSurveys, 395-418. New York: John Wiley and Sons.

Groves, R.M. (1989). Survey Errors and Survey Costs. New York: John Wiley and Sons.

Jenkins, C. (1992). Questionnaire Research in the Schools and Staffing Survey: A Cognitive Approach.Proceedings of the Section on Survey Methods Research, 434-439. Alexandria. VA: AmericanStatistical Association.

Jenkins, C., Ciochetto, S., and Davis W. (1992). Results of Cognitive Research on the Public SchoolQuestionnaire for the Schools and Staf$ng Survey (SASS-3A). Internal Census BureauMemorandum.

Jobe, J. B., and Mingay, D.J. (1991). Cognition and Survey Measurement: History and Overview. AppliedCognitive Psychology, 5:175-92.

Krueger, R.A. (1988). Focus Groups: A Practical Guide for Applied Research. Newbury Park, CA: Sage.

Lessler, J., Bercini, D., Tourangeau, R., and Salter, W. (1986). Cognitive Laboratory Studies of the 1986Supplement to the National Health Interview Survey Final Results. Proceedings of the Section onSurvey Methods Research, 478-80. Alexandria, VA: American Statistical Association.

Lessler, J., Tourangeau, R., and Salter, W. (1989). Questionnaire Design in the Cognitive LaboratoryResearch. Vital and Health Statistics: Series 6, Cognition and Survey Measurement, No, 1.Washington, DC: U.S. Department of Health and Human Services (DHHS Publication No. (PHS)89-1076).

Merton, R. A., Fiske, M., and Kendall, P. (1956). The Focused Interview: A Manual of Problems andProcedures. Glencoe, IL: Free Press.

National Education Goals Panel (199 1, 1992). The National Education Goals Report: Building a Nationof Learners. Washington, DC: U.S. Government Printing Office.

Oksenberg, L., Cannell, C., and Kalton, G. (1991). New Strategies for Pretesting Survey Questions.Journal of Oflcial Statistics, 7, 3:349-65.

Presser, S., and Blair, J. (1994). Survey Pretesting: Do Different Methods Produce Different Results? InP.V. Marsden (cd.), Sociological Methodology, 73-104. Washington, DC: The AmericanSociological Association.

Schwartz, N. (1995). What Respondents Learn from Questionnaires: The Survey Interview and the Logicof Conversation. International Statistical Review, 63: 153-177.

Appendix A:

Moderator’s Topic Guide for Parent Groups

20

Written consent was obtained from each participant.

NHES: 93 SCHOOL SAFETY AND DISCIPLINE

MODERATOR’S TOPIC GUIDE FOR PARENT GROUPS

I. Warm-up and Exdanation (1 O minutes)*

A. Introduction

1. Thanks for agreeing to come.

2. Your participation is very important.

3. A focus group is like an opinion survey, only with very broad, general questions

B. Purpose of Focus Group

1. This evening we will be talking about safety and discipline in the junior high (highschool). Information from this group and other groups with parents and students willbe used to help develop a questionnaire on this subject for a survey of parents andstudents across the country.

2. Everyone has different experiences; all of your ideas, comments and suggestions willbe helpful.

3. There are no right or wrong answers. It is all right to disagree with other comments.We hope to get many points of view.

4. All comments - both positive and negative - are welcome.

c. How the Group Works

1. The session is being tape recorded, and observers are present behind the one-waymirror. However, all comments are confidential - used for research purposes only.

2. This is a group discussion, so you need not wait for me to call on you. Please speakone at a time so the tape recorder can pick up everything.

3. We have a lot of ground to cover, so I may change the subject to move ahead. Pleasestop me if you want to add something.

D. Introduction of Participants

1, Let’s take a moment to go around the table and introduce ourselves. Tell us your firstname and what you like to do in your spare time. Also tell us how many children youhave and the age and grade of your child in junior high (high school).

*Participants were asked to sign a consent form in the waiting room.

21

Written consent was obtained from each participant.

11. School Safety (30 minutes)

* A. I’d like to start out talking with you about occurrences that may make your child feel afi-aidor unsafe at school, OR may cause you to worry about him or her. These occurrences maytake place either during the school day or while your child is going to or from school (I’mNOT talking about environmental hazards such as asbestos, dangerous playgroundequipment, etc.) . What actions, behaviors, events come to mind? (LIST ON EASEL)(Probe: physical/verbal abuse, theft, vandalism, substance use, locker crime)

* B. How many of you feel these kinds of occurrences at school have been a personal issue foryour child at any time in junior high (high school) (TAKE COUNT)? In general, whataspects of school safety concern or worry you the most?

* C. Let’s consider something else (something you touched on earlier) -- the issue of disruptivebehavior, whether in the classroom, hallway, playground, school bus, etc. What kind ofdisturbances or infractions contribute to the atmosphere of a school being unsafe or notconducive to learning? (ADD ITEMS TO LIST ON EASEL)

D. Tell me how important you consider these major to minor disturbances that we’ve mentionedto be? Why do you say that?

* 1. What impact does it (can it) have on a child’s behavior, sociability, learning, feelingstoward school, attendance, emotional state (e.g., fear)?(Probe: Does it interfere with learning? What percentage of class time do you feelteachers spend dealing with disruptive behavior?)

2. (Opt. ) Is there another issue you would compare it to, or put it on the same par with?How do you see it in relation to other issues parents of junior high/high schoolstudents have to deal with?

E. How would you rate your child’s school in terms of it being a safe, pleasant and aconducive place to learn? Where would you put it on a scale of 1 - 5 where 1 is “not atall safe/pleasant/conducive to learning” and 5 is “very safe/pleasant/conducive tolearning.” (GO AROUND TABLE TO GET RATINGS)

1. Those of you who gave higher ratings (say, 4- 5), how do you know that yourchild’s school is safe and provides a proper atmosphere for learning?

2. (Opt.) Probe: Where/how do you get your information -- e.g., Have there beenmeetings with parents to explain school security and safety? Has the PTA lookedinto the situation, made recommendations and did the administration listen/act?Have the local police been visible at the school -- talking at assemblies, etc?)

F. (Opt.) What do you actually look for to determine if a school is safe? In other words,how do you evaluate a school -- what criteria do you use? (Probe: Is it realistic to expecta school to be 100 percent safe?)

*High priority question.

22

Written consent was obtained from each participant.

* G. What would you do (or have you done) if you thought your child’s school was unsafe?(Probe: switch to another school -- public or private; restrict your child’s behavior; bandtogether with other parents; speak to school officials)

H. How do you know when a school is unsafe? (Why did some of you assign ratings of1-2 to your child’s school?) Can you give me some examples of what makes a schoolunsafe?

1. What role does rumor play? How would you react if you heard that someone atyour child’s school had been attacked? (Probe: Does that make the school unsafe?)

2. (Opt.) Some people are afraid that safety in the schools may be getting worse.What does “getting worse” mean to you?

III. Alcohol and Drug Use (25 minutes)

* A. Teenage alcohol and drug use has already come up in our discussion. At this point, I’dlike to talk more about these issues. Before we do, however, I want to pass out a pieceof paper with some statements on it, and I’d like you to indicate on a scale of1 -5 how much you agree (or disagree) with each one. Please be as honest as you can;do not put your names on these sheets.(DISTRIBUTE HANDOUT CONTAINING THE FOLLOWING STATEMENTS AND A5-POINT AGREEIDISAGREE SCALE)

1.

2.

3.

4.

5.

6.

7.

8.

9.

10,

Young people should not be allowed to drink alcohol until they reach the age of18.

Drinking is taboo by anyone in my (our) household.

I think some parents get overly upset by their child using marijuana (grass) on anoccasional basis.

I do not allow my child to attend parties or other social events unless I check to besure an adult chaperon will be present.

I do not allow my child to have parties at my house with alcohol (including beer,wine, wine coolers).

Experimenting with alcohol or marijuana is just a part of growing up today.

Smoking is a relatively harmless form of teenage rebellion.

If my child had to experiment with something, I would rather it were alcohol thanmarijuana.

I feel I am less tolerant of teenage drinking than other parents of teenagers I know.

I could understand if my child tried cocaine or heroin only once.

23

Written consent was obtained from each participant.

(HAVE PARTICIPANTS PASS PAPERS FORWARD)

* B. How common do you think it is for junior high (high school) students to attend functionsoutside of school where alcohol is available? How about marijuana? Hard drugs (cocaine,heroin)? Amphetamines?

* c. Let me get your reaction to something else. A parent in one of my groups said recentlythat he and his wife occasionally ~ their teenagers a drink so that having a drink willnot be a big deal to them -- in other words, it won’t be an allurement or “forbiddenfruit.” How do you feel about this?

* D. Some statistics have shown that the majority of teenagers try alcohol in high school. Atwhat age would you ~be alarmed if you heard your son or daughter were at a partyand had a few beers or wine coolers? Conversely, at what age (or under what conditions)would you be alarmed to hear this?

* E. Do you think you would actually know if your child were using cigarettes, alcohol,marijuana, or other drugs? How would you know? (Probe: Have you ever talked withyour child about his/her use of cigarettes, alcohol, or drugs, or do you just know?)

* F. Do you have any rules in your household about drinking? (What are they?) How aboutmarijuana use? Other drugs? Smoking? Are there any consequences or punishments forusing these things? (What are they?)

IV. Discipline Policy (Rules and Remdations) (35 minutes)

A. Now I’d like to change the subject somewhat and talk to you about discipline policy.Before asking you some general questions; however, I’d be interested in hearing whatwould happen at your child’s school if the following situations arose . . .

1. A student at your child’s school was caught drinking alcohol during the schoolday? (Probe: Would this be a fair action if your child had just a few sips of beerthat another student had brought to a school football game?)

2. One student physically attacked another student? (Probe: Would this be a fair actionif your child had gone through several weeks of verbal bullying and either your songot into a fistfight, or your daughter slapped the person bullying her?)

3. (Opt.) A student shouted obscenities at a teacher who was reprimanding him/her?(Probe: Would this be a fair action if your child muttered the same obscenitieswhile walking away after a reprimand?)

B. Do you feel that discipline (or the lack thereof) is a problem in the junior high (high)schools? When I use the term “school discipline, ” I’m talking about disruptive behaviorin the classroom as well as some of the other items we discussed earlier. (We’re not

24

talking aboutthat?

C. What do youschools?

1. Should

Written consent was obtained from each participant.

discipline for dress code infractions or tardiness to school.) Why do you say

think school officials should be doing to ensure safe and well-disciplined

suspensions and expulsions be used as forms of punishment?

2. The mayor of Washington has suggested introducing corporal punishment back intothe schools -- what do you think of that?

3. How do you feel about random locker searches, metal detectors, drug-sniffing dogs,random urine testing, hall patrols?

* D. Do you know whether or not your child’s school has a discipline policy?

1. Do you know if it is written down? What form does it take (e.g., handbook)?

2. Have you ever seenhead it? Do you have a copy?

3. How was it developed? Who put it together? (Probe: Was there any input fromparents? Students?)

4. What does it cover (e.g., disruptive behavior, substance use)?

E. How do you feel about the discipline policy at your child’s school? Is there anything youparticularly like about it? Particularly dislike about it?

1. Do you consider it too strict? Too lenient? In what way(s)? How do you judgethis (i.e., what standard are you using)?

2. Do you feel that the school acts (i.e., sets policy) only after an incident occurs or isit more proactive in nature, setting policy in advance of something happening?

F. Having a policy is one thing; how it functions can be something else. What do you thinkabout how the discipline policy at your child’s school actually works?

1. Are regulations consistently and fairly applied? What standard are you using toevaluate this?

2. Are there separate rules (or are the rules interpreted differently) for first-timeoffenders versus repeat offenders?

3. Is the punishment the same for possessing drugs, using drugs, and distributingdrugs (i.e., encouraging others to use)? How do you feel about this?

4. Do you know if there is a way for the discipline policy at the school to get changedor amended? (Probe: Can you think of any circumstances that might cause you orother parents to appeal for a change in how the policy is applied?)

25

Written consent was obtained from each participant.

* G. We’ve talked so far about safety and discipline at the junior high (high school) level. I’dlike you to think back for a minute to the time when your child was in elementary/gradeschool. Would you say that safety and discipline were of greater concern, lesser concern,or of the same concern to you then as they are now? (TAKE COUNT)

1. What safety and discipline concerns did you have for your child in elementary/gradeschool? (What things made your child feel unsafe or afraid, or caused you to worry?)

2. At what age/grade level did you become concerned about the possibility of disruptivebehavior and/or substance use in the school and its possible impact on your child?

v. Alcohol Drug Education in the School (20 minutes)

A. In the time remaining, I’d like to get back to the subject of alcohol and drugs and talk withyou about alcohol and drug education in the junior high (high school). Is it important foryou to have alcohol/drug education as part of the school curriculum?

1. (Opt.) Probe: What aspects of your child’s education do you consider to be moreimportant? Less important? On a par with alcohol/drug education?

* B. Does your child’s school currently have an alcohol/drug education program? (TAKECOUNT)

1. What do you know about it?

2. Are drug and alcohol education handled together (as part of same course) orseparately?

3. Do you know what overall message your child is getting? (Probe: Is it a “no use”message or a “responsible use” message?) Does this agree or conflict with your ownattitude toward education on this subject?

C. How do you feel about the adequacy of the alcohol/drug education your child is given inschool? How are you making this judgment?

D. (Opt.) Are there any drug education programs for parents at your child’s school (e.g., tohelp them form their own peer or support group)?

VI. Wrap-up

A. We’ve come to the end of the session. Is there anything you’d like to add on any of ourtopics tonight?

B. Thank you for coming. Your opinions have been very valuable to us. Please seereceptionist on your way out.

26

Appendix B:

Moderator’s Topic Guide for Adolescent Groups

27

Advance written consent was obtained from each student’s parent.Written consent was also obtained from each student,

NHES: 93 SCHOOL SAFETY AND DISCIPLINE

MODERATOR’S TOPIC GUIDE FOR ADOLESCENT GROUPS

I. Warm-up and Explanation (15 minutes)

A. Introduction

1. Thanks for agreeing to come.

2. Your participation is very important.

3. A focus group is like an opinion survey, only it’s done in a group.

B. Purpose of Focus Group

1. This aflernoon we will be talking about safety and discipline in the junior high (highschool). Information from this group and other groups with students like yourselveswill be used to help develop a questionnaire on this subject for a survey of parents andstudents across the country.

2. Everyone has different experiences; all of your ideas, comments, and suggestions willbe helpfi.d.

3. There are no right or wrong answers. It is all right to disagree with other comments.We hope to get many points of view.

4. All comments - both positive and negative - are welcome.

5. I would like to hear from everybody.

C. How the Group Works

1. The session is being tape recorded and observers are sitting behind the one-way mirror.However, all comments are confidential - used for research purposes only.

2. This is a group discussion. You need not wait for me to call on you, but please speakone at a time so the tape will be clear. Also, be sure to give others in the room achance to speak.

3. We have a lot of things to talk about, so I may change the subject in order to moveahead.

28

Advance written consent was obtained from each student’s parent.Written consent was also obtained from each student.

4. Before we start, I have consent forms for you to sign. I’ll read it out loud and thenyou can sign it and pass it to me. NOTE: Prior written consent was obtained fromeach student’s parent.]

D. Introduction of Participants

1. I’d like to take a few minutes for us to get to know each other. Instead of having youintroduce yourselves (the way it’s usually done), I’m going to ask you to introduce theperson sitting next to you. Take a couple of minutes right now to talk to the personon your right. Find out his/her first name, age, grade in school, name of schoolattending, number of brothers and sisters and what hobbies or after-school activitiesthe person is into (i.e., what he/she likes to do in his/her spare time.) (LIST ABOVEITEMS ON EASEL FOR REFERENCE)

11. School Safety (30 minutes)

* A.

* B.

* c.

* D,

I’d like to start out talking about things that make you (or your friends) feel afraid or unsafeat school. These are things that may take place either while you are in school or while youare going to or from school. What comes to mind first? What different things can you thinkof? (LIST ON EASEL) (Probe: physical/verbal abuse, theft, vandalism, substance use, lockercrime)

Which things on this list concern you and your friends the most? Write the word “Most” onyour pad and next to it the item on the easel that concerns or worries you most at yourschool. Which one worries you the next most after that? Write “Next Most” on the next lineand then the item from your experience that fits best.

(IF NOT ALREADY MENTIONED) Let me bring up something else here -- disruptivebehavior. Does this go on at your school? What do you think of when I say “disruptivebehavior?” (What kinds of disruptive behaviors make the atmosphere at school seem unsafeor make it difficult for students to learn?) (ADD TO LIST)

1. Do you think teachers spend too much or too little time dealing with disruptivebehavior?

Are these various things we’ve put up on the easel important to you? Why do you saythat?

1. Do you think things are different in a safe school than in an unsafe school? (Probe:How easy or hard is it to learn in a school that doesn’t have a safe atmosphere?)

*High priority question.

29

Advance written consent was obtained from each student’s parent.Written consent was also obtained from each student.

2. How else are students affected (e.g., in terms of behavior, sociability, feelings towardschool, attendance, fear)?

E. (Opt.) How do you know if a school is safe (any school, not necessarily your school)? Writedown three things that would tell you a school is safe.

* F. Think of someone you know who goes to an unsafe school and write down three thingsthat would tell you a school is unsafe.

1. How would you feel if you heard a rumor that someone had been attacked at school?(Probe: Does that make you think the school is unsafe?)

* G. How safe do you think ~ school is? Where would you put your school on a scale of 1-5 where 1 is “very unsafe/a difficult atmosphere to learn in” and 5 is “very safe/an easy

atmosphere to learn in?” (HAVE SCALE ON EASEL) Write down the number on the scalethat tells how safe you think your schooi is.

1. Can you give me an example of something that happened at your school that made itseem unsafe; either something that happened to you, or to one of your friends. Howdid this make you feel?

2. How do you know or find out about other students who are victims of crimes?

* H. What do students do if they think their school is unsafe? Do they change their behavior orhabits in any way to protect themselves (e.g., avoid certain corridors or bathrooms atschool)?

111. Alcohol and Drug Use (20 minutes)

A. Use of alcohol and drugs has come up a number of times in our discussion. I’d like to talkwith you more about this subject, but before we get into that, I want to pass out a piece ofpaper with some statements on it, and I’d like you to indicate on a scale of 1-5 how muchyou agree (or disagree) with each one. Please be as honest as you can; there is no need toput your names on these sheets.(DISTRIBUTE HANDOUT CONTAINING THE FOLLOWING STATEMENTS AND A5-POINT AGREE/DISAGREE SCALE. BE SURE EVERYONE UNDERSTANDSSCALE). Alcohol means beer, wine, wine coolers and hard liquor.

1. Young people should not be allc.wed to drink alcohol until they reach the age of 18.

2. Drinking by teens is not allowed in my house.

3. I think some parents get overly upset by their child using marijuana (grass) on anoccasional basis.

30

Advance written consent was obtained from each student’s parent.Written consent was also obtained ffom each student.

4. I cannot go to parties or other social events unless my parents know an adult chaperonwill be there,

5. I cannot have parties at my house with any alcohol.

6. Trying alcohol or marijuana is just a part of growing up today.

7. Smoking cigarettes is a pretty harmless way for teenagers to show their independence.

8. Lots of teenagers try drugs like coke, heroin, or amphetamines once or twice.

9. It is easy for kids I know at my school to obtain alcohol.

10. It is easy for kids I know at my school to obtain marijuana.

11. It is easy for kids I know at my school to obtain other drugs such as coke, heroin oramphetamines.

(HAVE PARTICIPANTS PASS PAPERS FORWARD)

* B. Do you remember what grade you were in when your classmates were first thinkingabout smoking cigarettes? How about drinking alcohol? Using marijuana? Other drugs?

c. Do you think your parents would know if you were using cigarettes, alcohol, marijuana,other drugs? How would they know?

D. Are there any rules in your household about drinking? (What are they?) How aboutcigarettes? Marijuana? Other d rugs (coke, heroin, amphetamines)? Are the re anyconsequences or punishments for using these things?(What are they?)

1. Have your parents ever talked to you about using cigarettes, alcohol, marijuana, otherdrugs? How do you feel about their views on these subjects?

2. At what age do you think it is all right to have alcohol at a party? (At what age, orunder what conditions, do you think this would be wrong?)

Iv. Discipline Policy (Rules and Regulations) (40 minutes)

A. Now I’d like to change the subject somewhat and talk to you about discipline policy atschool. What does the word “discipline policy” mean to you? What else do you call it?(AFTER PARTICIPANTS GIVE THEIR INTERPRETATION) In this part of the discussion,when I use the term “discipline policy, ” I will be referring to the rules and regulations atschool that cover disruptive behavior in class as weil as some of the other behaviors welisted on the easel. (We’re not talking about discipline for violating the dress code or comingto school late.)

31

Advance written consent was obtained from each student’s parent.

Written consent was also obtained from each student.

B. Before I ask you some general questions on school discipline, I’d be curious to know whatwould happen at your school if the following situations came up...

1. A student was caught drinking alcohol during the school day? (Probe: Would this bea fair action if your best friend had just a few sips of beer that another student hadbrought to a school football game?)

2. One student physically attacked another student? (Probe: Would this be a fair actionif, for example, someone at school was spreading false rumors about you and theserumors were hurting your relationships with your friends, so during a confrontationwith the person, you hit him/her?

3. (Opt.) A student shouted a couple of four-letter words at a teacher who was chewinghim/her out? (Probe: Would this be a fair action if you said the same words whilewalking away from a teacher after he/she had yelled at you mostly because he/she wasbeing unfair or was in a bad mood, not because you had done something really wrong?

c. What do you think the school administration should be doing about safety and discipline inthe junior high (high school)?

1. Do you think that suspensions and expulsions should be used to discipline students?

j*. Do you know what “corporal punishment” is? You may have heard that the mayorof Washington, DC has suggested that corporal punishment be put back in theschools -- what do you think of that?

3. How do you feel about random locker searches, metal detectors, drug-sniffing dogs,random urine testing, hall patrols?

D. Do you know if your school has a discipline policy -- or a set of rules or regulationsabout discipline?

1. Is it written down anywhere? Where (e.g., handbook)?

2. Have you ever seen or read it? Do you have a copy?

3. Who put it together? (Probe: Did students have any say in it? How about parents?)

4. What things does it cover (e.g., disruptive behavior, substance use)?

E. How do you feel about the discipline policy at your school? Is there anything you likeabout it? Dislike about it?

1. Do you consider it too strict? Too lenient? In what way(s)? How do you judge it(i.e., is there a standard of some kind you’re using)?

32

Advance written consent was obtained from each student’s parent.Written consent was also obtained from each student.

* F. What do you think about how the discipline policy at your school actually works?

1. Are rules and regulations applied fairly to different students? What standard are youusing to say this (how do you know)?

2. Are there different rules for kids who do something wrong the first time and kidswho have done it several times?

3. Is the punishment the same for using drugs, having drugs in your possession, or fordistributing drugs (i.e., encouraging others to use)? How do you feel about that?

4. Does the discipline policy at your school ever get changed? What makes this happen?Do students ever appeal for a change in the policy? (Under what circumstances?)

5. Do you feel that the school makes rules only after an incident occurs, or does it setpolicy before something happens?

v. Alcohol/DruK Education in the School (15 minutes)

A. In the time we have left, I’d like to get back to the subject of alcohol and drugs, and talkwith you a little bit about alcohol and drug education in the junior high (high school). Doesyour school have an alcohol/drug education program? (TAKE COUNT)

1. Tell me about the program. What is it like? When do you have it?

2. Are there separate units on alcohol and drugs, or are they all together in the sameprogram?

* B. How do you feel about the alcohol/drug education program at your school? Is it good/notgood? How do you judge it?

1. Do you remember anything from it?

2. Do you think it has affected you? How about your friends?

3. What overall message are you getting from the program? Would you say it’s a “Don’tUse [at all]” message or a “Use Responsibly” message?

c. Do you think it’s important to have alcohol/drug education as part of the school curriculum?Tell me why you say that.

VI. Wrap-up

A. We’ve come to the end of the group. (IF TIME) Is there anything you’d like to add onanything we’ve talked about?

33

Advance written consent was obtained from each student’s parent.Written consent was also obtained from each student,

B. Thanks for coming. Your thoughts have been very helpful to us. Please see thereceptionist on your way out.

34

Appendix C:

Form for Behavior Coding of Interviewer-Respondent Interactions fromTaped Interviews

35