an overview of survey techniques, sampling strategies, modes of observation, interviewing, focus...
TRANSCRIPT
Research MethodsAn Overview of Survey
Techniques, Sampling Strategies, Modes of Observation,
Interviewing , Focus Groups
1
Lecture outline: Overview of material covered in chapters 6, 8-12 in text
Modules created for each chapter in text
Ch6: Measuring Crime
Ch. 8:Sampling
Ch. 9: Survey Research
Ch. 10: Qualitative Interviewing
Ch. 11 Field Observation
Ch 12: Agency Records, Content Analysis, and secondary Data
2
Module 1: Surveys( see Chapter6, 9 in text)1. Survey data collection methods:A look at
UCR, Self Report , and Victimization Surveys
2. The Basics of Survey Design
3. Types of Surveys
3
Topics appropriate to survey researchCounting crime – asking people about
victimization counters problems of data collected by police
Self-reports – dominant method for studying the etiology of crimeFrequency/type of crimes committed
Prevalence (how many people commit crimes) committed by a broader population
4
Topics appropriate to survey researchPerceptions and attitudes – to learn how people
feel about crime and CJ policy( e.g. Death Penalty)
Policy proposals – search for ways to respond to crime that are supported by the general public( e.g Downsizing Prisons)
Targeted victim surveys – used to evaluate policy innovations & program success( e.g. domestic violence)
Other evaluation uses – e.g., measuring community attitudes, citizen responses, etc.( e.g. criminal thinking scales)
5
The Basic Structure of a Survey InstrumentIntroduction: state the general purpose of survey and
emphasize voluntary nature of survey,confidentiality and protection of human subjects.
Section1: Factual non-threatening questions( demographics here)
Section 2: Core questions( ordering questions; contingency questions)
Section 3: Scales( Likert, Gutmann)Example of Matrix questions--Likert Scale: TCU
Criminal Thinking Scale used by Taxman, et. al. in recent study of criminal thinking( class handout, Nov. 26th)
6
Types of questions: Open- and close-ended questionsOpen-ended – respondent is asked to provide his
or her own answerWhat were your initial expectations of the program?
________________________________________________
Closed-ended – respondent selects an answer from a list or scaleOn a scale of 1 to 5, where 1 is low and 5 is high, rate
your initial expectations of the program.
7
1 2 3 4 5
Low High
Open versus closed?Choice between these question types will
depend on the aim of your research, the study design, the population being study, etc.
Advantages and disadvantages to both.
Possible gains in reliability (closed-ended) balanced with possible losses in breadth of information and validity.
One strategy is to combine open- and close-ended questions.
8
Designing questions… Make items clear – avoid ambiguous questions;
do not ask “double-barreled” questions
Short items are best – respondents like to read and answer a question quickly
Avoid negative items – leads to misinterpretation
Avoid biased items and terms – do not ask questions that encourage a certain answer
Closed question response categories must be exhaustive and mutually exclusive.
9
Survey data collection methods1. Self-administered mail surveys,
computer based surveys, and response rates
2. In-person interview: best for complex questions ( role of interviewer is key)
3. Telephone interviews; fast, low cost; CATI- computer assisted telephone interviewing
4.A comparison of three methodsFocus: Racial Profiling Research( see text
pp 256, 257)
10
Focus: Australia’s National Crime Victimization Survey Methodology
The MPHS was conducted as a supplement to the monthly LFS. Each month one eighth of the dwellings in the LFS sample were rotated out of the
survey. In 2012-13, all of these dwellings were selected to respond to the MPHS each month . In these dwellings, after the LFS had been fully completed for each person in scope
and coverage, a person aged 15 years and over was selected at random (based on a computer algorithm) and asked the various MPHS topic questions in a personal interview.
If the randomly selected person was aged 15–17 years, permission was sought from a parent or guardian before conducting the interview. If permission was not given, the parent or guardian was asked the crime questions on behalf of the 15–17 year old.
Questions relating to sexual assault, alcohol or substances contributing to the most recent physical or face-to-face threatened assault were not asked of proxy respondents. Only those persons aged 18 years and over were asked questions on sexual assault.
Data was collected using Computer Assisted Interviewing, whereby responses were recorded directly onto an electronic questionnaire in a notebook computer, usually during a telephone interview.
11
A Link to the USA National Crime Victimization Survey
http://www.bjs.gov/index.cfm?ty=dcdetail&iid=245
12
In Review: Comparing UCR and NCVS—See Text Chapter UCR data are based on reported criminal acts
(offender characteristics)NCVS data based on individuals actually
victimized (characteristics of victims)Whenever possible, review both data sources
and compare findings
Assessment of NCVSDocument a massive amount of crime that goes
unreportedUnderestimate crime rateInsignificant crimes tend to be forgottenVictims of several crimes may also forget about
all the crimesFemales do not report victimization if her
abuser live in the same household Whites and college graduates are more likely to
report being victimized
Assessment of NCVSNCVS respondents are interviewed every six
months (7 interviews)Reported victimization rates usually decease
with each interview (awareness of victimization)
Overestimation of some crimesRespondents might mistakenly interpret some
noncriminal events as crimes“Telescoping “ effect
Self-reports data Created to complement UCR and NCVSInterviews or questionnairesDemonstrate the prevalence of offending (the
proportion of respondents who have committed a particular offence)
Incidence of offending (the average number of offences per person in the study)
Samples for self-reportsTarget: Adult inmates of jails and
prisons: PREA findings on sexual assaultTarget: Adolescents, usually high
school studentsThe most important finding: delinquency
is very common Middle-class youth commit as much crime
as working-class youthTesting criminological theories: Travis Hirschi’s
Research Study testing Control Theory( compared to subculural & Strain)
Assessment of self-report studiesFocus on minor and trivial offenses
(truancy, running away from home, minor drug and alcohol use)
Although recent studies (NYS) asked subjects about rape and robbery
Respondents might not to tell the truth (reliability issues)
If respondents lie…. Self-report data can be checked against police
records, school records, interviews with teachers and parents
The use of, or threat of , polygraph validation (20% change their initial responses when threatened with a “lie detector”)
Subsequent interviewing of subjects permits probing regarding the details and context of acts
Use of “lie scales”
Focus: Self Reported Substance Use Among Detainees in Australiahttp://www.aic.gov.au/media_library/conferen
ces/evaluation/mcgregor.pdf
http://192.190.66.70/documents/7/E/8/%7B7E8D4A8E-A5AF-4D3B-8821-ED8A1BA489B6%7Drpp93.pdf
21
Focus: Research on Extent of Substance Use by High school students in USAQuestion: Can we collect reliable estimates
from survey questionnaires distributed to high school students across the country?
The Survey: Lets take a close look:http://
www.casacolumbia.org/addiction-research/reports/national-survey-american-attitudes-substance-abuse-teens-2012
http://www.drugabuse.gov/publications/drugfacts/high-school-youth-trends
22
UCR, NCVS, and self-reportsNone of the three is perfectFor the best estimates of the actual number of
crimes, NCVS data are preferable For the best estimates of offender
characteristics, self-reports and NCVS are preferable
UCR are superior for understanding the geographical distribution of crime
Self-administered questionnairesCan be home-delivered
Researcher delivers questionnaire to home of sample respondent, explains the study, and then comes back later
Mailed (sent and returned) survey is most commonResearchers must reduce the trouble it takes
to return a questionnaire
Goal is High Response Rate
24
Computer-based self-administration
Via email, website Issues:» representativeness»mixed in with, or mistaken for,
spam» requires access to Web»sampling frame?
25
Module 2:More Survey Research What we will Cover:
1.Interviewing techniques and Focus Groups2. A Comparison of Survey Methods: phone/ computer( survey monkey),
mail, and in-person3. Field Research: The good, the bad, and the ugly
4. Agency Records, Content Analysis and Secondary Data
26
In-person interview survey: See text, ch 9
Typically achieve higher response rates than mail surveys (80-85% is considered good)
Demeanor and appearance of interviewer should be appropriate; interviewer should be familiar with questionnaire and ask questions precisely
When more than one interviewer administers, efforts must be coordinated and controlled
Practice interviewing
27
Specialized interviewing• Two variations:»General interview guide – less
structured, lists issues, topics, questions you wish to cover; no standardized order
» Standardized open-ended interview – more structured, specific questions in specific order; useful in case studies, retrieves rich detail in responses
28
Telephone surveys 94% of all households now have
telephones Random-Digit Dialing» Obviates unlisted number problem» Often results in business, pay phones, fax
lines Saves money and time, provides safety
to interviewers, more convenient may be interpreted as bogus sales calls;
ease of hang-up
29
Comparison of the three methods Self-administered questionnaires are
generally cheaper, better for sensitive issues than interview surveys
Using mail – local and national surveys are same cost
In person or phone Interviews – more appropriate when respondent literacy may be a problem, produce fewer incompletes, achieve higher completion rates
Validity low in survey research; reliability high Surveys are also inflexible, superficial in
coverage
30
Tips on self-report items Convince subjects you will
guarantee confidentiality and anonymity
Minimize possible social undesirability you are asking respondents to admit
Phrase questions in non-judgmental manner
Bear in mind “fading memory” when setting time frame
31
Focus groups: See Putting it all together, pp256,257 in text
12-15 people brought together to engage in guided group discussion of some topic( e.g. addicts& recovery)
Members are selected to represent a target population, but cannot make statistical estimates about population
Most useful when precise generalization to larger group is not necessary
May be used to guide interpretation of questionnaires following survey administration
Examples: Drug addicts and reentry, homeless sex offenders, teenagers and sexting
32
Getting OutModule 3: Field
Research
33
Field Research: See Text Chapter 11• Field research encompasses two different
methods of obtaining data:
• Direct observation( example-bar room behavior; seat belt use, community disorder indices)
• Asking questions; modes of recording observations
• May yield qualitative and quantitative data
• Often no precisely defined hypotheses to be tested
• Can be used to make sense out of an ongoing process
Topics Appropriate to Field Research• Gives comprehensive perspective – enhances validity
• Go directly to phenomenon, observe it as completely as possible
• Especially appropriate for topics best understood in their natural setting• How street-level drug dealers distinguish customers
• Studies of ‘vice’, e.g., prostitution and drug-use.
• Aspects of physical settings, Disney World, social control
Various Roles of the Observer (Gold, 1969)• Complete participant – participates fully; true
identity and purpose are not known to subjects• E.g., posing as a bar patron; becoming a police officer; or
corrections worker
• Participant-as-observer – make known your position as researcher and participate with the group• E.g., study of active drug users; Julie Mueler and the Guardian
Angels
• Observer-as-participant – make known your position as a researcher; do not actually participate • E.g., Observational study of police patrol—”ride-along” ; research
on gangs
• Complete observer – observes without becoming a participant • E.g., court observation, Chicago neighbourhood
study( windshield studies)
Observer statusBe aware of, and document role of
researcher (extent of participation)
Be aware that all observation is subjective.
Be aware of the possible “effect of participation”.
Be aware of often competing ethical and ‘scientific’ values related to all observational studies.37
Asking Questions• Field research is often a matter of going where the action is and simply watching and listening
• Also a matter of asking questions & recording answers
• Field research interviews are much less structured than survey interviews
• Ideally set up and conducted just like a normal, casual conversation
Preparing for the Field• Access to formal organizations
• Find a sponsor, write a letter to executive director, arrange a phone call, arrange a meeting
• Access to subcultures
• Find an informant (e.g., person who works with offenders), use that person as your “in”
• Snowball sampling is useful as informant identifies others, who identify others, etc.
Module 4:Sampling
An Overview of Sampling Techniques: See Ch.8 in textSampling DistributionsThe Normal CurveSampling ErrorProbability SamplingNon-Probability SamplingIllustrations: NCVS, British Crime Survey,
Australian Crime Survey
41
Key Terms to Remember: SamplingSimple random samplingTable of random numbersConfidence intervalsStratified samplingPopulationProbability vs. non-probability samplingEqual probability of selection
method( EPSEM)Focus: random sampling from probation case
files42
Module 6: Recording Observations in the Field
• Note taking, tape recording when interviewing and when making observations
• Videotaping or photographs can make records of “before” and “after” some physical design change
• Field notes – observations are recorded as written notes, often in a field journal; first take sketchy notes and then rewrite your notes in detail
• Structured observations – observers mark closed-ended forms, which produce numeric measures
Linking Field Observations and Other Data
• Useful to combine field research with surveys or data from official records• Baltimore study of the effects of
neighborhood physical characteristics on residents’ perceptions of crime problems (Taylor, Shumaker, & Gottfredson, 1985)
• Perceptions – surveys;
• Physical problems – (1) observations, (2) actual population and crime information - census data & crime reports from police records
Strengths and Weaknesses of Field Research
• Provides great depth of understanding
• Flexibility (no need to prepare much in advance)
• More appropriate to measure behavior than surveys
• High validity; quant. measures – incomplete picture
• Low reliability – often very personal
• Generalizability – personal nature may produce findings that may not be replicated by another
• Precise probability samples can’t normally be drawn
Other Sources to ConsiderModule 7: Secondary Data, Content Analysis
46
Secondary Data: See text, ch. 12• Data from agency records – agencies collect
a vast amount of crime and CJ data
• Secondary analysis – analyzing data previously collected
• Content analysis – researchers examine a class of social artifacts (typically written documents) See Text example; p 329 Terrorist Recruitment
Topics Appropriate for Agency Records
• Most commonly used in descriptive or exploratory studies
• Content analysis often center on links between communication, perceptions of crime problems, individual behavior, CJ policy
Types of Agency Records
• Published Statistics – gov’t organizations routinely collect and publish compilations of data (e.g., ABS, BOCSAR, AIC) often available in libraries and online
• Nonpublic Agency Records – agencies produce data not routinely released (e.g., police departments, courthouses, correctional facilities)
• New Data Collected by Agency Staff – collected for specific research purposes; less costly + more control
Units of Analysis in Criminal Justice Data
Criminal Activity• Incidents• Crimes violated• Victims• Offenders
Court Activity• Defendants• Filings• Charges and Counts• Cases• Appearances• Dispositions• Sentences
Apprehension• Arrests• Offenders• Charges• Counts
Corrections• Offenders• Admissions• Returns• Discharges
Sources of Reliability and Validity Problems
• Virtually all CJ record keeping is a social process – “social production of data”
• Records reflect decisions made by CJ personnel as well as actual behavior by juveniles and adults
• Discretion factors in to recordkeeping
• CJ organizations are more interested in keeping track of individual cases than in examining patterns
• Potential for clerical errors due to volume of data
Content Analysis• Systematic study of messages – can be
applied to virtually any form of communication
• Decide on operational definitions of key variables
• Decide what to watch, read, listen to & time frame
• Analyze collected data
• Well suited to answer “who says what, to whom, why, how, and with what effect?”
Aspects of Sampling and Coding in Content Analysis 2
• Reminders:
• Remember operational definition of variables, and their mutually exclusive & exhaustive attributes
• Pretest coding scheme
• Assess coding reliability via intercoder reliability method and test-retest method
Secondary Analysis• Sources – websites, libraries
• AIC, http://www.aic.gov.au/
• BOCSAR, http://www.bocsar.nsw.gov.au/
• BJS, NCVS, ICPSR, NACJD
• Advantages – cheaper, faster, benefit from work of skilled researchers
• Disadvantages – data may not be appropriate to your research question; least useful for evaluation studies (which are designed to answer specific questions about specific programs)
Review: Key Terms in Chapters 6-12 in text