data collection and reliability all this data, but can i really count on it??
Post on 04-Jan-2016
213 Views
Preview:
TRANSCRIPT
Questionnaires
• Controlling problems• Equal numbers of positively and negatively
worded statements• Alternating positive and negative statements • Providing confidentiality or anonymity to
respondents
Designing Questionnaires
• Online resources• www.surveymonkey.com• http://pareonline.net/getvn.asp?v=5&n=3• http://www.peecworks.org/PEEC/PEEC_Inst/I0
004E536• http://www.statpac.com/surveys/
Questionnaires
• Take the following questionnaire
http://www.surveymonkey.com/s/PNBKFJ7
• Strengths? Weaknesses?
Observations
• Observations - direct observations of behaviors• Provide first hand account (ameliorates issues
of self-reporting in questionnaires)• Natural or controlled settings
• Ex – classroom vs. lab (child attachment studies)
• Structured or unstructured observations• Ex – frequency counts vs. narrative record
• Detached or involved observers
Observations
• Inference• Low inference - involves little if any inference
on the observers part• On-task/Off-task behavior instrument
• High inference - involves high levels of inference on the observers part
• Teacher effectiveness – PDE form 430
Observations
• Controlling observer effects• Observer bias
• Training• Inter-rater reliability (Cronbach’s alpha)• Multiple observers
• Contamination - knowledge of the study influences the observation
• Training• Targeting specific behaviors• Observers do not know of the expected outcomes• Observers are “blind” to which group is which
Observations
• Observer effects• Halo effectHalo effect - initial ratings influence subsequent
ratings
• Hawthorne effectHawthorne effect - increased performance results from awareness of being part of study
• LeniencyLeniency - wanting everyone to do well
• Central TendencyCentral Tendency - measuring in the middle
• Observer DriftObserver Drift - failing to record pertinent information
Examples to critique
• Measures• Questionnaire – Psychological School
Membership Survey used with middle school students
• Interview protocol – for teachers & counselors regarding professional development issues
• Observation instrument – PDE 430 for student teachers
• What are 2 benefits and 2 limitations of this measure?
Validity/Reliability and Trustworthiness
• Why do we need validity and reliability in quantitative studies and “trustworthiness” in qualitative studies?
We can’t trust the results if we can’t trust the
methods!
Thought Question
• On the ACT and SAT assessments, there is a definitive script that test administrators are required to follow exactly. What measurement issue are the test makers addressing?
Reliability of Measurement
• Reliability - The extent to which measures are free from error
• Error is measured by consistency
Reliability of Measurement
• Sources of error• Test construction and administration
• Ambiguous questions, confusing directions, changes in scoring, interrupted testing, etc.
• Subject’s characteristics• Test anxiety, lack of motivation, fatigue,
guessing, etc.
Reliability of Measurement
• Reliability• Measurement
• 0.00 indicates no reliability or consistency• 1.00 indicates total reliability or consistency• < .60 = weak reliability• > .80 = sufficient reliability
Reliability of Measurement
• Types of reliability evidence• Test-retest
• Testing the same subject using the same test on two occasions
• Limitation - carryover effects from the first to second administration of the test
• Equivalence (i.e. parallel form)• Testing the same subject with two parallel (i.e. equal)
forms of the same test taken at the same time• Limitation - difficulty in creating parallel forms
Reliability of Measurement
• Internal consistency• Testing the same subject with one test and
“artificially” splitting the test into two halves
• Limitations - must have a minimum of ten (10) questions
• Often see “Chronbach’s alpha” for reliability coefficient
Reliability of Measurement
• Agreement/ Inter-rater reliability• Observational measures• Multiple observers coding similarly
Reliability of Measurement
• Enhancing reliability• Standardized administration procedures
(e.g. directions, conditions, etc.)• Appropriate reading level• Reasonable length of the testing period• Counterbalancing the order of testing if
several tests are being given
Next Week
• Tuesday – Validity and Review• Thursday – Opportunity to display brilliance!
• Multiple choice
• Short Answer
• Essay – TAKE HOME. Critique Triplett article (see next slide).
Take Home Exam
Critique the article based on the following:• Introduction and research problem• Review of literature/ theoretical framework• Methods of data collection (including participants)
and data analysis• Results and conclusions including issues of
trustworthiness. Be sure to address whether we should trust the claims that the authors have made and why we should or should not trust the claims.
top related