data management seminar, 8-11th july 2008, hamburg 1 summary common sources of error

Data Management Seminar, 8-11th July 2008, Hamburg1

SummaryCommon Sources of Error


InstrumentPreparation

Sampling and Preparing

Survey Administration

Data Entryand Verification

SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors During Instrument Preparation

Errors in translations (especially: careful if cooperate with other countries)The same terms in different questions translated differentlyComments from translation verification not implementedNational adaptations not or incorrectly documented on NAFDeletion of international questions and categories without strong reason and approvalInternational question stem changed


Errors During Instrument Preparation (ODC)

Conversion in SurveySystem started before paper instruments are reviewed start AFTER international verificationCopy and paste to wrong text, numeral, or instruction check paper version against previewPaper questionnaires and ODC questionnaires do not match in terms of disabled questions, variables or additional elements verify within country before submitting to IEA DPC






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors During (Within-school) Sampling

School sampling frame incorrect / major parts of the population not includedSample not approved by the IEA DPCReplacement of ineligible schools (frame errors like closed schools)Missing selection of participation in Regional Module or on-line data collection when creating new project in WinW3SSchools did not list all eligible students and teachers or left out eligible but excluded them follow-up






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors During Codebook Adaptation

Length of a national variable specified to small (one digit for a variable with 6 or more options)Wrong labeling/numbering of categories or dimensions when they were deleted or addedDeletion of variables for questions that are not administered set to „hide“ in WinDEMCodebooks for different data entry computers are differentIf ODC is used, codebooks for WinDEM do not match with ODC data (changes made after export from SurveySystem)






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors During Administration

Instruments distributed to wrong person (only to designated individuals)Sampled student/teacher replaced do not substitute!Participation status not tracked (Tracking Forms) or monitored (On-line Monitor Report)Spare Booklet assignment not documented (Tracking Forms)






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors During Material Receipt

Receipt of materials not tracked appropriatelyIncomplete materials contact school coordinator and try to resolveIncomplete materials because of low on-line participation consider following up to schools with paper questionnairesIf ODC used, “Monitor” reports not taken into account continuously






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors during Scoring

Single items are all scored by the same scorerReliability booklets are marked by the same scorer in the instrument and in the reliability Scoring sheetMain scorers enter codes directly in reliability booklets before reliability Scoring has been done






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors During Data Entry in WinW3S

If ODC was used, questionnaire type not tracked correctly in WinW3SParticipation status not entered in WinW3SSpare booklet assignment not performed in WinW3SITLANG in NAF and WinW3S not set if more than one language was used


Errors During Data Entry in WinDEM

Not all materials enteredWrong or duplicate IDs, column shifts, invalid values(All avoided by using WinDEM) train staff thoroughlyWrong treatment of out-of-range values enter data “as is”!Missing values used incorrectly (use ‘9’ for any omitted response, ‘7’ for any invalid response and valid values for all interpretable responses)


Errors During Data Entry in WinDEM

Data entered on different computers not combined or combined multiple times (leading to duplicate IDs)Values for Check-All-That-Apply questions used incorrectly (use 1 for CHECKED, 2 for NOT CHECKED)Filters interpreted enter data “as is”!


Errors during Verification

WinDEM checks not performedChecks performed but data not correctedManipulating the data files with MS Excel (data file will be cut of after 255 variables)






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dou

t

Scoring


Errors during Data Send-out

Data submitted, but without documentation (Tracking Forms, NAF)NAF or instruments not sent for all languages usedCompleted Tracking Forms are printed from WinW3S send paper copies or scans of original forms WinW3S file not exportedDouble-punching data missingDocumentation incomplete or does not match dataAfter data is submitted, national centers are not prepared to answer questions from the IEA DPC Although data processing is exhausting, it’s not time for vacations, yet ☺


SummarySummaryData Processing at the IEA DPC Data Processing at the IEA DPC


ContentContent

• Data and Documentation Submission• Data Processing at the IEA DPC

– Adapting the File Structure– Cleaning Steps

• Further Schedule• Contact Addresses


In general 3 month after last day of administrationLatest possible dates– Southern Hemisphere: March 27, 2009– Northern Hemisphere: August 28, 2009

Material to be Submitted to IEA Material to be Submitted to IEA DPCDPC


Set of Final Instruments

National Adaptation Forms(NAF) version IV

Exported WinW3S database

National Codebooks

Verified Data sets

Reliability Scoring Data

Data Submission

Documentation Data

Tracking Forms Double Punching Data


Please use the WebFTP sites to exchange materials between countries and the - ISC at ACER and/or- IEA Secretariat and/or - IEA DPC Please inform the recipients via email as soon you have uploaded any materialsFurther information on the IEA DPC WebFTP-Server has been provided in a separate WebFTP-Server Manual

File-Exchange using the IEA File-Exchange using the IEA DPC WebFTP-ServerDPC WebFTP-Server


ContentContent



• Further Schedule• Contact Addresses


Data Processing at the IEA DPCData Processing at the IEA DPC

Data & Codebook

sInstrument

s & Forms

INP

UT

IEA DPC

Country

1. Import to SAS and adapting the

file structure


Adapting the File StructureAdapting the File Structure

WinW3S database and WinDEM data files are merged and converted to SAS data filesUnique system numbers (CSYSTEM, TSYSTEM, SSYSTEM) are added– System numbers usually correspond to the

position of the record in the original WinDEM data files

– If a data set is missing either in WinW3S or in WinDEM, higher system numbers will be assigned


Structure of the Merged Data Structure of the Merged Data SetSet

Variable ValueIDBOOK 2

IDPUNCH 111

IDSCORA 101

...,ITPART1..2,IDSCHOOL,IDCLASS,...

3...1001

100101

IDSTUD 10010101

IDCHECK 270

TOKEN01 3

CB2PDO1 1

SSYSTEM,... 000001

Addedduring

cleaningprocess

Example: Student Achievement Data File

Information:

TrackingForms

Information:Student

AchievementBooklets

WinDEM(ISAxxxC2.DBF)


Data Processing at the IEA Data Processing at the IEA DPCDPC

Country Databas

eNew Data

Structure

Reports, Statistic

s & Docu.

INP

UT

OU

TP

UTStructur

e Check

Background

Cleaning

IEA DPC

Country

ID Cleaning

Linkage Cleanin

g

Data & Codebook

sInstrument

s & Forms2. Cleaning

Country communication will continue after data send-out!


ContentContent



• Further Schedule • Contact Addresses


Further ScheduleFurther Schedule

Software: – July 21, 2008: WinW3S– August 18, 2008: SurveySystem– August 18, 2008: WinDEM

Documentation in Survey Operations Procedures Manuals:– July 21, 2008: SOP (MS), Unit 2 (WinW3S I)– August 18, 2008: SOP (MS), Unit 3 (WinW3S II,

WinDEM, SurveySystem)

Material Submission for Main Survey: March 27, 2009 (SH); August 28, 2009

(NH)


ContentContent



• Further Schedule • Contact Addresses


Contact AddressesContact Addresses

tel: +61 3 9255 5555

e-mail:[email protected]

Web: http://iccs.acer.edu.au

tel: +31 20 625 3625 e-mail: [email protected]

InternationalStudy Center (ISC)

IEA Secretariat


Contact AddressesContact Addresses

tel: +49 40 48 500 750

e-mail: [email protected]

IEA DPC Sampling

IEA DPC tel: +49 40 48 500 611

e-mail: [email protected]: https://webftp.iea-

dpc.org


Thank you very much for your Thank you very much for your attention!attention!

data management seminar, 8-11th july 2008, hamburg 1 summary common sources of error

Documents