data management seminar, 8-11th july 2008, hamburg iccs 2009 main survey field operations

Data Management Seminar, 8-11th July 2008, Hamburg

ICCS 2009 Main SurveyField Operations


Field Operations - Overview

Main Survey Operations OverviewOn-line Data CollectionSupport MaterialsMain Survey TimelineNext StepsContact Information


Main Survey Operations Overview


InstrumentPreparation

Sampling and Preparing

Survey Administration

Data Entryand Verification

SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Verification of

Adaptations

TranslationVerification

Adapt Instruments & DocumentAdaptations

CorrectAdaptations &

TranlsateInstruments

CorrectInstruments &

AssembleBooklets

LayoutVerification

CorrectInstruments

(if necessary)

Printing &Send Out

Instrument Preparation


NRCISC/IEA Sec./DPCtime

Time planning before administration

Layout verification

Implement changes &assemble booklets

Translation verification

Adapt instruments

2 weeks

1 week

1 week

2 weeks

4 weeks

Start data collection15

weeks

Instruments send out

Printing & send out

Translate instruments

Verification of adaptations

1 week

4 weeks


due dateNRC/ISC/IEAtime

Time planning before administration - SH

Layout verification

Implement changes &assemble instruments

Translation verification

Adapt & translate instruments

1 week

1 week

2 weeks

1 week

Instruments send out July 11, 2008

August 22, 2008September 19,

2008September 26,

2008October 10,

2008

October 24, 2008Start data collection

Printing & send out2 weeksOctober 24,

2008

Adapt instruments

Verification of adaptations July 25, 2008

July 18, 2008

4 weeks

4 weeks


Meanwhile…






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Contact Schoolsand get

Within-SchoolInformation

Perform Within-School

Sampling

Prepare List of Schools

and send it to DPC

Recruit and Train Test Administrators

DPC provides list of sampled

Schools

Recruit and Train Scorers

Translate School Coordinator Manual

Translate Test Administrator Manual

Recruit and Train School Coordinators

Sampling & Preparing Survey Admin.


School Contact

If not yet done, send sampling frame (list of schools) to the IEA DPCSampling of schools has been or will be done by the IEA DPCList of sampled schools has been or will be provided by the IEA DPCContact schools, ensure participationIdentify liaison person (School Coordinator)Provide School Coordinator ManualRecord status for each sampled school in WinW3S


School Contact

Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible, also combining the two to only one document)


Within-School Sampling (Students)

Two levels: students and teachersIdentification of classes on Class Listing Forms– Class name– Number of students

Random within-school sample of classesIdentification of students within sampled classes on Student Listing FormsSend list of sampled students to School Coordinator on Student Tracking FormWinW3S serves as survey administration tool


Within-School Sampling (Teachers)

Identification of eligible target grade teachers on Teacher Listing FormsWithin-school random sample of target grade teachersSend Teacher Tracking Form to School Coordinator


Within-School Sampling

WinW3S will support NRCs and DMs with– Storing school contact information– Listing of classes / teachers / students– Sampling of classes / teachers– Assignment of booklets– Survey administration– Tracking of participation

Will be subject today and tomorrow


Assigning Questionnaires

School QuestionnaireTeacher– Every teacher will receive one questionnaire

Student– Achievement Booklet (one of seven rotated forms)– Questionnaire (one form, no rotated forms in main

survey)– Regional Module Questionnaire

WinW3S will assign forms and provide labels for questionnaires


NRCSchoolstime

Contacts schools

Decision to participate

Sends listing forms

List students/teachers

Samples teachers

Sends tracking forms

Sends questionnaires

Start data collection

Name School Coordinator

Samples classes

List classes/providenumber of teachers

Time Planning before Administration


School ContactInstrumentstime

Contact schools

Send listing forms

Sample teachers

Send tracking forms

Send questionnaires

Implement changes & assemble

booklets


Printing

Data collection

Sample classes

Time Planning before Administration






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Monitor Test Administrators

Ad hoc Support

Arrange for Material Receipt

Train Scorers

Train Data Entry Staff







SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Organize Material for (direct) Data Entry

Organize Material for Scoring and Reliability Scoring

Check if Information and Material is complete

Get back to School

if necessary

Enter Participation Information

Receiving Material






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Adapt InternationalCodebooks to

National Questionnaires

Use NationalAdaptation Forms

Use adaptedCodebooks for

Data Entry in WinDEM

Codebook Adaptation






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Conduct and Monitor Scoring

Conduct and Monitor

Reliability Scoring

Organize Material forData Entry

Conduct and Monitor

Reliability Scoring

Conduct and Monitor Scoring

Retrain orDiscuss items

Scoring


Scoring

All open ended items have to be scored according to scoring guideIdeally at the beginning of the scoring process, the first incoming booklets should be scored for reliability, so that measures can be undertaken to improve reliability of scoring (e.g., discuss and retrain problematic items)Reliability scoring required for open ended items for ~700 booklets (100 booklets per rotated form) Occupations of parents/guardians have to be scored according to ISCO 88

Has been subject to Marker Training yesterday and today






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Complete Participation Information

in WinW3S

Verify DataEnterData

in WinDEM

Data Entry

EnterDouble Punching Data

in WinDEM


Data Entry

Data entry software (WinDEM) and international codebooks for the MS are provided by the DPC on August 18, 2008National staff should be trained on data entry, even if all or parts are being outsourcedComputer workstations need to be set upNational Centers enter data and double punching data in WinDEMBefore send out, National Centers need to verify data from paper questionnaires using verification checks provided in WinDEM

Will be subject tomorrow through Thursday






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


Set of Final Instruments

National Adaptation Forms(NAF) version IV

Exported WinW3S database

National Codebooks

Verified Data sets

Reliability Scoring Data

Data Submission

Documentation Data

Tracking Forms Double Punching Data






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sendout

Scoring


On-line Data Collection


On-line Data Collection (ODC)Used for all questionnaires that National Centers should answer– National Context Questionnaire– Survey Activities Questionnaire

International Option for– School Questionnaire– Teacher Questionnaire

Analysis of split half design during field trial showed that the use of the on-line mode in general is feasibleAll nine countries participating in ODC option during the FT received individual feedbackOnly countries that tried out ODC in FT can use ODC option for MS


On-line Data Collection in MS

True mixed mode designGoal:– Reduce response burden by offering respondents

preferred mode– Save time on data entry and data processing– Improve quality of data

Teachers and school principals should be assigned to the preferred mode– Knowlegde of NRC about feasibility/preferences– School coordinator could contactS school principal

and teachers

Ensure fall-back to paper questionnaire


Codebook Adaptation


Sampling and PreparingAdministration

Data Entry and

Verification

Adm

inis

trati

on

Rece

ivin

g M

ate

rial

Data

SendoutCodebook Adaptation

Prepare

ODC Monit

or

OD

C


School contactInstrumentstime

Time planning before administration ODC

Contact schools

Sample students / teachers

Send tracking forms

Send questionnaires

Implement changes & assemble

booklets


Printing

Data collection

PrintingPrepareODC

Send quest./cover letters

Send listing forms

Sample classes


On-line Data Collection (ODC)

Based on concept of cultures, one setup per language and countryParticipants receive “cover letter”– Label with login information provided by WinW3S– Culture specific URL– Personal username and password– Sent to school coordinator for distribution

Same administration period as paper questionnaires


On-line Data Collection (ODC)

NRCs can monitor completion status of on-line questionnairesIf necessary, NRCs instruct School Coordinators to remind teachers/school principals to complete the questionnaireNo data entry at National Center needed


Support Material


Support Material

Survey Operations Procedures Manual (MS)– Unit 1: School contact and material preparation– Unit 2: Within school sampling and test

administrationSchool Coordinator ManualTest Administrator ManualNational Quality Control Monitor ManualWithin School Sampling Software (WinW3S)


Support Material

Survey Operations Procedures Manual (MS)– Unit 3: On-line data collection, scoring and data entry

IEA SurveySystem DesignerScoring GuideCodebooksWindows Data Entry Manager (WinDEM)

Distribution of all materials and software packages through secure section of study website at http://iccs.acer.edu.au


Main Survey Timeline


Main Survey Timeline - SH

- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires

October 10, 2008

Submit assembled instruments and NAF version III for layout verification to ACER

September 26, 2008

Receive verified instrumentSeptember 19, 2008

Submit translated questionnaires and NAF version II for international translation verification to IEA Sec.

August 22, 2008

Receive feedback on national adaptationsJuly 25, 2008

Submit national adaptations in NAF version I to ISCJuly 18, 2008

Receive instruments & NAFJuly 11, 2008


Main Survey Timeline - SH

Final deadline for submitting data to DPCMarch 27, 2009

- Scoring of open-ended items and occupations- Data entry and verification

November –March, 2009

Proposed MS administration period (subject to country conditions)

October – December, 2008

Submit on-line questionnaires definition files for verification to DPC

October 17, 2008

Give feedback about MS procedures using theon-line Survey Activities Questionnaire

April, 2009


Main Survey Timeline – NH

- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires

February, 2008

Submit assembled instruments and NAF version III for layout verification to ACER

February, 2008

Translation verificationJanuary, 2008

Translate questionnaires and prepare NAF version II for international translation verification at IEA Sec.

December, 2008

Receive feedback on national adaptationsNovember, 2008

Submit national adaptations in NAF version I to ISCNovember, 2008

Receive instruments & NAFJuly 11, 2008


Main Survey Timeline - NH

Final deadline for submitting data to DPCAugust 28, 2009

- Scoring of open-ended items and occupations- Data entry and verification

April –August, 2009

Proposed MS administration period (subject to country conditions)

March – May, 2009

Submit on-line questionnaires definition files for verification to DPC

February, 2008

Give feedback about MS procedures using theon-line Survey Activities Questionnaire

September, 2009


Next Steps


Next Steps

National centers should plan and prepare for– School contact– Adaptation & translation of instruments– Test administrators– Scorers (CCE items and occupations)– Data management (data entry and verification)

Be prepared to receive Main Survey instruments on July 11, 2008


Contact Information


Contact Information

Field Operations / Data Management:

IEA Data Processing and Research Center

Falk Brese Michael Jung

[email protected]


Thank you for your attention!

data management seminar, 8-11th july 2008, hamburg iccs 2009 main survey field operations

Documents