iccs 2009 idb seminar – nov 24-26, 2010 – iea dpc, hamburg, germany iccs 2009 design, databases...

ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany

ICCS 2009Design, Databases and Materials


Content

Survey DesignStatus of PreparationInternational Database

Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide

Seminar MaterialAnalysis (Do‘s and Dont‘s)


Survey Design – Objective

ICCS 2009 is intended to investigate the ways in which young people are prepared to undertake their roles as citizens.It will report on:

Student achievement (civic knowledge)Student disposition and attitudesTechers‘ teaching and class management practicesSchool governance and climateNational context of civic and citizenship educationAsian, European and Latin American regional issues

4


Survey Design – Surveys

Student SurveyStudent Achievement BookletsStudent Questionnaire

Teacher SurveyTeacher Questionnaire

School SurveySchool Questionnaire

National Context SurveyNational Context Questionnaire

5


Survey Design – Target Populations

Students: All students enrolled in the grade that represents eight years of schooling, counting from the first year of ISCED Level 1, providing the mean age at the time of testing is at least 13.5 years (usually grade 8). Students older than 17 years are not part of the target population.Teachers: All teachers teaching regular school subjects to students of the target grade (regardless of the subject or the number of hours taught) during the ICCS testing period, and have been employed at school since the beginning of the school year.

6


Survey Design – Participating Countries

38 countries from around the world

7 Source: Schulz et al., 2010, ICCS 2009: International Report, IEA, Amsterdam


Survey Design – Sample Structure

Stratified multi-stage probability sampling designRandom samples were required for each populationMinimum sample sizes:

150 schools1 intact class per school, aiming for an achieved sample size of 3,000 target grade students15 target grade teachers (or all if number of target grade teachers was less or equal to 20)

8



More than 5.300 schools, 140.000 target grade students, and 62.000 target grade teachers

9



Reporting of data from the student survey need to be cautious for some countries and reflect the following issues:

HKG and NLD did not meet sampling requirements.BFL, CZE, DNK, NZL, NOR, CHE, THA met guidelines for sampling participation rates only after replacement schools were included.ENG nearly satisfied guidelines for sample participation only after replacement schools were included.GTM, KOR, PRY surveyed the same cohort of students but at the beginning of the next school year.SVK national desired population does not cover all of international desired population.

10



Reporting of data from the teacher survey need to be cautious for some countries and reflect the following issues:

AUT, BFL, DEN, ENG, HKG, LUX, NZL, NOR, and CHE did not meet sampling requirements.CZE, SWE, THA met guidelines for sampling participation rates only after replacement schools were included.IRL nearly satisfied guidelines for sample participation only after replacement schools were included.SVK national desired population does not cover all of international desired population.GRC and NLD data from the teacher survey could not be reported.

All the above is annotated in the international reports.

11



More information about the sampling design will be detailed in chapters 2 and 10 of the ICCS 2009 Technical Report

12


Survey Design – CIVED Link

Data from 17 countries could be linked to data from the IEA CIVED study from 1999.To cover a similar target population, four countries surveyed an additional grade (grade 9): GRC, NOR, SWE, SVN

13


Content





Status of Preparation

Data itself confirmed by countriesData Processing at the IEA DPC

Data preparation doneDerived variables addedWeights added

15



Changes since last data send out (v3.1, April 2010)Current data version 3.2Scale SCSTUDOP (Principals' perceptions of student opportunities to participate in community activities) in School data file (ICG) changed so that category “not offered at school” is now coded “0” instead of receiving a missing code3 new scales added to Student Questionnaire data file (ISG):

NISB: National Index of Socioeconomic BackgroundPAREDYRS: Highest educational level in approximate years of schoolingEXPEDYRS: Expected years of schooling (recoded from SISCED)

16



Timeline for future data send out(November 23 sent out to NRCs)January 2011: put on IEA website for public download, including Technical Report and IDB User Guide

17


Content





International Database - Content

Country specific filesData files (labeled SPSS format, SAS format, national variables where applicable)

DocumentationCodebooks (WinDEM format)Technical ReportIDB User Guide

Data and documentation will be published in January 2011 on IEA website: www.iea.nl

19


International Database - Files Types

ICG School Questionnaire FileITG Teacher Questionnaire FileISG Student Questionnaire FileISA Student Achievement FileISR Student Reliability FileISE European Module Student FileISL Latin American Module Student FileISS Asian Module Student File

20



Additional grade data (GRC, NOR, SWE, SVN)JSG Student Questionnaire FileJSA Student Achievement FileJSR Student Reliability FileJSE European Module Student File

21


Cases included:School questionnaires that were returned(CPART = 3) Teacher questionnaires that were returned (TPART = 3)AND where schools met the minimum within school teacher participation rate of 50%Student data that were returned (SPART = 3)AND where schools met the minimum within school student participation rate of 50%NOTE: A student was counted participating if either achievement or questionnaire data was available (regional module data only was not sufficient)Indicator variable: INICS09 = 1

22



Note that the student files and also the teacher file may contain cases from schools where the school questionnaire has not been returned.

23



International Database - Data Structure

From the surveyID variables (record, scorer, puncher, linking) and, for students, booklet rotationListing and tracking variables (e.g., ITPARTR) Response variables and, for students, constructed response scores

24


International Database - Data Structure

From the data processing and weighting stageParticipation (SPART, TPART, CPART) and adjudication flags (INICS09)Stratification information (IDSTRATE/I)Design and estimation weightsVariables needed for variance estimation (JKZONES, JKREPS)Meta data (VERSION, IDCNTRY, DPCDATE)

25


With the information in the NAFs national adaptations could be reflected and data were recoded to fit international database structure in the majority of casesFull back-translation into English will be important for readers of the report and also for secondary research (especially if more than only country specific terms < > were adapted)

International Database - National Adaptations

26


Main goal: document the comparability or non-comparability of country dataDocumentation of important cultural and other national adaptations that affect the meaning of a variable


27


If national adaptations made recoding of data necessary, this has been documented in the National Adaptations ReportsIf international comparability is not achievable, data for international variable(s) were set to not administered (but country data were still available as national variable)If possible, data for not administered variable(s) were gang punched to a certain value

E.g., ITLANG has been gang punched to ‘1’ if only one language was used for all instruments in a country


28


National Adaptations Reports will inform and support secondary analysisWill be published as part of the ICCS 2009 complementary publication (IDB User Guide)


29


International Database - Variables

Identification variables (ID…) Tracking variables (IT…)Survey instrument variablesWeighting variablesScale scores

30


Identification Variables

Unique IdentifiersIDSCHOOL

School ID4 digitsUsed in all files

31



IDCLASSClass ID6 digitsFirst 4 digits identify schoolLast two digits used to identify class within schoolUsed in the student files (ISG, ISA, ISR, ISE, ISL, ISS)

32



IDTEACH Teacher ID6 digitsUsed in teacher file (ITG)IMPORTANT NOTE: Teachers are NOT linked to classes and NOT directly linked to students!

33



IDSTUDStudent ID8 digitsFirst 6 digits include the corresponding class (IDCLASS)Used in student data files

34


Tracking Variables

CPARTTPARTSPART Final Participation IndicatorsITLANG Administration Language (1, 2, ..., 5)IDCNTRY Country ID for ReportingVERSION Data Sendout Version (currently 32)DPCDATE Date Exported (currently 20101116)

35


Survey Variables

IC2G01A…IC2G23 survey instrument variables in school data file (ICG)IT2G01A... IT2G29I survey instrument variables in teacher data file (ITG)

36


Survey Variables

CI2COM1…CI106M1 survey instrument variables in student achievement data file (ISA)IS2G03... IS2G18G student background variables in student questionnaire data file (ISG)IS2P19A... IS2P36F student perception variables in student questionnaire data file (ISG)

37


Survey Variables

EST01A…EST12 student achievement variables in European module data file (ISE)ESP01A…ESP13D student perception variables in European module data file (ISE)LST01…LST16 student achievement variables in Latin American module data file (ISL)LSP01A…LSP12I student perception variables in Latin American module data file (ISL)ASP01A…ASP08E student perception variables in Asian module data file (ISS)

38


Survey Variables

All survey instrument variables that have been adapted for national purposes, have been recoded to fit the international schemeNational variables will be provided to country, but not be part of international database

39


Variables for Weighting and VE

School data fileTOTWGTC Final school weight – school studyJKZONEC Jackknife zones – school studyJKREPC Jackknife replicate code – school study

40



Teacher data fileTOTWGTT Final teacher weight – teacher studyJKZONET Jackknife zones – teacher studyJKREPT Jackknife replicate code – teacher study

41



Student data filesTOTWGTS Final student weight – student studyJKZONES Jackknife zones – student studyJKREPS Jackknife replicate code – student study

42


Content Knowledge Variables

PV1CIV…PV5CIV: Civic knowledge plausible valuesCentered to mean of 500 and standard deviation of 100 scale points (for equally weighted countries)NWLCIV: National civic knowledge score (WLE)

43


Scales & Derived Variables

For derived indices, see the list of indices(C:\ICCS2009\Documentation\ICCS2009_Indices.xls)Example: RELIG

Students’ belonging to a religionDummy coded to ‘No religion’ (code 0) – ‘Religion’ (code 1)

44


Scales & Derived Variables

For derived scales, see the scale documentation(C:\ICCS2009\Documentation\ICCS2009_Scales.doc)Example: POLDISC

Students' discussion of political and social issues outside of schoolIRT WLE scores with mean of 50 and standard deviation of 10 for equally weighted countriesItems included

IS2G13AIS2G13DIS2G13FIS2G13G

45


Missing Values

Special codes represent missing dataCodes contain information on nature of missing dataCodes and levels of missing data should be checked before usage

46


(SPSS)

9

99

...

Missing Values – “Omitted”

No Response

47


(SPSS)

7

97

...

Missing Values – “Invalid”

Invalid Response

48


Question not reached

Question omitted

(SPSS)6

96…

Missing Values – “Not Reached”

(SPSS)2

49

(SPSS)9

99…


(SPSS)

System-missing

.

Missing Values – “Not Administered”

• Nationally not administered variables

50


(SPSS)

5

95

...

Question 2 does not apply according to

filter question

Missing Values – “Logically Not Applicable”

51


Codebooks

Contain information related to the structure of file types including the

SourceFormatDescriptive labelsResponse option codes for all variables

One codebook file is provided for each of the instruments/ file typesAvailable in .txt format that can be opened with text editorAll dictionary information can be found directly in the SPSS file

52


Codebooks

53


Content





Seminar Material

AgendaSeminar agenda

CodebooksFor each of the file types (ICG, ITG, ISA, ISG, ISR, ISE, ISL, ISS)

DataSPSS data (one file per country and file type)

55


Seminar Material

DocumentationList of IndicesScale documentationDraft chapters of IDB User Guide

All these is DRAFT information (do not cite or quote). Information will be published in the IDB User Guide.

56


Seminar Material

InstrumentsSchool QuestionnaireTeacher QuestionnaireStudent QuestionnaireRegional Module Questionnaires

European (questionnaire part only)Latin American (questionnaire part only)Asian

PresentationsAll presentations of this seminar

57


Seminar Material

Software7-Zip: compressing and archiving files (freeware)Adobe Reader 9: reading PDF docuementsIEA IDB Analyzer: analyzing IEA data

WorkSPSS script needed for seminar hands-on training

AddIns_ICCSSupport material and examples for hands-on training of statisticial significance tests

HLM_TrainingSupport material and examples for hands-on training of HLM sample analysis

58


Content





Analyzing ICCS 2009 Data

ICCS 2009 is a survey and collected observational dataWe know what schools, teachers, and students characteristics are, what they think, and the context in which this occursWe have current cross-sectional information, while the reason or effect might have occurred a while backThe data has invaluable descriptive powerWe can make statements about associations and correlations, not causation

60



ICCS 2009 is NOT an experimentWe do not control the assignment of students and teachers to ‘treatment’ groupsEvents have already happened and we record what has happenedWe can not establish causality, or direct effects

61



School variables ONLYUse ICG file

Teacher variables ONLYUse ITG file

Student variables ONLYUse ISA, ISG, ISE, ISL, and/or ISS file

62



Student variables using school level attributes (combined)

Merge corresponding files (i.e. disaggregate school variables to students)Supported by the IEA IDB AnalyzerUse the student weight (TOTWGTS)Mind that there might be students of a school of which the school questionnaire was not returned or vice versa, so watch levels of missing data

63



School variables using aggregated student level characteristics

Not supported by the IEA IDB AnalyzerIf at all, use with caution (ecological fallacy)

64



Teacher variables using school level attributes (combined)

Merge corresponding files (i.e. disaggregate school variables to teachers)Supported by the IEA IDB AnalyzerUse the teachers weight (TOTWGTT)Mind that there might be teachers of a school of which the school questionnaire was not returned or vice versa, so watch levels of missing data

65



Teacher data should not be merged with student dataThere is no direct linkage between teachers and studentsTeachers teaching in the school a student is attending might have never seen that student or taught a lesson to the student

66


Thanks for your attention!

iccs 2009 idb seminar – nov 24-26, 2010 – iea dpc, hamburg, germany iccs 2009 design, databases...

Documents