computer-assisted measurement and coding of educational qualifications in surveys (camces) ·...
TRANSCRIPT
Computer-assisted measurement and coding of educational qualifications
in surveys (CAMCES)
ESRA conference Ljubljana, July 15/19, 2013
Silke SchneiderGESIS – Leibniz Institute for the Social Sciences
1
Contents• Project background• Project motivation
– Best practice – Persistent problems
• Proposing a solution
2
General information• CAMCES: Computer Assisted Measurement and Coding
of Educational Qualifications in Surveys• Funded for 3 years (07/2013-06/2016)• Located in GESIS’ Survey Design and Methodology
Department• Project Team:
– Silke Schneider– Jessica Herzing– Verena Ortmanns
3
Concept: educational attainment• Educational attainment = highest educational
qualification achieved by an individual• Key social background variable• Correlated but different:
– School grades– Knowledge, skills and competences (as measured in
standardised tests, e.g. PISA, PIAAC…)– Investments in education (years of education)
4
What does educational attainment measure?
• Eligibility for future education and employment (formal qualification)
• Validated competences (validation through examination, not standardised testing)
• Proxy/signal for actual qualification (i.e. competences) in market processes
• Proxy/Signal for third variables (trainability, perseverance, motivation, status…)
5
(...)
Relevance for social and policy research
• Micro level: result of an individual’s educational career– Core variable for studies on social structure, inequality
and mobility– Control variable for many other studies– Applies to the whole adult population – Parental education: indicator of social background– Less data collection effort than for skills and
competences• Macro level: Output and structure of educational systems
6
(...)
Measurement in cross-national surveys
• Cross-national research continuously grapples with the comparable measurement of educational attainment.
• Main limitations of available measures:– Years of education: limited validity/reliability, only time
invested in education. No effects of types of education or certification assumed.
– CASMIN: restricted scope of countries, documentation lacking
– ISCED: usually only levels implemented, then no effects of types assumed and heaping in specific categories; inconsistent implementation in different surveys
7
Best practice in general social surveys
• Country-specific questions and responses• Normally 1 or very few questionnaire items• Qualifications or education levels aggregated in
5-15 categories (different across surveys)• Presented to respondent on showcard• Coding is also country-specific• Ex-ante output harmonisation using "bridging"
8
Persistent problems: data collection
• Educational expansion and reforms multiplied number of qualifications held in population but invisible in much data
• Immigration and student mobility introduce foreign qualifications into survey country
• Even more complicated when collecting proxy information (e.g. parents’ or partner’s education)
9
Persistent problems: data processing and analysis
• No established standard (though ISCED dominant)• Lack of standards makes variables difficult to
harmonise over time, surveys and countries• Resulting in crude aggregations of limited analytic
value, flexibility and comparability - we actually introduce extra error!
• Same result with crude ex-ante output harmonisation
• Plus: error prone, data quality management difficult• Validity of research outputs?
10
aR2, reg ISEI on ..., ESS round 5
20
25
30
35
40
45
50
55
60
65
70
HU BG CY PL UA PT SI ES DE SK GR FI ø NL HR CZ BE RU FR IL DK SE CH NO EE IE UKedulvla edulvlb ISCED 2011 ES-ISCED yrs
Conclusion
• Current best practice does not satisfy user needs for – valid, – recodeable and – comparable educational attainment variables
• Instead produces variables of limited analytic potential and comparability
12
(...)
First steps: concept and classification
✓ Development of a detailed cross-national coding scheme for educational attainment
✓ Develop bridging tables for around 30 European countries
✓ Quality checks of resulting variables• But: long lists of response categories difficult to handle
in survey process– Too much text on show cards– Demanding for respondents and interviewers
13
Current developments• Official ISCED mappings currently being
developed world-wide• More and more surveys have been using
computer technology (CAWI, CAPI, CATI)• Survey software available• Make better use of technical opportunities by
developing new tools: CAMCES
14
Instrument design and database interface
• Options:– database search field with probes– search field with autocomplete and probes– custom showcards (what appears depends on
available information such as age, country of education)
• most simple variant: search tree/nested showcards• Which options most realistic for CAPI, which for CAWI?
– Exploratory items in online panel: open question– Cognitive pretesting and focus group– Pilot studies
15
Project aim:provide a new service
• consisting of survey measurement instruments, database and software interface
• that enable the accurate, detailed and cross-nationally comparable measurement, coding and harmonisation of highest educational qualification obtained
• in computer-assisted surveys (where content can be shown visually to respondents, i.e. CAPI, CAWI)
• during data collection• covering all European countries
16
Proposed work packages1. Development and validation of an international
database of educational qualifications;2. Development and test of questionnaire items and
database queries;3. Development and test of an interface for computer
assisted surveys;4. Test of the integrated service in CAWI and CAPI
pilot studies
17
(...)
Expected benefits• Efficiency through standardisation and automation of
data collection, coding and harmonisation processes• Better coverage of foreign, rare, and outdated
educational qualifications• More accurate information and analytic value through
detailed measurement• Database useable as standard resource for post-hoc
harmonisation for researchers and data archives
18
(...)
Potential extensions of the project• Extend range of countries covered• Development of instruments for telephone interviews (CATI) • Extension of instrument to measure related concepts:
– educational transitions and educational career– dropout
• Enrich database by relevant related information:– educational programmes– educational institutions– fields of education and training
• Integrate plausibility checks in surveys? Risky (artificial consistency)
19
(...)
Cooperation partners• DIW, Berlin (SOEP): Jürgen Schupp, Simone Bartsch• AIAS, University of Amsterdam (WageIndicator Web
survey): Kea Tijdens• TNS Infratest: Ulrich Schneekloth, Sarah Schmidt• Informally:
– GESIS experts on cognitive pretesting, questionnaire translation...
– DASISH experts on automated occupation coding: Peter Elias, Eric Harrison
20
(...)
