reporting international test data ch. 3 measuring civic knowledge and understanding iccs nrc meeting...
TRANSCRIPT
Reporting International Test DataCh. 3 Measuring Civic Knowledge
and Understanding
ICCS NRC meeting
Madrid, Feb 2010
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge
• Instrument
• Described scale
• Results
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge - Instrument
• 80 item (reduced to 79 - CI2HRM2 removed)• 73 MCQ, 6 constructed response• 62 new, 17 CIVED trend• Items showing extreme item by country
interaction (>1.5 logits) removed from scaling for estimates of test scores in individual countries
• Report to include example release items illustrating range of item types and content (including % correct by country and scale location)
NRC MeetingMadrid
February, 2010
X 2 X
XX XX XX XX
XXX XXXX
XXXXX 1 XXXXX
XXXXXXX XXXXX
XXXXXX XXXXXXX
XXXXXXXX XXXXXXXXX
XXXXXXXXXX 0 XXXXXXXX
XXXXXXXXX XXXXXXXX
XXXXXXXXX XXXXXXXXX XXXXXXXXX
XXXXXXXX XXXXXXXX
XXXXX -1 XXXXX
XXXXX XXXX
XXX XXX
XX X X
-2 X X
14 25 5 37 46 75 2 4 49 56 74 51 71 6 9 28 55 59 77 10 32 34 68 19 27 44 20 30 33 36 65 72 26 40 41 50 53 57 58 61 70 11 16 18 21 23 31 64 66 69 12 17 35 76 78 29 3 38 43 47 63 7 13 15 22 48 62 67 79 1 52 54 42 60 73 24 39 45 8
• rp = 0.62 (relative shift item difficulties = ln(0.62/0.36) ≈0.49 logits
Measuring Civic and Citizenship Knowledge – Described Scale
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Described Scale
• Item map (ordered item descriptors)
0.70 Justifies the use of the separation of powers between the judiciary and the parliament.
0.70 Evaluates a risk of violent protest action in terms of possible outcomes.
0.67 Integrates government accountability through FOI legislation with the capacity of citizens to keep themselves informed.
0.56 Describes national elections as an element of "democracy".
0.53 Justifies the opposition to movie censorship as a possible restriction of freedom of expression.
0.5 Integrates the responsibility of citizens to obey the law with the right of citizens to vote in elections.
… …
-0.19 Illustrates with an example the responsibility of voters in democracy.
-0.19 Interprets that a statement expressing a position about "flags" and "anthems" is an opinion rather than a fact.
-0.2 Relates equity and social justice to the provision of services by governments without charge.
-0.37 Generalizes the impact of environmental degradation across to the whole world.
-0.4 Generalizes learning the main language in a country to potential for community participation.
-0.4 Describes the applicability of the universal declaration of human rights in terms to all people.
… …
-0.86 Relates the responsibility of citizens to obey the law with the right of citizens to vote in elections.
-0.96 Relates equality and inclusiveness to equity as necessary conditions for accepting people who have served punishment for crimes in the community.
-0.97 Relates an example from a community setting to the concept of equity.
-1.19 Hypothesizes about the relationship between the secret ballot and freedom of voter choice.
-1.77 Identify informing consumers as a key aspect of promoting ethical consumerism
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Described Scale
• Item map (ordered item descriptors)
3 0.70 Justifies the use of the separation of powers between the judiciary and the parliament.
3 0.70 Evaluates a risk of violent protest action in terms of possible outcomes.
3 0.67
Integrates government accountability through FOI legislation with the capacity of citizens to keep themselves informed.
2 0.56 Describes national elections as an element of "democracy".
2 0.53 Justifies the opposition to movie censorship as a possible restriction of freedom of expression.
2 0.5 Integrates the responsibility of citizens to obey the law with the right of citizens to vote in elections.
… … …
2 -0.19 Illustrates with an example the responsibility of voters in democracy.
2 -0.19 Interprets that a statement expressing a position about "flags" and "anthems" is an opinion rather than a fact.
2 -0.2 Relates equity and social justice to the provision of services by governments without charge.
1 -0.37 Generalizes the impact of environmental degradation across to the whole world.
1 -0.4 Generalizes learning the main language in a country to potential for community participation.
1 -0.4 Describes the applicability of the universal declaration of human rights in terms to all people.
… … …
1 -0.86 Relates the responsibility of citizens to obey the law with the right of citizens to vote in elections.
1 -0.96
Relates equality and inclusiveness to equity as necessary conditions for accepting people who have served punishment for crimes in the community.
1 -0.97 Relates an example from a community setting to the concept of equity.
B1 -1.19 Hypothesizes about the relationship between the secret ballot and freedom of voter choice.
B1 -1.77 Identify informing consumers as a key aspect of promoting ethical consumerism
395
479
563
L1
L2
L3
B1
0.8 logits
NRC MeetingMadrid
February, 2010
300
350
400
450
500
550
600
650
700
750
Measuring Civic and Citizenship Knowledge – Described Scale
2 Items
22 Items
16 Items
39 Items
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Described Scale
Level 1
Students working at Level 1 demonstrate familiarity with equality, social cohesion and freedom as principles of democracy. They relate these broad principles to everyday examples of situations in which protection of or challenge to the principles are demonstrated. Students also demonstrate familiarity with fundamental concepts of the individual as an active citizen: they recognise the necessity for individuals to obey the law; they relate individual courses of action to likely outcomes; and they relate personal characteristics to the capacity of an individual to effect civic change.
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Described Scale
Level 2
Students working at Level 2 demonstrate familiarity with the broad concept of representative democracy as a political system. They recognise ways in which institutions and laws can be used to protect and promote a society's values and principles. They recognise the potential role of citizens as voters in a representative democracy, and generalise principles and values from specific examples of policies and laws (including human rights). Students demonstrate understanding of the influence that active citizenship can have beyond the local community. They generalise the role of the individual active citizen to broader civic societies and the world.
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Described Scale
Level 3
Students working at Level 3 make connections between the processes of social and political organisation and influence, and the legal and institutional mechanisms used to control them. They generate accurate, hypotheses on the benefits, motivations and likely outcomes of institutional policies and citizens' actions. They integrate, justify and evaluate given positions, policies or laws based on the principles that underpin them. Students demonstrate familiarity with broad international economic forces and the strategic nature of active participation.
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Described Scale
Level 1: broad concepts; “big ideas”; mechanistic
Specificity
Level 2: interconnectedness of systems; wider sphere of citizen influence
Evaluation
Level 3: holistic; strategic
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Results
Comparison of means
• Means range from 380 to 576– Levels 1 to 3
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Results
Comparison of means
Multiple comparison
Country
Finland ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Denmark ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Korea, Republic of ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Chinese Taipei ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Sweden ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
P oland ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Ireland ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Switzerland † ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Liechtenstein ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Italy ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Slovak Republic² ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Estonia ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
England ‡ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
New Zealand ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Slovenia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Norway † ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Belgium (Flemish) ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Czech Republic † ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Russian Federation ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Lithuania ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Spain ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Austria ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Malta ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Chile ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Latvia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Greece ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Luxembourg¹ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲ ▲ ▲ ▲
Bulgaria ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲
Colombia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲
Cyprus ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲
Mexico ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲
Thailand ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲ ▲ ▲ ▲
Guatemala ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲
Indonesia ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲
P araguay ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▲
Dominican Republic ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
▲ Average achievement signif icantly higher than in comparison country
▼ Average achievement signif icantly low er than in comparison country
† Met guidelines for sampling paticipation rates only after replacement schools were included.
‡ Nearly satisfied guidelines for sample participation only after replacement schools were included.
¹ Survey administration did not follow international guidelines.
² National Desired Population does not cover all of International Desired Population
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Results
LevelLevel boundary
Items (N)
Items (%)
Students All (%)
Students Max (%)
Students Min (%)
3 16 20 28 58 1
>563
2 39 49 31 39 7
>479
1 22 28 26 44 10
>395
B1 2 3 16 61 2
Comparison of levels
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Results
Level No. countries with max % students in level
3 14
2 14
1 8
B1 2
Comparison of levels
NRC MeetingMadrid
February, 2010
Measuring Civic and Citizenship Knowledge – Results
Level No. countries > 60% students in levels
2- 3 23
B1 - 1 7
Comparison of levels
NRC MeetingMadrid
February, 2010
Gender differences in achievement• International means
– Girls 511– Boys 489 – Difference is statistically significant
• Mean for girls higher in all countries– Statistically significantly higher in 32 countries
• Range of mean differences 2 to 48 (median =22)
Measuring Civic and Citizenship Knowledge – Results
NRC MeetingMadrid
February, 2010
Trend Comparisons
• Calculated only on the basis of 17 trend items (different metric to ICCS)
• Dimensionality conflating results for complete scaling from both ends– CIVED through representativeness of
trend items– ICCS through broadening of framework
and constructs
Measuring Civic and Citizenship Knowledge – Results
NRC MeetingMadrid
February, 2010
Comparisons over time
• 17 countries used same translations of CIVED items for comparison
• CIVED population mean = 500
• ICCS population mean = 484– Difference is statistically significant
5 countries (stat sig one country)
11 countries (stat sig six countries)
Measuring Civic and Citizenship Knowledge – Results