Download - Assessing Progression in Creativity and Critical Thinking Skills by Stéphan Vincent-Lancrin (OECD)
ASSESSING PROGRESSION IN CREATIVE
AND CRITICAL THINKING SKILLS
Stéphan Vincent-Lancrin Deputy Head, Senior Analyst and Project Lead, OECD Centre for Educational Research and Innovation
Paris, 20 June 2016
objectives of the meeting
• What are we talking about when we talk about creativity and critical thinking in the higher education context?
• Peer learning about pedagogies trying to intentionally foster students’ creativity and critical thinking – Can we group them in categories?
– What characterises them?
• How are students assessed to show the acquisition of these skills?
• What advice to start an action research in education aligned with what we are doing in schooling?
Objectives of the meeting
• First day mainly about pedagogies:
– Design thinking
– Entrepreneurship
– Other approaches
• Second day morning mainly about evaluation and assessment:
– Standardised tools
– Faculty-designed asssessments
– Qualification frameworks
• Second day afternoon
– Discussing a possible international action research to showcase how higher education institutions can foster their students’ creativity and critical thinking
Structure of the meeting
skills for innovation
Skills and education for innovation « 21st Century Skills »
Innovation
Skills
Education and
training
Skills that tertiary-educated professionals
report as very important in their job
Percentage of employees reporting the following skills as very important in their job
Source: OECD, based on REFLEX and HEGESCO data
22.7
30.6
40.2
40.3
40.4
40.4
40.5
41.8
46.5
48.0
50.0
53.4
54.2
55.0
56.5
56.9
58.6
60.8
61.7
0 10 20 30 40 50 60 70
knowledge of other fieldswrite and speak a foreign language
assert your authoritynegociate
alertness to opportunitiespresent ideas in audience
willingness to question ideasmobilize capacities of others
analytical thinkingcome with news ideas/solutions
write reports or documentsacquire new knowledge
master of your own fieldcoordinate activities
use computers and internetmake your meaning clear
work productively with othersperform under pressure
use time efficiently
Critical skills for the most innovative jobs
(according to tertiary-educated workers) Likelihood (odds ratios) of reporting the following skills: people
in the most innovative jobs vs. least innovative jobs
Source: Avvisati, Jacotin and Vincent-Lancrin (2013), based on REFLEX and HEGESCO data
1.83
2.05
2.08
2.09
2.15
2.19
2.19
2.20
2.35
2.36
2.36
2.42
2.51
2.58
2.60
2.71
2.81
3.00 3.90
1
assert your authorityknowledge of other fields
negociateperform under pressure
use time efficientlywork productively with others
use computers and internetwrite and speak a foreign language
write reports or documentsmaster of your own field
make your meaning clearmobilize capacities of others
acquire new knowledgecoordinate activities
analytical thinkingalertness to opportunitiespresent ideas in audience
willingness to question ideascome with news ideas/solutions
Top 10 skills for future jobs according to
employers (2016)
2020
1. Complex problem solving
2. Critical thinking
3. Creativity
4. People management
5. Coordinating with others
6. Emotional intelligence
7. Judgment and decision making
8. Service orientation
9. Negotiation
10. Cognitive flexibility
2015
1. Complex problem solving
2. Coordinating with others
3. People management
4. Critical thinking
5. Negotiation
6. Quality control
7. Service orientation
8. Judgment and decision making
9. Active listening
10. Creativity
World Economic Forum
What individual skills should
education systems foster?
Technical skills (know-what and know-
how)
Creativity and critical thinking
skills (Critical thinking,
observation, curiosity, ability to make
connections, imagination,...)
Social and behavioural skills (Self-confidence, energy,
perseverance, passion, leadership, collaboration,
communication)
Some comments on these skill
categories
• They are domain-specific
– Skills are generally domain-specific: one is creative in a field, one knows how to behave/communicate in a specific context, one has problem-solving skills in a field, one has content knowledge in a field
• They can become « domain-generic »
– A skills becomes « domain-generic » when one has gained it in a number of domains or settings, so that it becomes a « habit of mind » (a disposition or a stabilised skill) that one can apply to new fields
• They overlap and may reinforce each other
But
• They are different and cannot be reduced to a single skill (or measure)
Technical skills
Skills in thinking and
creativity
Behavioural and social skills
(character)
Skills and education for innovation « 21st Century Skills »
Well-being
Skills
Education and
training
higher education for innovation
Higher education for innovation
Innovation
Skills
Education and
training
• Which graduates get highly innovative jobs?
• Do higher education systems foster skills for innovation?
• Which pedagogic practices are associated with highly innovative jobs?
What share of graduates of a given field
have a highly innovative job?
20.3
22.9
23.4
25.4
27.6
28.1
28.1
28.4
28.8
30.9
32.7
36.5
37.6
0 10 20 30 40 50
law
health
humanities
others
business
sciences & maths
education
average
social sciences
architecture
agriculture
arts
engineering & computing
Product /service innovation
What share of graduates of a given field
have a highly innovative job?
12.4
12.6
13.2
15.2
16.0
18.4
18.6
21.0
23.9
24.2
28.6
28.8
38.4
0 10 20 30 40 50
law
others
humanities
social sciences
education
health
business
average
arts
agriculture
architecture
sciences & maths
engineering & computing
Technology, tool, instrument innovation
What share of graduates of a given field
have a highly innovative job?
30.4
31.5
35.1
36.1
37.6
38.0
38.3
38.5
39.5
39.9
41.7
41.7
48.3
0 10 20 30 40 50
law
others
business
humanities
social sciences
arts
average
health
agriculture
architecture
engineering & computing
education
sciences & maths
Knowledge / method innovation
Higher education for innovation
Innovation
Skills
Education and
training
• Which graduates get highly innovative jobs?
• Do higher education systems foster skills for innovation?
• Which pedagogic practices are associated with highly innovative jobs?
Strong points of higher education
0 2 4 6 8 10 12 14 16 18 20
assert your authoritymobilize capacities of others
negociatealertness to opportunities
willingness to question ideascoordinate activities
write and speak a foreign languagemake your meaning clear
come with news ideas/solutionspresent ideas in audienceknowledge of other fields
use time efficientlyuse computers and internet
perform under pressurework productively with others
write reports or documentsacquire new knowledge
analytical thinkingmaster of your own field
Ranking of 3 top strong skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data
Strong points of higher education
0 2 4 6 8 10 12 14 16 18 20
assert your authoritymobilize capacities of others
negociatealertness to opportunities
willingness to question ideascoordinate activities
write and speak a foreign languagemake your meaning clear
come with news ideas/solutionspresent ideas in audienceknowledge of other fields
use time efficientlyuse computers and internet
perform under pressurework productively with others
write reports or documentsacquire new knowledge
analytical thinkingmaster of your own field
Ranking of 3 top strong skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data
Weak points of higher education
0 2 4 6 8 10 12 14 16 18 20
acquire new knowledgemaster of your own field
make your meaning clearcoordinate activities
work productively with otherswillingness to question ideas
analytical thinkingperform under pressure
use time efficientlyalertness to opportunities
come with news ideas/solutionswrite reports or documents
mobilize capacities of othersknowledge of other fields
use computers and internetpresent ideas in audience
assert your authoritynegociate
write and speak a foreign language
Ranking of 3 top weak skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data
Weak points of higher education
0 2 4 6 8 10 12 14 16 18 20
acquire new knowledgemaster of your own field
make your meaning clearcoordinate activities
work productively with otherswillingness to question ideas
analytical thinkingperform under pressure
use time efficientlyalertness to opportunities
come with news ideas/solutionswrite reports or documents
mobilize capacities of othersknowledge of other fields
use computers and internetpresent ideas in audience
assert your authoritynegociate
write and speak a foreign language
Ranking of 3 top weak skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data
Higher education for innovation
Innovation
Skills
Education and
training
• Which graduates get highly innovative jobs?
• Do higher education systems foster skills for innovation?
• Which pedagogic practices are associated with highly innovative jobs?
Innovation and the relative emphasis on
practice- and theory-based instruction
0.95
1
1.05
1.1
1.15
1.2
engineering business health education science others
practice score theory score
Odds ratios between innovators and non-innovators, by field of study
Source: OECD, based on REFLEX and HEGESCO data
Innovation and the relative emphasis on
practice- and theory-based instruction
0.95
1
1.05
1.1
1.15
1.2
any innovation technology, tools product, service knowledge,methods
practice score theory score
Odds ratios between innovators and non-innovators, by type of innovation
Source: OECD, based on REFLEX and HEGESCO data
ICT-enhanced pedagogic models to
foster « skills for innovation » in STEM
• Virtual and remote laboratories
• Educational games
• Technology-enhanced cooperative learning
• Real-time formative assessment
• Skills-based assessment
Problem-based learning in higher
education
Please cite this paper as:
Hoidn, S. and K. Kärkkäinen (2014), “Promoting Skills forInnovation in Higher Education: A Literature Review on theEffectiveness of Problem-based Learning and of TeachingBehaviours”, OECD Education Working Papers, No. 100,OECD Publishing.http://dx.doi.org/10.1787/5k3tsj67l226-en
OECD Education Working PapersNo. 100
Promoting Skills forInnovation in HigherEducation
A LITERATURE REVIEW ON THEEFFECTIVENESS OF PROBLEM-BASEDLEARNING AND OF TEACHING BEHAVIOURS
Sabine Hoidn, Kiira Kärkkäinen
• PBL more beneficial regarding long-term retention and application of knowledge and skills
• PBL has positive impact on students’ motivation, satisfaction, and attitudes toward learning
• Traditional approaches seem to have an edge for test scores and knowledge acquisition, but not significant
fostering and assessing creativity and critical thinking in education
• 2 networks:
– Higher education
– School (primary and secondary)
• 3 possible domains:
– One STEM discipline
– One arts education discipline
– One generic approach to innovation/creativity
• Activities in primary-secondary education:
– Develop a pedagogical toolkit (rubric, pedagogical activities and assessments, student work)
– Document pedagogies to foster the selected skills/dispositions
• Activities in tertiary education:
– Share about pedagogies and identification of skills
– Pedagogical intervention like in primary and secondary?
Ongoing OECD project on assessing
progression in creativity and critical thinking
Participants in 14 countries (June 2016)
• Brazil
• China
• Finland
• France (3 teams)
• India
• Italy
• Hungary
• Netherlands
• Russia
• Slovakia
• Spain
• Thailand
• UK (Wales)
• USA (4 teams)
Initial version of the rubric (not to be shared)
CREATIVITY
(Coming up with ideas and solutions)
CRITICAL THINKING
(Questionning and evaluating ideas and
solutions)
Progression
INQUIRE
Feel, empathise, observe, describe relevant experience and information
Explore, seek and generate ideas
Understand context/frame and boundaries of the problem
Review alternative theories and opinions and compare/find perspectives on the problem
Level 1 Level 2 Level 3 Level 4
IMAGINE
Make connections, integrate other disciplinary perspectives
Stretch and play with unusual/risky/radical ideas
Identify strengths and weaknesses of evidence, arguments, claims and beliefs
Challenge assumptions, check accuracy, analyse gaps in knowledge
Level 1 Level 2 Level 3 Level 4
DO / SHARE
Envision / Express / Produce / Prototype new product / solution / performance
Appreciate the novelty of solution and/or possible consequences
Appraise / Base / Justify opinion/products on logical, ethical or aesthetic criteria/reasoning
Acknowledge own bias (as perceived by others) and uncertainty/limits of endorsed opinion/solution
Level 1 Level 2 Level 3 Level 4
Creativity
Inquire
Imagine
Do / Share
Feel, empathise, observe, describe relevant experience and information
Explore, seek and generate ideas
Make connections, integrate other disciplinary perspectives
Stretch and play with unusual/risky/radical ideas
Envision / Express / Produce / Prototype new product / solution / performance
Appreciate the novelty of solution and/or possible consequences
Critical thinking
Inquire
Imagine
Do / Share
Understand context/frame and boundaries of the problem
Review alternative theories and opinions and compare/find perspectives on the problem
Identify strengths and weaknesses of evidence, arguments, claims and beliefs
Challenge assumptions, check accuracy, analyse gaps in knowledge
Appraise / Base / Justify opinion/products on logical, ethical or aesthetic criteria/reasoning
Acknowledge own bias (as perceived by others) and uncertainty/limits of endorsed opinion/solution
A rubric to help develop lesson plans
• Mapping of the different steps of a lesson plan on the rubric
• Not all dimensions need to be covered
• Generally task are mainly about creativity or critical thinking (but can cover some habits of both)
• Develop scoring rubrics
• Develop student self-assessment rubrics
• Develop domain-specific rubrics
• To be further simplified
A rubric to…
The NCU rubric on creative thinking
• A, B: Defining problem, use of evidence
• C: Identifies/challenges assumptions
• D: Logical and inventive combination of ideas
• E: Contradictory perspectives
• F, I: Multiple solutions to problem
• G, J: Originality
• H: Changes when necessary
• K: Concludes by weighing evidence
• L, M: Shares coherently
Creativity / match with NCU rubric
Inquire
Imagine
Do / Share
Feel, empathise, observe, describe relevant experience and information - A
Explore, seek and generate ideas – F, G
Make connections, integrate other disciplinary perspectives - D
Stretch and play with unusual/risky/radical ideas - G
Envision / Express / Produce / Prototype new product / solution / performance
Appreciate the novelty of solution and/or possible consequences – K, L?
Critical thinking/ match with NCU rubric
Inquire
Imagine
Do / Share
Understand context/frame and boundaries of the problem - A
Review alternative theories and opinions and compare/find perspectives on the problem – E, F
Identify strengths and weaknesses of evidence, arguments, claims and beliefs
Challenge assumptions, check accuracy, analyse gaps in knowledge - C
Appraise / Base / Justify opinion/products on logical, ethical or aesthetic criteria/reasoning – B, M
Acknowledge own bias (as perceived by others) and uncertainty/limits of endorsed opinion/solution
• What is missing?
• What would you omit?
• How would you simplify?
Your feedback on the current version of
the OECD rubric
a controlled pedagogical intervention in higher education?
A pedagogical toolkit
A. The toolkit A1.
Rubric for
assessment
Dimensions
Levels of progression
A2.
Pedagogical activities
Specific for each domain
Designed to test
dimensions and levels of progression
A3.
Set of exercises
Specific for each domain
Designed to prepare for
the assessment
A4.
Portfolio of student work
For each domain
As examples of different skill levels
Contextual data collection
B1.
Subject-based assessment
Standardised assessment of
academic achievement
(maths and science; visual
arts and music)
B2.
Creativity assessment
Standardised test for
creativity
(EPoC)
(domain-specific)
B3.
Survey questionnaires
School principals
Teachers
Students
B4.
Interviews / focus groups
Teachers
Students
School-based intervention
School year
A A A A
Primary school group - Ages 8-9. STEM, Arts or interdisciplinary domain
Secondary school group - Ages 13-14. STEM, Arts or interdisciplinary domain
A
What factors influence the outcomes?
• Pre-tests and questionnaires at the beginning of the intervention: – Are differences related to baseline in achievement, creativity, to student
beliefs, to pedagogies, to socio-economic bacground, etc.?
• Observations and discussions within the network
B
B
School-based intervention
School year
A A A A
Primary school group - Ages 8-9. STEM, Arts or interdisciplinary domain
Secondary school group - Ages 13-14. STEM, Arts or interdisciplinary domain
A
What effects for the intervention?
• Measures after the intervention: – Post-tests and -questionnaires
– Qualitative observations of pedagogies
– Interviews, ESM, executive functions
• Matched control group (with some kind of intervention as well)
Control group
Control group
B
B
B
B
• What are the differences across country school networks in terms of beliefs/attitudes and pedagogies (and baselines)?
• Is there change in student and teacher beliefs/attitudes and practices?
• What effect on standardised achievement tests?
• What effect on standardised creativity tests?
• Any difference by socio-economic background, by achievement level, by type of pedagogy, etc.?
• What are the perceived hurdles to implementing these pedagogical approaches?
Questions we would like to answer
What domains (ideally)?
Domains
• Engineering
• Teacher training/education
• Humanities / Social science
• Innovation projects
• Arts
• Science
• Professional / technical education
Levels
• 20 year olds
• First year students
• Last year of bachelor
Institutions
• Top tier
• Average
• Short cycles
• Questionnaires (beliefs/attitudes, pedagogy, anchoring vignettes) – Students
– Teachers
– Deans
• Domain-specific achievement tests (with embedded questions on interest, curiosity, attitudes, and domain-specific pedagogies)
• Domain-specific creativity tests
• Domain-specific critical thinking tests
• (Good domain general instruments?)
What instruments?
• Under what conditions could HEI be interested?
– Should we work with a few institutions?
– Could we have a large network of institutions?
– Could we build on existing initiatives?
• What funding model?
– Would HEIs cover their own costs? Who else could fund
their costs?
– Who could fund the international costs (OECD development and coordination work)?
Size of pilot and funding model
other suggested ways to support the agenda
www.oecd.org/edu/innovation
www.oecd.org/edu/universityfutures
www.oecd.org/edu/internationalisation
THANK YOU