1 paris 06/06/2007 2 nd oecd workshop on measuring education and health volume output alain gallais,...
TRANSCRIPT
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
1
Draft chapter 2 on education output
Background document: « chapter 2: education services »
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
2
General links between output and outcome
Hill services and result of the production processPeter Hill (1975) says the production of services is inseparable from their
use or consumption: two economic units are simultaneously involved in service production whereas production of a good takes place within a single producer unit.
Output of services should be viewed as the attributable contribution of the producer to the outcome.
UnitProduct Producer Consumer
Good Output
Service Outcome
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
3
Definitions: education output Eurostat handbook on prices and volume in NA“the quantity of teaching received by the students, adjusted to allow for the
qualities of the services provided, for each type of education”
“In the area of education, the output can be defined as the quantity of teaching (that is, the transfer of knowledge, successfully or not) for a pupil, whereas the related outcomes are the skill and knowledge that a pupil achieves.”
ISCED definition“organised and sustained communication designed to bring about
learning”, where
“Communication” involves the transfer of information (messages, ideas, knowledge, strategies, etc.) between two or more persons; […]
“Learning” is taken as any change in behaviour, information, knowledgeknowledge, understanding, attitudes, skills, or capabilities which can be retained and cannot be ascribed to physical growth or to the development of inherited behaviour patterns
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
4
Definitions: education output (2)
This handbook proposal “expected transfer of knowledge and skills provided by an
education unit, for each type of level of education / curriculum” Transfer of knowledge and skills can be seen as the true quantity
indicator, and change in academic scores due to school could provide a right measurement.
Expected, we could say « average », means that each pupil receiving the same teaching is supposed to consume the same output, although some pupils will assimilate more than others (what the Eurostat handbook called « successfully or not »). This difference in outcome is supposed due to the pupil/consumer.
The ISCED-97 levels of education and secondarily fields of education define the natural strata.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
5
ISCED-97 levels of education
Level 0 Pre-primary education
Level 1 Primary educationor first stage of basic education
Level 2 Lower secondaryor second stage of basic education
Level 3 (Upper) secondary education
Level 4 Post-secondary non-tertiary education
Level 5 First stage of tertiary education
Level 6 Second stage of tertiary education
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
6
ISIC rev. 4 and ISCED 97
ISIC rev 4 classes ISCED-97 levels of education
8510 Pre-primary and primary education
Levels 0 and 1
8521 General secondary education Levels 2 and 3 oriented general
8522 Technical and vocational secondary education
Levels 2 and 3 oriented vocational and technical
8530 Higher education Levels 4, 5 and 6
8541 Sports and recr. education
Not classified in ISCED-97 levels of education
8542 Cultural education
8549 Other education n.e.c.
8550 Educational support activities Not explicitly mentioned in ISCED-97 levels of education
Fo
rma
l edu
catio
n (o
ur fo
cus)
info
rma
l
Su
ppo
rt
act
iviti
es
But no R&D or adm., like in COFOG or UOE
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
7
Basic formula
Ivolume
= Iquantity (number of enrolled pupils)
x Iquality ed (average transfer of knowledge)
x Iquality non ed (total expenditures / core educational) Without information on the transfer of knowledge, we find the
number of pupils, 2nd best quantity indicator for Eurostat. If we assume that the transfer is continuous and proportional
to time of teaching, we get the “pupil x hours”, 1st best quantity indicator for Eurostat.
Non educational quality is theoretically better estimated by an additive formula if detail by secondary product is known.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
8
Consistency between quantityand quality indicators
If pupil-hours are the quantity indicator and change in scores the quality indicator, it leads to a double accounting:
Imagine a pupil with an attendance of 90% instead of 100%. As he has received 10% less teaching, it is probable that he will perform
only 90% of the change in score of his “normal” schoolmates. If we cumulate the quantity effect (pupil hours) and the quality indicator
(change in scores), we obtain 81% instead of 90%, the expected transfer of knowledge for this pupil.
transfer of knowledge= number of pupils getting 1 year older * average transfer of 1 year= number of pupils moving up 1 year * average transfer of 1 grade
If the quality indicator relies on scores at a given age, the quantity indicator should be the simple number of enrolled pupils. If it relies on scores at a given grade, the quantity indicator should be the number of pupils moving up.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
9
Overview of inputs, output, outcomes in volume
Inputs Activities / Processes
Output
Direct outcome
Indirect
outcomes
Number of teachers and
staff, by qualification
Pupil-hours, class size, pedagogy
Employment, future real
earnings from labour market
growth of GDP
Number of teachers
Pupil-hours, class size, pedagogy
Transfer of skills and
knowledge by levels of education
Knowledge and skills:
estimated by scores, degrees
Well-rounded
citizens ?
other envir.
factors
Pupils: inherited skills, family backgr.,
entry educ. status, gender?
Methods Input
Output w/o expl. outcome-
based quality adjustment
Output with expl. outcome-based
quality adjustment
Educ. outcome: assessment
and achievement
Human capital, others
Aggregates National accounts Welfare / policy
productivity
Cost-effectiveness
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
10
Education output = average transfer of knowledge and skills, some remarks
No accounting of the distribution of individual levels of knowledge and skills, although “reduction of inequalities” is usually one of the targets of the education policy: consistent with the idea of “individual services” and common rules in NA for FC, income, etc.
No accounting of healthy, wealthy, well-rounded people / citizens, which are sometimes difficult to measure and which are more “indirect” outcomes.
We measure usually education status (outcome) at the end of a curriculum, rarely at the beginning, we need therefore a model (part of the pupil / consumer, time-lag) in order to estimate the part of the output in the outcome.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
11
The model: education output and outcome
chart 1: Education status with age
0
100
200
300
400
500
600
700
0 3 6 9 12 15 18age
leve
l of k
now
ledg
e an
d sk
ills
inherited skills family contribution education unit better school
valu
e ad
ded
by th
e ed
ucat
ion
unit
exit
- en
try
in th
e ed
ucat
ion
syst
empr
imar
y +
seco
ndar
y ed
ucat
ions
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
12
Size of the SD: does it matter for NA?
The SD of the distribution of education levels can be standardized to 10% or 20% of the average
PISA, PIRLS, TIMSS standardized to 20%, US NAEP to 10%. The human capital theory (relying on future real earnings) suggests
8-10% (2 studies) or 20% (1 study).
But it does not matter in a NA view The convention for the relative size of 1 SD has an impact both on
cross-section (better school vs average school) and longitudinal distances (between ages 6 and 15 for instance, which will provide a measurement for average schools).
The quality adjustment in equivalent number of years of teaching is a ratio between cross-section and longitudinal distances.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
13
Relative scores across agesAcademic scale across ages with 1 SD = 10%
0
0.002
0.004
0.006
0.008
0.01
0.012
age 15 age 12 age 9 age 6
1 SD at 15
1 SDat 12
1 SDat 9
Academic scale across ages w ith 1 SD = 20%
0
0.001
0.002
0.003
0.004
0.005
0.006
0.007
0.008
0.009
age 15 age 12 age 9 age 6
1 SD at 15
1 SD at 12
1 SD at 9
The larger the SD at age 15, the larger the output between ages 6 and 15, and constant proportions between numerator and denominator of QA.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
14
=> For PPP, let us scale the average outcomeat age 15 = 500 and 1 SD = 20% (PISA)
PISA measures a national average outcome at age 15. To reflect the output of the national school systems, it should be neutralized of the family contribution (ESCS correction) and subtracted of inherited skills. The formula should be: ESCS corrected PISA – X.
Box 2.1: The PISA Economic, Social and Cultural Status (ESCS)
The PISA index of economic, social and cultural status (ESCS) is designed to capture broad aspects of a student's family and home background. It is derived from sub-indices based on:
- i) the highest occupational status of the student's parents (ISEI: the International Socio-Economic Index of Occupational Status);
- ii) the highest level of education of the parents converted in years of schooling;
- iii) an index based on home possessions (in fact three sub-indices: the PISA index of family wealth; the PISA index of home educational resources; and the PISA index of possessions related to “classical” culture in the family home).
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
15
Estimation of what is to be subtracted to ESCS corrected PISA scores: at least 180
Ideally, we should observe inherited skills at age 15, or more exactly inherited skills + average family contribution (for ESCS only neutralizes the differences in family contribution) at age 15. Requires a sophisticated model.
But we can assume that education status at age 6 reflects only inherited skills + average family contribution. Can be extrapolated with the help of 2 national longitudinal tests: the US NAEP and the UK 4 fey stages, which suggest about 180 at age 6 in average (with a convention of 1 SD = 20%).
Inherited skills + average family contribution can increase between ages 6 and 15, so that we could subtract more.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
16
Age
US NAEP(1 SD = 10%)
UK 4 key stages(1 SD ~ 20%)
re-scaled US(age 15 =
500,1 SD = 20%)
re-scaled UK(age 15 =
500,1 SD = 20%)
re-scaled US (age 15 =
500,1 SD = 10%)
re-scaled UK(age 15 =
500,1 SD = 10%)
6 (203) (12.52) 182 180 352 340
7 15.51 217 223 368 362
8 253 267 385 383
9 231 288 310 401 405
10 323 353 418 426
11 27.49 359 396 434 448
12 394 426 451 463
13 429 456 467 478
14 33.79 465 487 484 493
15 (288) (34.71) 500 500 500 500
16 35.63 535 513 516 507
17 307 571 527 533 513
impact of 1 SD on output (age 15 - 6) 34% 31% 31% 31% 34% 31%
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
17
Time-lag management
grade \ year of outcome
2000 2001 2002 2003
9th grade (14-15 year old)
PISA 2000
8th grade (13-14 year old)
2/3 PISA 2000 + 1/3 PISA 2003
7th grade (12-13 year old)
1/3 PISA 2000 + 2/3 PISA 2003
6th grade (11-12 year old)
PISA 2003
Total 2000: 2 PISA 2000 + 2 PISA 2003Total 2001: PISA 2000 + 8/3 PISA 2003 + 1/3 PISA 2006
Lower secondary education
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
18
Time-lag management and revisionsYears Lower secondary
(hypothesis of 4 years)Primary(hypothesis of 5 years)
2000 PISA 2000then ½ PISA 2000 + ½ PISA 2003
PISA 2000 then PISA 2003
then 1/5 PISA 2003 + 4/5 PISA 2006
at last 1/5 PISA 2003 + 3/5 PISA 2006+ 1/5 PISA 2009
2001 PISA 2000
then 1/4 PISA 2000 + 3/4 PISA 2003
at last 1/4 PISA 2000 + 2/3 PISA 2003+ 1/12 PISA 2006
PISA 2000 then PISA 2003
then 1/15 PISA 2003 + 14/15 PISA 2006
at last 1/15 PISA 2003 + 8/15 PISA 2006+ 6/15 PISA 2009
2002 PISA 2000
then 1/12 PISA 2000 + 11/12 PISA 2003
at last 1/12* PISA 2000 + 2/3* PISA 2003 + 1/4* PISA 2006
PISA 2000 then PISA 2003 then PISA 2006
then 6/15 PISA 2006 + 9/15 PISA 2009
at last 6/15 PISA 2006 + 8/15 PISA 2009+ 1/15 PISA 2012
2003 PISA 2003
then ½ PISA 2003 + ½ PISA 2006
PISA 2003 then PISA 2006
then 1/5 PISA 2006 + 4/5 PISA 2009
at last 1/5 PISA 2006 + 3/5 PISA 2009+ 1/5 PISA 2012
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
19
Other outcome-based quality indicators
Although we think that academic scores are the first best direct education outcomes, some curricula could be better estimated (or are more traditionally estimated) by obtainment of a degree or professional insertion (this latter an indirect outcome).
Professional insertion: employment rate (in the right qualification if possible) and real earnings can be used (multiplied) and are often suggested by human capital theory (discounted sum of future real earnings as return on “GFCF” in education), but they are to be considered in an incremental point of view: what is obtained after the curriculum minus what would have been obtained without, if we want to measure the “output”.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
20
Which output/outcome for tertiary education? Degrees are certainly the most natural outcome indicators
for tertiary education, but they are of different “values”. Harmonization of degrees in Europe with the Bologna
process (bachelor, master, doctorate), but what are the relative values of these 3 degrees?
Distinguish by (ISCED-97) field of education; Double degree for one curriculum: double outcome? Curriculum shared between different countries/units; The output should not be null for students leaving without
degree. Time-lag adjustment. Number of credits reflect perhaps more directly the “quantity
of transfer of knowledge and skills”, i.e. our definition of the output, and avoid most problems quoted above. Still need to distinguish by field of education and perhaps by “prestige” of university.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
21
If not “outcome-based” then “outcome-oriented”
If the output is defined as the attributable contribution of the producer to the outcome, and if the outcome cannot be directly measured, then some “inputs-based” or “process-based” quality indicators could be used, but “outcome-oriented”, it means converted in an expectation of incremental contribution to the outcome.
Example of the Italian model of the class size (see item 3.3). If the more classical way of treating different qualities as
separate goods was chosen, it would lead to a “quality presentation” but not to a “quality adjustment”: the price index would be nothing else than a cost weighted combination of cost indices, i.e. the same price index as in the input method.
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
22
Stratum for time-series Quantity Quality (ed) CommentPre-primary education Pupil-hours None Primary education: normal
Pupils
Contribution to scores
The sub-stratification of ISCED levels could be replaced by an estimate of the non-educational part (social services)
Primary education: special
Lower secondary: normal
Lower secondary: special
Upper secondary education: general + pre-technical or pre-vocational
Do not forget to subtract the entry education status
Upper secondary education: vocational
Employment rate after 1 year and/or real earnings if no scores available
In an incremental point of view and net of the economic trend
Post-secondary non-tertiary education
More occup. specific prog. tertiary education (5B)
More theoretically based programmes tertiary education (5A+6)
Credits 1st best
Degrees 2nd best
Pupils 3rd best
by field of ed.
“value” of degrees?
Paris06/06/2007
2nd OECD workshop on “measuring Education and Health volume output” Alain GALLAIS, OECD/STD/NAFS
23
Stratum for cross-country Quantity Ed. quality Non ed. quality
ISCED 0: Pre-primary education
Pupils
(pupil-hours?)
None Should come in a future step from the UOE data collection, based on the “ancillary services”, afterreconciliation with NA.
ISCED 1: Primary education Pupils ESCS corrected average PISA score- 180
ISCED 2: Lower secondary: normal
Pupils
ISCED 3: Upper secondary education
Pupils
ISCED 4: Post-secondary non-tertiary education
Pupils None
ISCED 5B: More practical and occupationally specific programmes tertiary education
Pupils None at present
ISCED 5A and 6: More theoretically based programmes tertiary education
Pupils
(ECTS?)
(degrees?)
None at present (expectation of degrees?)