measuring education - 1 what to measure? international comparisons
TRANSCRIPT
Measuring education - 1
What to measure?
International comparisons
1 – Inputs, outputs, outcomes
The National Accounts framework
National accounting framework
One aim of National Accounts = value production in the economy as a whole
Production = transformation of inputs (L, K) into outputs (goods & services: G & S)
Measuring production begins with identifying all the units which produce the G & S in question.
Next, consider which of the transactions of these producers need to be identified in measuring production in the National Accounts
Then, data on these transactions need to be obtained and aggregated
Education as a non market service
In the National Accounts, producers are allocated to industries according to a classification system of industries twinned with a classification system of products.
International Standard Industrial Classification (ISIC),
linked with the Central Product Classification (CPC)
Education is Division 85
Education as a non market service
Education (Division 85)– 851 Pre-primary and primary education– 852 Secondary education
• 8521 General secondary• 8522 Technical and vocational secondary
– 853 Higher secondary– 854 Other education– 855 Educational support activities
Source: International Standard Industrial Classification, revision 4
Education as a non market service
Education and health services: government provisions them free of charge or at
not economically significant prices= non-market output.
Price is “not economically significant” if it covers less than half of the costs of the service
Price paid cannot be the basis for valuing it Valuation for NA = sum of the costs incurred in its
production
Education as a non market service
Costs = sum of:– Intermediate consumption (the goods and services
used up in producing the service)– Compensation of employees (costs of teachers,
doctors, nurses, etc.)– Consumption of fixed capital (depreciation of
school and buildings, of medical equipment)– Other taxes, less subsidies, on production
Education as a non market service
How to measure the output of education & health?
(1) Use costs in place of prices
(2) Impute some value for the relative valuation by the consumer to generate aggregation weights.
Solution (1) Assume that, on average, the relative valuation by the consumer equals the relative cost
incurred by producers?
Government producers: Argument = in a democracy, via the electoral
process, consumers have influence over the production decision of governments so that the provision of non-market services is socially optimal.
relative costs = relative utility / willingness to pay
Solution (2) Impute some value for the valuation by the consumer ?
(2) Impute some value for the relative valuation by the consumer to generate aggregation weights.
implies a fully developed consumer or welfare perspective in the estimation of the volumes of health and education services.
= controversial – from a NA perspective if total value of health or
education services total expenditure– if the construction of weights entails delicate
questions such as the necessity to put a monetary value on human life
Back to the pb of aggregation
How to aggregate the production of 300 cars and 10,000 salads?
Answer for market products = market price Idea that it reflects the relative utility of G & S
Back to the pb of aggregation
But how do you aggregate 300 heart surgery operations and 10,000 eye treatments with no market price?
To describe the volume or price of a product category, you have to assume all goods in that category fulfill the same need
+ the price deflator implies an evaluation of the increases in quality
Output in value / in volume
GDP at current prices is not so interesting as growth of GDP and its components in volume
Growth in volume excludes changes in price levels between two periods
Only elements of the National Accounts that can be disaggregated in terms of prices and volumes are useful in analyzing economic growth, productivity and inflation
Previous methods and problems
Previous methodologies : measuring the volumes or the prices of inputs to obtain a measure of the volume or price of outputs
Pb: Such input-based methods cannot capture any productivity change
Productivity growth exists when more or better output can be produced with the same resources.
Quality of output does change in education services ignoring productivity changes = foregoing important information and measuring volume growth inaccurately
Reminder: volume and price indexes
Remember that in principle, deflating sales to consumers by a consumer price index = calculating directly a volume index by weighting together quantity indices which represent the volume of each homogeneous service
The amount spent on each service – a function of its price - provide the weights to aggregate these quantity indices
No difference in concept to construct a price index and then deflates sales or to construct a volume index directly
Reminder: volume and price indexes
With no quality change, the price index for education services would be a weighted average of the price indices of teaching service activities
This price index can then be used to obtain a volume measure of output by deflation
The issue of quality adjustments
Require the identification of characteristics such as speed, engine size for a car or the processor speed for a computer
Idea = isolate price changes due to changes in characteristics from those price changes that constitute ‘inflation’
The issue of quality adjustments
One possibility to measure quality = to put together several characteristics into a single indicator that reflects the contribution of the product to outcome.
Ex: price indices for health care. Triplett (1998) suggests quality-adjusted life years (QALYs) as a single dimensional measure could be used for quality-adjustment of different treatments within a product group
The issue of quality adjustments
Should not be affected by any other factors that influence consumer outcome (e.g., socioeconomic background of students or lifestyle of patients) and
Should reflect the normal / average /expected effect of the activity on “state of knowledge” of the consumer.
‘Normal’, ‘average’ or ‘expected’ because to the extent possible, measures of service production should not be influenced by the individual capacity of the consumer to make use of these services
The issue of quality adjustments
Discussion on the necessity to invoke utility and outcome
Characteristics that matter for consumers have to be identified for quality adjustment
“glance over the national accounts production boundary” to deal with quality change in education services
The quality of services cannot be identified without considering indicators falling outside the production boundary
OECD conference in June 2007: current issue!
In-, output, outcome, utility
General links between output and outcome
Hill services and result of the production processPeter Hill (1975) says the production of services is inseparable from their
use or consumption: two economic units are simultaneously involved in service production whereas production of a good takes place within a single producer unit.
Output of services should be viewed as the attributable contribution of the producer to the outcome.
UnitProduct Producer Consumer
Good Output
Service Outcome
Definitions: education output Eurostat handbook on prices and volume in NA“the quantity of teaching received by the students, adjusted to allow for the
qualities of the services provided, for each type of education”
“In the area of education, the output can be defined as the quantity of teaching (that is, the transfer of knowledge, successfully or not) for a pupil, whereas the related outcomes are the skill and knowledge that a pupil achieves.”
ISCED definition“organised and sustained communication designed to bring about
learning”, where
“Communication” involves the transfer of information (messages, ideas, knowledge, strategies, etc.) between two or more persons; […]
“Learning” is taken as any change in behaviour, information, knowledgeknowledge, understanding, attitudes, skills, or capabilities which can be retained and cannot be ascribed to physical growth or to the development of inherited behaviour patterns
Definitions: education output (2)
OECD (2007) proposal “expected transfer of knowledge and skills provided by an
education unit, for each type of level of education / curriculum”
Transfer of knowledge and skills can be seen as the true quantity indicator, and change in academic scores due to school could provide a right measurement.
Expected = « average », means that each pupil receiving the same teaching is supposed to consume the same output, although some pupils will assimilate more than others (ES handbook: « successfully or not »). This difference in outcome is supposed due to the pupil/consumer.
Basic formula
Ivolume
= Iquantity (number of enrolled pupils)
x Iquality ed (average transfer of knowledge)
x Iquality non ed (total expenditures / core educational) Without information on the transfer of knowledge,
we find the number of pupils, 2nd best quantity indicator for Eurostat.
If we assume that the transfer is continuous and proportional to time of teaching, we get the “pupil x hours”, 1st best quantity indicator for Eurostat.
Overview of inputs, output, outcomes in volume
Inputs Activities / Processes
Output
Direct outcome
Indirect
outcomes
Number of teachers and
staff, by qualification
Pupil-hours, class size, pedagogy
Employment, future real
earnings from labour market
growth of GDP
Number of teachers
Pupil-hours, class size, pedagogy
Transfer of skills and
knowledge by levels of education
Knowledge and skills:
estimated by scores, degrees
Well-rounded
citizens ?
other envir.
factors
Pupils: inherited skills, family backgr.,
entry educ. status, gender?
Methods Input
Output w/o expl. outcome-
based quality adjustment
Output with expl. outcome-based
quality adjustment
Educ. outcome: assessment
and achievement
Human capital, others
Aggregates National accounts Welfare / policy
productivity
Cost-effectiveness
Education output = average transfer of knowledge and skills, some remarks
No accounting of the distribution of individual levels of knowledge and skills, although “reduction of inequalities” is usually one of the targets of the education policy: consistent with the idea of “individual services” in NA
No accounting of healthy, wealthy, well-rounded people / citizens, which are sometimes difficult to measure and which are more “indirect” outcomes.
Education status (outcome) is measured at the end of a curriculum, rarely at the beginning we need a model (role of the pupil / consumer) in order to estimate the part of the output in the outcome.
Other outcome-based quality indicators
OECD: academic scores as first best direct education outcomes
But some curricula could be better estimated by obtainment of a degree or professional insertion (indirect outcome).
Professional insertion: employment rate (if the right qualification if possible) and real earnings can be used (multiplied)
Coherent with human capital theory (discounted sum of future real earnings as return on “GFCF” in education),
Pb of evaluation: what is obtained after the curriculum minus what would have been obtained without, if we want to measure the “output”.
2 – International comparisons: historical landmarks
A little bit of history
Until after WW2, educational systems were evaluated without any indicators
Inspectors judged what went on in classes according to what they had learnt to be “good teaching”
1) The 1970s OECD, 1973: “a system of indicators aiming at
orienting policy”. Goal: increase social well-being. Social scientists
A set of 47 indicators…
A little bit of history
… that were never calculated because there was no data to do so and governments were not willing to spend money to collect it
Failure
2) The 1980s
1983: A Nation at Risk (the US). Strong criticism
1986: Education Indicators. A guide for policymakers
= a period when the focus is on the quality of teaching (pedagogy). Heated debates on indicators
A little bit of history
3) The 1990s
1991: OECD conference Making Education Counts: Developing and using international indicators.
At first, nothing on students’ achievement
Only the educational system: costs, enrolments
= a management approach: reducing costs, optimizing resources
First volumes of Education at a Glance: 1992
A little bit of history
4) The 2000s
Strong focus on students attainment. “educ at a glance”:- A: output- B: resources- C: access- D: learning environment, organization of schools
Chapter on output was last until 2002: moved to 1st position. From 5 indicators of students attainments to 14
= a dramatic change in perspective
Psychometric approach, cognitive sciences have become dominant
1995: International Adult Literacy Survey / 2000: PISA
A little bit of history
The Lisbon criteria (2000): “knowledge economy” Reduce to 10 % the proportion of early school leavers
(indicator 23). Reduce by at least 20% the proportion of students with bad
reading results (indicator 5). At least 85 % of youth should complete secondary education
(indicator 4). Increqe by at least 15 % the number of graduates from
scientific fields, while simultaneously reduging the gender gap (indicator 12).
Participation rate to continuing education: 12,5% of the adult population (indicator 19).
3 – ISCED
The International Standard Classification of EDucation
The ISCED classification
1) A little bit of history (again) UNESCO, 1975-1978 ISCED 1978 From the start, goal = int. comparisons
1990: need for change More adult education, more distance learning, more
various education providers On-the-job training (Germany) not taken into
account
ISCED 1997
The ISCED classification
2) What makes building an international classification so difficult?
Definitions must be precise but also work in very different contexts
– Ex: “completed” education when there is no exam/ diploma?
– Ex: what is “basic education”?
– Ex: “Vocational” training: theoretical or actual ?
Risk = countries including very different things in a given category meaningless comparisons
The ISCED classification
2) What makes building an international classification so difficult?
The right number of items
– 30 is not usable, 4 is too little…
– Usually: 2 to 6 different levels, hierarchical
Being understandable across the world: translation problems
– Ex ISCED: 6 official languages + 100 countries using it as a tool in a foreign language
The ISCED classification
2) What makes building an international classification so difficult?
Accepting differences within a category
– 2 exact same programs do not exist = need to define what is the more important criteria.
– ISCED: level rather than length or organisation of education
– Ex: primary education may last 4 to 8 years
… but clearly defining boundaries
– Implies spending much time discussing borderline, infrequent cases
ISCED-97 levels of education
Level 0 Pre-primary education
Level 1 Primary educationor first stage of basic education
Level 2 Lower secondaryor second stage of basic education
Level 3 (Upper) secondary education
Level 4 Post-secondary non-tertiary education
Level 5 First stage of tertiary education
Level 6 Second stage of tertiary education
ISIC rev. 4 and ISCED 97
ISIC rev 4 classes ISCED-97 levels of education
8510 Pre-primary and primary education
Levels 0 and 1
8521 General secondary education Levels 2 and 3 oriented general
8522 Technical and vocational secondary education
Levels 2 and 3 oriented vocational and technical
8530 Higher education Levels 4, 5 and 6
8541 Sports and recr. education
Not classified in ISCED-97 levels of education
8542 Cultural education
8549 Other education n.e.c.
8550 Educational support activities Not explicitly mentioned in ISCED-97 levels of education
Fo
rma
l edu
catio
n (o
ur fo
cus)
info
rma
l
Su
ppo
rt
act
iviti
es
But no R&D or adm., like in COFOG
The ISCED classification
3) The basics of the ISCED classification
0 to 6: relies on belief in a hierarchy of skills: from basics to more elaborated
Cannot simply be based on number of years? No: number of hours, curriculum, goals of one year very different from one country to another
– Ex: more time, same goals for students with disabilities : same level in more years
– Ex: selective tracks that go faster: less years, more skills
The ISCED classification
3) The basics of the ISCED classification
0 to 6: relies on belief in a hierarchy of skills: from basics to more elaborated
OK but hierarchy must be made explicit
– Ex: theoretical knowledge higher than practical skills?
Definition of “education”: the object of the classification
The ISCED classification : defining education
“ISCED does not intend to provide a comprehensive definition of education, still less to impose an internationally standardized concept of the philosophy, aims or content of education, or to reflect its cultural aspects. Indeed, for any given country the interplay of cultural traditions, local customs, socio-economic conditions, at the very least, will have resulted in a concept of education in many ways unique to that country, and any attempt to impose a common definition would not be productive.
However, for the purposes of ISCED, it is necessary to prescribe the scope and coverage of the educational activities to be covered by the classification.”
The ISCED classification: defining education
“Within the framework of ISCED, the term education is thus taken to comprise all deliberate and systematic activities designed to meet learning needs. This includes what in some countries is referred to as cultural activities or training. Whatever the name given to it, education is understood to involve organized and sustained communication designed to bring about learning. The key words in this formulation are to be understood as follows:”
- organized
- sustained
- communication
- learning
The ISCED classification
4) Ex of problems: level 4 “Post-secondary non-tertiary education”
Was created to put short programs that were classified as secondary or as tertiary depending on countries made comparisons invalid
A category that was created to get rid of those programs, making comparisons between the rest relevant
But not understood, not used as was expected
5) Ex of debate: cut-off btw 5 & 6?
Has to be well defined because countries will always want to put more in the highest level
6 does not include research Masters, only PhD
4 – What outcome at what level?
What are the expected outcomes of
Preschool ?
– Not easily measured. Ex: evaluation of Perry School Project, Abecedarian Project, Head Start…
– Range from grade repetition before 15, to wage at 25, to number of arrests before 40…
What are the expected outcomes of
Primary education ?– “basic knowledge”: literacy and numeracy. But when do
you measure them (at what age?) and how ( next class)
Middle and high school ?– Goal of international comparisons not possible to rely
on “what students know” (ex: history, geography, literature… national, hence not comparable)
– Define a number of “skills” + measure them without cultural bias? ( next class)
These = goals of PISA
What are the expected outcomes of
Universities ?
5 – International rankings of higher education institutions (“universities”)
ARWU
Since when does it exist?
- answer: 2003
Why such a success?
- Context: focus on education and research as investment
- Internet: more widespread information
Increased international competition
Who?
Which institutions should be included?
- ARWU: all institutions with Nobel Prizes, Fields Medals or frequently cited researchers
- Times: all institutions quoted by the 5010 experts
- Financial times: Masters programs
- Leiden: universities only
What should count?
How do you calculate a score?
- ARWU: awards, citations (researchers, articles, top 20 reviews)
- Times: experts quote 30 establishments they consider as excellent in their field + survey of recruiters + citations + staff/students ratio
- Financial times: Experts + indicators on international openness
- Leiden: bibliometrics
What do these rankings do?
Bibliometrics important size effect: More researchers, more papers
Countries where research done outside of teaching establishments (“pure” research) : not counted disadvantaged
The difficulty lies in the many aspects of higher education– Teaching (basics, research or business oriented?)
– Research (applied, fundamental?)
– Implementation (patents, consulting…)
Put all on the same scale implies weighting
What do these rankings do?
ARWU stresses research only, and excellence in research (Nobel Prizes). Leaves students, their careers, apart
Times: research = 20%
CC: Depends on who the ranking is aimed at:
- Students? Can they be Misleading?
- University managers?
- Policy makers?
Consequences
These rankings have massive institutional, practical consequences
– Ex of France: quest for “Critical size” (Paristech, PSE…)
– Ex on research practice: pressure to publish under this or that affiliation
Pb: indicators, rankings are quickly translated into straightforward evaluation: “x is better than y (in general)”. Nevermind sizes, means, goals…
Trade-off between simplicity and completeness
6 – A few important figures
A few important figures
OECD avg
Share of GDP spent on initial education: OECD
A few important figures
The share of educ/GDP is the result of 3 factors:
- Size of the school-age population
- Percentage going to school
- Spending per pupil
A few important figures
Yearly spending per pupil, primary to higher education, ppp$, 2005 (OECD)
A few important figures
Yearly spending per pupil, primary education, ppp$, 2005 (OECD)
A few important figures
Yearly spending per pupil, higher education, ppp$, 2005 (OECD)
A few important figures
Spending per pupil in sec and higher, relative to spending on primary education (=100)
2005 (OECD)
A few important figuresNumber of science graduates for 100 000 25-34 year-
olds in the labor force
« science » here includes biology, physics, maths, statistics, computer sciences, engineering and technical professions, making and transformation, building and architecture
Men Wmen Total
A few important figures
Spending on higher education institutions, GDP% (2005)
Grey = private / red = government
A few important figures
Proportion of 15-yr-olds having repeated a grade at least once (2003)
Lessons to be learnt
Output / outcome
Indicators on processes / on outcomes: what is “evaluation”?
Indicators sometimes shape the debate… but sometimes don’t (ex: debate on education in Fr since 1990)
Rankings of universities on a global scale: recent. Competition within/ between countries.
Importance of the design of classifications (ex: ESEC for occupations)