1

Paper 1 for the EFA Global Monitoring Report, 2004 Jaap Scheerens (01/03/04) THE CONCEPTUAL FRAMEWORK FOR MEASURING QUALITY

In this chapter an input-process-outcome-context framework is introduced as a basis for defining quality and to categorize different measures of quality in education. It will be shown that this framework can be used to clarify a broad range of quality interpretations: productivity, effectiveness, efficiency, responsiveness, equity and a more eclectic use of quality indicators. Broad applicability will also be illustrated by addressing criticism as to the “narrowness”, ‘linearity” and authoritarian nature of the framework. Both with respect to the assumed information requirements of “rational planning” as regarding the control aspects of “multi-level governance” interpretations exist that come a long way in doing justice to the complexity of social reality. In this way some modified interpretations on both dimensions (rational planning and multi-level control) will be described, culminating in models that could be thought of as the complete antipodes of the more common interpretations. In the second part of the chapter the main components of the framework will be used as a basis for categorizing and describing input indicators, process indicators, outcome indicators and context indicators.

An input-process-outcome-context framework for assessing educational quality Perspectives on education quality can be clarified on the basis of a conceptual framework that describes education. The most frequently used way to do so is to depict education as a productive system, in which inputs are transferred into outcomes. Steps in elaborating this basic scheme consist of: a) including a context dimension, that functions as a source of inputs and constraints

but also as a generator of the required outputs that should be produced; b) differentiating outcomes in direct outputs, longer term outcomes and ultimate

societal impact; c) recognising the hierarchical nature of conditions and processes, which comes

down to considering the functioning of public education as just another example of “multilevel governance”.

The model depicted in Figure 1 shows the basic ingredients of this framework.

2

Figure 1: A basic systems model on the functioning of education

In the schematic presentation of the framework in Figure 1 there are various options in choosing the levelat which level the central “black-box” is described. When analysing the impact of policy measures at the national level one might choose the education system in a country as the central black box. In applications where the quality of schools is at the centre of attention one would choose the school as the level where the transformation of inputs to outputs is studied. A perhaps more interesting option, however, is to distinguish several levels in the central black box, for example the national educational system, the school level and the level of the group where the teaching and learning at school takes place, traditionally the classroom level. But other options are possible: the students could explicitly be incorporated as a separate level and one might want to include local community as a level as well. Finally, one or more controlling levels could be placed in the context. In Figure 2 the structure of the quality framework that was presented in the 2002 EFA Monitoring Report is shown. In this model inputs are described at three different levels, the school, the students and the household/local community. Macro-level policies and conditions are placed in the context dimension. The central box, where transformation processes take place, is differentiated into a school and teaching/learning level.

context

outputs inputs Process or throughput

school level

classroom level

Figure 2source E

Figure 3empirica

• curric• textb

mater• teach

trainicomm

• adequ• paren

Stud• aptitu

perse• nutrit• schoo• attend• gende

Hou

• paren• house• comm

labou• cultu

• Macpoli

dececivi

INPUTS

3

: An input-process-outcome framewFA Global Monitoring Report, 2002

is a variation on the basic scheme tl school effectiveness research (Sch

School ulum content

ooks and learning ials er qualifications, ng, morale and itment ate facilities t/community support

ent characteristics de, ability verance/ commitment ion and health l readiness ed ECCE r

sehold/community characteristics tal attitudes hold income unity economic and

r market conditions ral/religious factors

S• high • stron• posit• safe a

envir• incen

result• flexib

Tea• suffic• active• integ

asses• appro• appro

langu

Conro-economic and fiscal • National gcies, political stability, educationntralization and governance, sources o

l service quality teacher re

PROCESS

ork for assessing education qu

hat was used to summarise theeerens, 1990).

chool climate expectations g leadership ive teacher attitudes nd gender-sensitive

onment tives for good s ility/autonomy

ching/learning ient learning time teaching methods

rated systems for sment and feedback priate class size priate use of age

co• l• g• p• p

t• h

• f• d

s

• ooo

textual factors oals and standards for • Educa

, curricular guidelines, • Particif funding and allocation, complcruitment/deployment • Engag

• Peer epromo

OUTCOMES

ality;

findings of

Achievement gnitive development iteracy, generic skills ood citizenship ersonal development ositive attitudes owards learning ealthy behaviour

Attainment ormal completion iplomas/qualification

Standards fficial learning bjectives (desired utcomes)

tion system management pation, progression, etion and transition ement and use of time ffects, parental support, tion policies

4

Inputs. teacher experience. per pupil expenditure. parent support

OutputsStudentachievement,adjusted for:. previous achievement. intelligence. SES

Context. achievement stimulants from higher administrative levels. development of educational consumerism. 'covariables', such as school size, student-body composition, school category, urban/rural

PROCESS

School level. degree of achievement-oriented policy. educational leadership. consensus, cooperative planning of teachers. quality of school curricula in terms of content covered, and formal structure. orderly atmosphere. evaluative potential

Classroom level. time on task (including homework). structured teaching. opportunity to learn. high expectations of pupils' progress. degree of evaluation and monitoring of pupils' progress. reinforcement

Figure 3: An integrated model of school effectiveness, Scheerens, 1990

In this model various levels are thrown together in the context dimension, the central black-box again has two levels, and student input characteristics are incorporated in the outputs, namely as variables for which output indicators are adjusted. This latter interpretation is known as the “value-added perspective” which implies that the effects of malleable conditions of schooling over and above the influence of student background characteristics are focussed at.

5

It is worth underlining that the incorporation of a context-dimension adds considerably to the flexibility and broadness of applicability of the framework. As was already indicated the context can be seen as a generator of inputs, as a level that determines or co-determines the definition of desired outcomes that should be generated, and as a level that judges quality and provides feedback. In more practical terms the context-dimension gives room for situational adaptation to local conditions. A final analytic distinction to qualify the impact of “context” is the one between malleable conditions and “given” environmental constraints, sometimes also indicated as “antecedent” conditions. Malleable conditions are in the hands of actors on the scene, like national policy planners, local constituencies, school managers, and teachers. Antecedent conditions already “exist”. Background characteristics of students, such as cognitive aptitude or socio-economic status of their home-background, are examples of “given” factors. At higher levels, the school or system level, the distinction becomes more arbitrary. For example, school size could be seen as a given condition, but also, perhaps in a longer term perspective, as a variable that is subject to change in national policies about desired scale of educational provisions. Another example is the composition of the student population of a school in terms of, for example, average socio-economic status. This variable is usually treated as a “given” condition, out of reach of policies aimed at the improvement of the primary process of teaching and learning. However, a school might have explicit recruitment, selection and admission policies, in order to control student composition. Having clarified the basic “working” of the input-process-output-context framework it can be concluded that the framework is quite general and flexible in describing the functioning of education. This initial conclusion will be further elaborated and critically analysed by considering the usefulness of the framework as a basis for defining aspects of education quality and by referring to criticism and “competitors” to the framework. The framework as a basis for defining different perspectives on education quality When examining the framework that was introduced in the preceding section, at least six ways in defining education quality can be chosen, by emphasising certain parts, aspects or relationships from the basic framework. a) The productivity view According to this view, the success of the systems is seen as depending on the attainment of the aspired outputs/outcomes. For example in the sense of a satisfactory quantity of school-leavers that have attained a specific level (which may be formalised as a diploma), or in terms of an acceptable level of employment of students with a certain diploma. According to this view output/outcome/impact

6

indicators are predominant or even the only type of quality indicators that need to be monitored. The introduction and application of performance standards is closely related to the productivity view of educational quality. Basically performance standards are norms, cutting scores or “thresholds” defined on a specific output, outcome or impact indicator. Standards can be absolute or comparative. An example of an absolute standard would be the statement that in a particular country at least 60% of the students at the end of lower secondary education reach a particular proficiency level in reading literacy. A example of a comparative or relative standard would be the statement that the countries in the European Union will all be among the 10 % highest scoring countries in the world on a particular international achievement test in 2009. Comparative standards are also described as benchmarks; and a specific country or organisation to which others wish to compare themselves could then be called the (inter)national benchmark. Standards and benchmarks in many contexts are not just used retro-actively, in order to evaluate, but also pro-actively, in the sense of targets. b) The instrumental effectiveness view According to the instrumental effectiveness view there is a clear perspective for the selection of context, input and process indicators, namely their expected effect on outcomes. To the extent that effectiveness or production functions can be completely specified, in other words outcomes can be totally predicted, context, input and process indicators could replace outcome indicators. The value of certain levels and forms of inputs and processes is determined by their instrumental potential. Clearly the instrumental perspective offers a more dynamic handle for policy, as it considers not only given constraints but also factors that are policy malleable. Modelling educational effectiveness and the results of empirical educational effectiveness research studies will be described extensively in chapter 4. The instrumental effectiveness perspective has a particular interest in malleable characteristics at school level. In a subsequent section of this chapter there will therefore be a specific emphasis on process indicators on school functioning. An important technical aspect of the instrumental effectiveness perspective is the notion of establishing the added value of schooling. This means that the effect of malleable input and process conditions on student performance is to be analysed over and above the impact of student background characteristics. The most straightforward way to accomplish this is to measure achievement at two points in time, minimally, so that learning progress can be assessed. c) The adaptation perspective This view “transcends” the instrumental effectiveness perspective by not only looking at the question how to do things right, but first of all considering the question on how to do the right things. In other words the adaptation perspective would lead to a critical analysis of educational goals. Conditions that allow for a continuous sounding

7

of changing contextual conditions for the education province would receive emphasis as means, while labour market outcomes or “social capital” could be considered as ends, according to this view. The adaptation perspective would also cover defining a part of the school curriculum as meant to adapt to the specific environmental context and local and national culture. As such, this perspective could be seen as an important complimentary perspective to the instrumental effectiveness view, which takes more or less universal standards in basic subjects for granted. At the macro-level of national education systems the adaptation perspective could focus at an inventory of institutional structures for continuously checking the complicated interactions between societal demands for education and the existing educational targets and priorities. At school level school-community relationships and parental involvement are examples of areas that should be focused from this perspective. d) The equity perspective When inputs, processes and outcomes are analysed for their equal or “fair” distribution among participants in education with different characteristics, equity is the primary facet of judging educational quality. Because the equity issue has several different interpretations, these will be outlined in a bit more detail than was done for the other quality perspective described so far. First of all equity of achievement outcomes can be studied by considering measures of between student and between school variation. While the primary interest of educational effectiveness studies is to assess the impact of malleable factors over and above student background characteristics achievement results at the individual student level are “adjusted” for the impact of these variables. At the same time there appears to be an additional impact of the average of relevant student background characteristics, such as socio-economic status on achievement. To the degree that these student level background conditions at individual or school level have a larger impact on achievement school systems can be judged as comparatively selective versus equitable (the recent debates about the PISA results for a country like Germany illustrate this issue).1

Secondly, research, particularly surveys, can provide information about the extent to which the levels of inputs and process are the same (or different) in all schools and all provinces within countries. In other words, how equitably are the resources or processes allocated or distributed across schools and provinces within a country? When the teaching force in each school is described, for example, can it be said that all pupils – in whichever school they are – have the same quality of teachers? Or the same provision of resources, and so on? It is important to view the levels of input and

1 The German Pisa results are characterised by relatively low average achievement levels; a large variation between students and between schools, and a relatively strong impact of SES related background characteristics on achievement. The categorical organisation of the German school system is seen as one of the causes of this state of affairs.

8

process provision and equity at the same time. If the achievement levels are all low (and much lower than they should be) but there is very little variation among schools, then we know that the schools all have the same very low level. The patterns of variation in school provisions are relevant information for educational policy. If the levels of school resources are very different among schools within provinces then it could be seen as the job of the provincial authority to do something about this. If there are large differences among provinces but few differences among schools within provinces then the national authority must do something to ensure more equity among provinces. Assuming that there is some measure of the socio-economic status (SES) composition of the student body within schools, then it would also be possible to examine if it is the poorer SES schools that have fewer provisions and the higher SES the higher levels of provision. Similar kinds of questions could be raised with respect to other grouping variables of schools, like: urban/rural and private/public. A third way of thinking of equity is based on the philosophy of compensation and “positive discrimination”. Related, and more neutral sounding, terms in education are: adaptive teaching and differentiation within classrooms. The basic idea is that something extra is done for students that are in some way or another “disadvantaged”. Examples are smaller classes, extra tutoring, ancillary services to schools, like free school meals, adaptive teaching approaches, increased learning time. School surveys can capture such measures or programs in a descriptive way. The distribution of “extras” for disadvantaged learners across schools within countries could also be seen as a specific example of equity in the sense of the first meaning, stated above (equitable distribution of inputs). As a fourth interpretation, a sophisticated combination of “school effectiveness” and “equity in schooling” is addressed in studying what is known as “differential effectiveness”. This branch of school effectiveness research specifically addresses the question which kind of school – and instructional conditions work best for disadvantaged as compared to more “advantaged” students. Fifth and finally, there is the meritocracy-interpretation of equity, based on the aspiration that each student should achieve according to his or her aptitude and irrespective of socio-economic status. Measurement implications for assessing equity in terms of meritocratic potential of educational provisions are the availability of scholastic aptitude measurements or intelligence tests. e) The efficiency perspective This perspective can be seen as a further demand on the productivity and instrumental effectiveness view, by considering the highest possible outcomes at the lowest possible costs. In Table 1 the concept of efficiency of schooling is clarified by comparing it to school effectiveness definitions.

9

Table 1: Distinction between school effectiveness and school efficiency, cited from Cheng, 1993, p. 4

Nature of school output

Nature of school input

In school/Just after schooling Short-term effects Internal (e.g. learning behaviour, skills obtained)

On the society level Long term effects External (e.g. social mobility, earnings, productivity)

Non-monetary (e.g. teachers, teaching methods, books)

School's Technical Effectiveness

School's Societal Effectiveness

Monetary (e.g. cost of books, salary, opportunity costs)

School's Technical Efficiency (internal economic effectiveness)

School's Societal Efficiency (external economic effectiveness)

It is vitally important for the economic analysis of efficiency and effectiveness that the value of inputs and outputs can be expressed in terms of money. For determining efficiency it is necessary that input costs like teaching materials and teachers' salaries are known. When the outputs can also be expressed in financial terms efficiency determination is more like a cost-benefit analysis (Lockheed, 1988, p. 4). It has to be noted, however, that a strict implementation of the above-mentioned economic characterisation of school effectiveness runs up against many problems. These already start with the question of how one should define the “desired output” of a school, even if we concentrate on the short term effects. For instance, the “production” or returns of a secondary school can be measured by the number of pupils who successfully pass their school-leaving diploma. The unit in which production is measured in this way is thus the pupil having passed his or her final examination. Often, however, one will want to establish the units of production in a finer way and will want to look, for instance, at the grades achieved by pupils for various examination subjects. In addition, there are all types of choices to be made with regard to the scope of effectiveness measures. Should only performance in basic skills be studied; is the concern also perhaps with higher cognitive processes and should not social and/or affective returns on education be established? Other problems related to economic analysis of schools is the difficulty in determining monetary value on inputs and processes and the prevailing lack of clarity on how the production process operates (precisely what procedural and technical measures are necessary to achieve maximum output). Relevant to the question on how useful one regards the characterising of effectiveness in economical terms is the acceptability of the school as a metaphor for a production unit.

10

f) The disjointed view Combinations or relations between the various elements of Figure 1 were central in the previous views that represent a particular perspective on education quality. An alternative view is to consider each element “on its own’ and judge whether it is manifested in an acceptable way, or at an acceptable level. In this way one could, for example, consider levels of teacher training, as a (minimum) requirement for being allowed to function as a teacher, class sizes could be judged in terms of being acceptable for being “manageable” units for teachers and students, and teaching strategies could be rated according to norms of good practice. The disjointed view is descriptively the simplest one, although in an evaluative sense it is perhaps the most arbitrary one. When considering, from a practical point of view, the way these perspectives on educational quality are being reflected in the composition and use of actual indicator systems it appears that the last perspective, the “disjointed view” is predominant (Scheerens et al., 2002) From a conceptual point of view the range of perspectives on educational quality that can be “generated” from the basic input-process-outcome-context framework appears to underline the heuristic value of this framework for addressing issues of defining and measuring educational quality. This leaves the following issues for the remainder of this chapter. The “conceptual conclusion” about the heuristic value of the input-process-outcome-context framework will be subjected to further critical analysis, in the next section. And finally, more detailed information will be given about the measurement of input, process, outcome, and context indicators. From rational planning to growth from within; a critical analysis of the input-process-outcome-context framework The input-process-outcome-context framework is a descriptive conceptual framework that was used to categorise a range of interpretations of educational quality. In order to critically analyse its acceptability for this purpose, it needs to be looked at in more dynamic terms, namely in terms of its value for policies aimed at quality enhancement in education. What are the implications of this model for the schematic picture it offers on the dynamics of a part of social reality as complex as education? Does it make true its claims of universal applicability or does it leave “blind spots” that should not be overlooked? Does it lend itself as a set of tools that are useful for different actors on the educational scene, or is it more exclusively useful for administrators and governors? What are the limitations for its application in non-western cultures? Is it sufficiently amenable to the complexities of social realty? Are there any implications for the issue of using quantitative and/or qualitative data for the monitoring of educational quality? It is clear that, within the context of this report,

11

only a limited answer can be given to questions of these kinds that border on important philosophical and epistemological questions. In what follows the emphasis will be on the general question that refers to the adequacy of dynamic applications of the framework to deal with the complexities of social reality. What will be sketched are in fact several interpretations that vary from rather formal and almost “mechanistic” interpretations to more interactive ones. Rational planning as the proto-type of a dynamic application of the input-process-outcome-context framework The key to transforming the descriptive input-process-outcome-context framework into “something” that is more practical and action oriented is to see “outcomes” as “desired outcomes”, goals or targets. When this step is made, it seems logical to see inputs and processes as means that should bring about the desired outcomes. Aspects of the context might be seen as just another category of means, or alternately as conditions that put certain constraints on the functioning of means that are applied. Of course the prefix “desired” also puts us in a world that is normative and evaluative. The technical term rationality is used to describe behaviour that is goal oriented. Since rationality lies at the basis of theories on planning and public policy-making, decision-making theory, micro-economic theory and theories on “organisational learning” it is rightly addressed as the rationality paradigm.

After describing the basic characteristics and “pure” forms of the rationality paradigm modified interpretations will be sketched that progressively relax informational and control assumptions, at the same time allowing for more uncertainty and complexity. At the extremes of both dimensions (the information dimension and the control dimension) mechanisms will be encountered that are the complete antipodes of the pure rationality model, and therefore to some extent its competitors. At the end of this section the balance will be made up with respect to the potential of various models to improve quality in education, and implications for the monitoring of educational quality will be drawn as well. The pure rationality model: synoptic planning The basic principles of the rationality paradigm are: a) behaviour is oriented toward preferred end states (as reflected in goals or

individual well-being); b) in situations where there is a choice between alternative ways to attain the

preferred end states, an optimal choice is made between these alternatives, which means that profit, well-being, or other preferred end states are maximised given the alternatives and constraints;

c) in organisational settings the alignment of individual preferences and organisational goals is a major issue.

12

The rationality paradigm is applied in formal and less formal models of planning, control, design and feedback and is attached to different units: organisations as a whole, sub-groups or departments and individuals. Apart from this, procedural vs. structural interpretations may be distinguished, the first referring to organisational processes and the latter referring to the design (division and interconnection) of units and sub-units. A further important distinction has to do with the question whether goals are considered as "given" to the social planner or designer. or that the process of choosing particular goals is seen as part of the planning process. In the first case the approach is "instrumental", whereas the term "substantial rationality" (Morgan, 1986, p. 37) is sometimes used for the latter. Stated more popularly the instrumental approach is inherent in the phrase "doing things right" whereas the substantial perspective asks the additional question of about "doing the right things". The ideal of "synoptic" planning is to conceptualise a broad spectrum of long-term goals and possible means to attain these goals. Scientific knowledge about instrumental relationships is thought to play an important role in the selection of alternatives. Campbell's (1969) notion of "reforms as experiments" combines a rational planning approach to social (e.g. educational) innovation with the scientific approach of (quasi-) experimentation. The main characteristics of synoptic planning as a prescriptive principal conducive to effective (in the sense of productive) organisational functioning, as applied to education, are: - "proactive" statement of goals, careful deduction of concrete goals, operational

objectives and assessment instruments; - decomposition of subject-matter, creating sequences in a way that intermediate and

ultimate objectives are approached systematically; - alignment of teaching methods (design of didactical situations) to subject-matter

segments; - monitoring of the learning progress of students, preferably by means of objective

tests. Focussing at the primary process of schooling, the synoptic planning approach in education applies most of all to curriculum planning, design of textbooks, instructional design and preparation of (series of) lessons. When the ideal of rational planning is extended to organisational structuring, related principles about "controlled arrangements" are applied to the division of work, the formation of units and the way supervision is given shape. "Mechanistic structure", "scientific management" and "machine bureaucracy" are the organisational-structural pendants of rational planning (cf. Morgan, 1986, ch. 2). The basic ideas go back to Max Weber, who stated the principles of bureaucracy as "a form of organisation that emphasises precision, speed, clarity, regularity, reliability, and efficiency achieved through the creation of a fixed division of tasks, hierarchical supervision, and detailed

13

rules and regulations". Although Mintzberg's conception of the professional bureaucracy, applicable to schools and universities, is often treated as the complete anti-thesis of classical bureaucracy, it should be underlined that the basic notion of standardisation and predictability of work-processes, be it with a considerable band-width of individual leeway, is retained. This “pure” form of rationality is the interpretation that is frequently criticised for being mechanistic, linear and authoritarian. In the educational domain a “narrow” orientation towards cognitive learning objectives is usually added to this image; although the model is in itself “empty” as far as the selection of substantive priorities is concerned. (One might as well follow a synoptic planning approach to stimulate and implement affective educational goals.) Despite these criticisms it is still the example for many approaches to educational innovation and school improvement, as for similar applications in other sectors. The well-known logical framework model is another case in point. A more fundamental kind of critique addresses the information requirements that the pure rationality model poses. In actual practice it is impossible in most situations to be literally “synoptic” in ones goal preferences, listing of all characteristics of the entrance situation, the full range of available alternative means, as well as having full information on their comparative effectiveness and efficiency. Therefore several modified forms on this dimension of information requirements have been developed; and these will be addressed in the next section. Modifications based on relaxing the information requirements As modifications with respect to the information demands inherent in the pure rationality model the following conceptualizations will be briefly sketched: Simon’s idea of “bounded rationality”, Lindblom’s “incrementalism” (also described as “muddling through”) and Cohen, March and Olsen’s “garbage can model of organisational decision-making”. Bounded rationalitySimon’s (1945) criticism with respect to the pure rationality model was aimed at the discrepancy he saw between the information processing requirements that the model pre-supposes and the limited possibilities that decision-makers have in real-life situations. Where the pure rationality model demands full information about the consequences of each and every choice, in actual practice knowledge about consequences is mostly partial. Moreover, it is considered quite unlikely that the preference of all possible future outcomes can be determined fully in advance. Finally, the third requirement, namely that all alternative actions should be considered, is considered unrealistic as well. The core idea of Simon’s bounded rationality is that the search for alternative solutions is stopped as soon as a

14

satisfactory alternative has been found. More realistic decision-making is thus characterised by considering an incomplete series of alternatives and possible outcomes, and also recognising that certain alternatives may still need to be discovered or developed, and by striving for “satisfactory” instead of “optimal” choices. Based on this latter characteristics the term “satisficing” is coined (March & Simon, 1958). Bounded rationality provides a model of planning and decision-making that is less synoptic and “mechanistic” than the pure rationality model, but it still presupposes a clear articulation of goals and a careful search for effective means and solutions. IncrementalismLindblom’s criticism of the pure rationality model is of the same nature as Simon’s, but his alternative conceptualisation departs more radically from the pure form. Incrementalism was explicitly developed as a “counter-model” to the pure rationality model, on each and every aspect. This contrast is shown in the table below, based on Hoogerwerf, 1959. Table 2: Comparison between synoptic rational planning and incrementalism Synoptic rational model Incrementalism (successive limited

comparisions) Clarification of norms and goals is a prerequisite for a systematic analysis of policy alternatives.

The selection of norms and goals and the empirical analysis of alternative actions cannot be separated and happens closely intertwined.

Policy formation is seen as an analysis of means-goals relationships, goals are stated first and then a systematic search for the most effective means must take place.

Because goals and means are hard to separate, means-goal analyses are hardly possible or, when carried out at all, hardly adequate.

The test of good policy is that it can be shown empirically that optimal means were selected for the appropriate goals.

The test of good policy is merely that different policy-analysts can agree on a policy, without agreeing about the most effective means for agreed goals.

The analysis (of goals and means) is considered to be comprehensive, each and every important factor needs to be taken into consideration.

The analysis is severely limited, because important possible effects, important alternatives and important values are neglected.

Analysis depends to a large extent on theory, and scientific knowledge.

Analysis and policy implementation depend on successive limited comparisons, starting from the entrance situation, and this characteristic severely limits the possibility to depend on theory.

15

Incrementalism paints a picture of planning and decision-making that is less clear-cut and somewhat “fuzzy”, oversees a more limited time-span, and is more in line with a context of mutual accommodation and negotiation than a context of central prescription and regulation. The garbage can modelIf incrementalism can be seen as a radical alternative to rational decision-making, Cohen, March and Olsen’s “garbage can model” put matters completely upside down. They speak of an organisation as “a collection of choices looking for problems, issues and feelings looking for decision situations in which they might be aired, and decision-makers looking for work” (Cohen, March & Olsen, 1972, 2). According to these authors decision-making in what they describe as “organised anarchies” is more like rationalising after the fact than pro-active rational planning. As a matter of fact they see educational organisations, particularly universities, as proto-types or organised anarchies. Such organisations, according to them, are characterised by “problematic preferences”, “unclear technology” and “fluid participation”. Important elements that the garbage can model brings to the fore are: - the importance of the ideo-syncratic preferences of individual actors (and in this

there is common ground with micro-economic theory in which “non task related”, “egoistic” preferences of decision-makers and actors in general are taken into consideration);

- a random element in organisational decision-making; - a specific view on the context of organisational decision-making when the

members of the organisation have a lot of leeway and autonomy. Modifications based on relaxing the control implications Strictly speaking the rationality paradigm does not have a control dimension. It is only when decisional processes are “projected” in social reality and organisational structures that an association with organisational procedure and structure is likely to be made. The pure rationality model pre-supposes a large degree of harmony in the organisational configuration, and this assumption might be challenged, as in fact is done in micro-economic theory, public choice theory in particular. The assumption that planning and decision-making follows a fixed sequence of phases, starting with the statement of goals is challenged by a “retro-active’ orientation, in which assessment and diagnosis come first in the planning cycle. Finally, more spontaneous development from within, that appears to be totally uncontrolled is briefly addressed by referring to theories on self-reference and autopoiesis. Creating market mechanisms: alignment of individual and organisational rationalityA central assumption in the synoptic planning and bureaucracy interpretation of the rationality paradigm is that organisations act as integrated purposeful units. Individual

16

efforts are expected to be jointly directed at the attainment of organisational goals. In the so-called political image of organisations (Morgan, 1986, ch. 6) this assumption is rejected, emphasising that "organisational goals may be rational for some people’s interests, but not for others" (ibid, p. 195). The fact that educational organisations consist of relatively autonomous professionals, and loosely coupled sub-systems is seen as a general condition stimulating political behaviour of the members of the organisation. In public choice theory the lack of effective control from democratically elected bodies over public sector organisations marks these organisations as being particularly prone to inefficient behaviour, essentially caused by the leeway that is given to managers and officers to pursue their own goals besides serving their organisation's primary mission2.Public choice theory provides the diagnosis of instances of organisational ineffectiveness, such as goal displacement, over-production of services, purposefully counter-productive behaviour, "make work" (i.e. officials creating work for each other), hidden agendas and time and energy consuming schisms between sub-units. When discretional leeway of subordinate units goes together with unclear technology this too adds to the overall nourishing ground for inefficient organisational functioning; see Cohen, March and Olsen's famous garbage can model of organisational decision-making (Cohen et al., 1972). Not only government departments but also universities are usually mentioned as examples of types of organisations where these phenomena are likely to occur. Market mechanisms and "choice" are seen as the remedy against these sources of organisational mal-functioning. Notes of criticism that have been made with respect to the propagation of choice in education are that parents' choices of schools are based on other than performance criteria (Riley, 1990, p. 558), that "choice" might stimulate inequalities in education (Hirsch, 1994) and that completely autonomous primary and secondary schools create problems in offering a common educational level for further education (Leune, 1994). The alleged superiority of private over public schools is the most supportive piece of empirical evidence for the claims of public choice theory, although the significance of the results in question is much debated (Scheerens, 1992). At the macro level there is no convincing evidence that national educational systems with more autonomy of schools perform better in the area of basic competencies (Meuret & Scheerens, 1995, OECD, 2001).3

Retroactive planning and the learning organisationA less demanding type of planning than synoptic planning is the practice of using evaluative information on organisational functioning as a basis for corrective or improvement-oriented action. In that case planning is likely to have a more "step by

2 A more extensive treatment of the implications of public choice theory for school effectiveness research is given elsewhere, Scheerens, 1992, ch. 2.

3 The initial results of the PISA 2000 study indicate a statistically significant effect of school autonomy on student performance that is unadjusted for student background characteristics. When such controls are made, as they should, the significant effect disappears.

17

step", incremental orientation, and "goals" or expectations get the function of standards for interpreting evaluative information. The discrepancy between actual achievement and expectations creates the dynamics that could eventually lead to more effectiveness. In cybernetics the cycle of assessment, feedback and corrective action is one of the central principles. Evaluation - feedback - corrective action and learning cycles comprise of four phases: - measurement and assessment of performance; - evaluative interpretation based on "given" or newly created norms; - communication or feedback of this information to units that have the capacity to take

corrective action; - actual and sustained use (learning) of this information to improve organisational

performance. In the concept of the learning organisation procedural and structural conditions thought to be conducive of this type of cycles are of central importance. Examples are: the encouragement of openness and reflectivity, recognition of the importance of exploring different viewpoints and avoiding the defensive attitudes against bureaucratic accountability procedures (Morgan, 1986, p. 90). When the cybernetic principle is seen as the basic regulatory mechanism there is room for autonomy and “self-regulation” at lower levels in the system. This is a particularly helpful phenomenon in education systems, given the usually large degree of professional autonomy of teachers, and tendencies to increase school autonomy as a result of decentralisation policies. The blind forces of autopoiesisThe core idea of Maturana and Varela’s concept of “autopoiesis”is the notion that living systems produce themselves. Their approach is to be seen as an additional point of view, next to the functional perspective according to which a living system purposefully adapts to environmental conditions. The question they asked themselves was: “what connection can a biological system make with itself as a biological system”. Their tentative answer being that the components that make the biological system produce themselves and each other and that the living system can only connect and relate to itself in the form of production relations (Reneman, 2000, p. 26, 27). How this “works” is explained by making reference to homeostatic systems. Homeostatic systems are cybernetic systems holding a critical variable between specific limits. “Living systems maintain their own structure between the limits dictated by their organization” (Reneman, 2000, p. 27). Luhmann (1995) defines complexity as a threshold where it is no longer possible to relate every element of a system to every other element. The capacity of a system for relating its elements depends on the elements’ connective capacity, which, in its turn depends on the internal complexity of the elements: … “because elements must already be constituted as complex in order to function as unity for higher levels of system formation” (ibid, p. 24). At the same time “elements owe their unity

18

exclusively tot he complexity of the system” (ibid, p. 27). So that there appears to be a reciprocal, circular relationship between the elements and the system as a whole. The integrating of a system depends on selections made within the system: “Every complex state of affairs is based on a selection of relations among its elements, which it uses to construct and maintain itself” (ibid, p. 25). Luhmann stresses the fact that these fundamental selections are contingent, which means that they could also have been otherwise. It should be noted that the implication of this view is directed internally, rather than externally. “The system’s focus shifts from an outlook on its environment to a view of the internal “eigen-complexity” of the system” (Reneman, 2000, p. 34). It is beyond the scope of the present presentation to go in more detail about Luhmann’s interpretation of autopoiesis for social systems. From the previous citations the following core characteristics of autopoiesis in social systems can be deduced: • repetitive, circular processes of auto-production • processes that are not planned, nor “managed” but happen unconsciously; systems

are seen as functioning without a controlling agent, or even an observing subject; • processes that are “identity seeking” (closure), but have degrees of freedom of

openness which depend on the notion that all relationships are contingent (see next point)

• processes that are “contingent”, which means that they are not inevitable “as they are”, but could also have been different

• analytically the processes that “come to be” depend on selections among an infinite set of possible relationships

• as a “starting point” for these selections Luhmann proposes what he calls the “Sinn” or “sense” of the system; translated to organizations this could mean that partly unconscious preferences of organization members that make up the “latent culture” of the organization are shaped by their pre-dispositions about the meaning of the organization and their own relationship to it.

In this way “autopoiesis” and self-reference provide a theory on the internal dynamics of the basic, semi-conscious nature of the organization culture. One way to interpret the “sense” of the latent culture would be to see it as the sum or the aggregate of relevant dispositions of the organization members. It is a well-documented phenomenon in other areas of educational research that so called “composition effects”, like the percentage of low achieving students, or the percentage of children from minority backgrounds, are quite important in explaining various aspects of the functioning of a school, the achievement results among them. Form a recent German study the researchers concluded that the nature of a school changed after the percentage of minority children reached a level of 40%. Similar reasoning could be applied to the compositional effects of teacher characteristics.

19

According to the image of autopoiesis and self-reference in organizations “control from outside” has been replaced by “development from within” and “culture” is brought to the fore as a perspective to analyze the functioning of schooling. As far as the effectiveness of the functioning or educational organizations is concerned, the importance of the individual value-positions of the staff members of the school and their aggregates (a teacher compositional effect) are stressed. In its turn this points at the importance of selection policies, of teachers, students and school heads. As far as the internal dynamics of school functioning are concerned the concept of autopoiesis underlines the importance of cyclic processes that go well or go bad. The “good” in this case would be organizational learning, and learning from ones mistakes. The bad would be getting stuck in repeating failure, accumulation of interpersonal distrust and uni-lateral actions. Argyris and Schön (1974) provide many examples of what they describe as obstacles to organizational learning. Studying such complex internal processes would require qualitative research, like participant observation, open interviews and focus group techniques. The merits of the input-process-outcome-context framework for analyzing educational quality revisited The framework allows for a broad range of quality interpretations: productivity, effectiveness, efficiency, equity and responsiveness. All of these quality perspectives are defined in terms of specific relationships between the components. The “eclectic” interpretation values each of the components “on its own”. The model is not about prescribing specific substantive educational objectives; in this the model is just an “empty” descriptive tool that can be used to systematically approach all kinds of educational priorities. The presentation, particularly the part on “informational” modifications of the pure rationality model, clearly shows that there is no ground to limit application to “closed”, “mechanistic” or “technocratic” interpretations. In most practical situations there is simply too much complexity and “contingency” because of situational constraints for authoritative solutions to be feasible. Instead, to the extent that scientific information is available, it should best be seen as input that has to be further verified in interactive processes of actors in the local setting. All that was said, (and will be said in the forthcoming sections) on the importance of context, notjust the institutional but also the cultural context, precludes such mechanistic interpretations. Subsequent modifications of the model with respect to the control dimension do away with autocratic images and underline the limits of social engineering and centralist control. Autonomy at the base of the system, in our case that means autonomy of teachers in classrooms, has always been a characteristic of educational organizations. Various modified interpretations of the rationality model sketch the possible advantages, but also the risks, of further stimulating this autonomy. The risks being spelled out by, for example, public choice theory, the “garbage can model” and by the concept of self-reference in organizations. The

20

concept of functional decentralization, which implies that centralist control in some domains, may be combined with decentralization in other domains, might allow considerable autonomy at various lower levels of the system, but still keep matters checked on one or two important dimensions, outcomes for example. As far as the monitoring of educational quality is concerned the various interpretations of quality and the modifications of the rational model would call for a broad set of methods, including methods that do justice to the interactive nature of some of the basic processes in education. The framework that was introduced, and that in the remaining part of this chapter will be used for the heuristic purpose of generating indicators to measure educational quality, includes political processes, and negotiation, but does not in itself take a political position. It will therefore not satisfy those who opt for an approach to social analysis that starts from a particular political a priori, for example what was called an emancipatory approach in the seventies.

Measuring educational outcomes In this section outcome indicators will be discussed. Outcome indicators are central in productivity and effectiveness interpretations of educational quality; but also play an indispensable role in assessing the equity, efficiency and responsiveness of schooling. A distinction is made between output, outcome and impact indicators. Output indicators are seen as the more direct outcomes of schooling, and most likely measured by means of a form of student assessment, like a standardized achievement test. Attainment indicators, as for example the number of students that complete a certain period of schooling without delay, are of a more administrative nature. Impact indicators are indicators on the social status of students that have reached certain levels of educational attainment. achievement and attainment measures Educational indicators based on output (achievement) and outcome (attainment) indicators have a central place in the education quality debate. Considering output and outcome indicators as the core criteria to judge educational quality confirms to the so-called productivity view on education, which is strongly rooted in economic perspectives. In the educational province the term “effectiveness” is often used to express a similar focus on outputs and outcomes. Yet, there are important, yet gradual, differences between the economical/sociological orientation towards productivity and the educational one. The prime focus in economic and sociological orientation toward educational productivity is the prediction of societal, economic well being at individual and organisational level. The preoccupation of psychological and educational views on productivity are more “forward looking” in the sense that the emphasis is on finding malleable conditions that can maximise educational output.

21

The fact that this dualism is not at all watertight is illustrated by the micro-economical approach of “education production functions”, which is similarly oriented to finding malleable conditions at school and classroom level. One could schematically represent these two orientations as two kinds of causal relationships: - malleable educational conditions and output; - school outcomes and their societal impact. It is also possible to depict this as one, more complicated, causal chain: Malleable educational conditions - output/outcomes – societal impact

The focus in this section will be on the distinction between two kinds of “productivity” or “effect” indicators, “output” indicators as measured by a certain kind of student achievement measure, on the one hand, and “outcome” indicators measured by means of educational attainment measures. It will be argued that student achievement based indicators have a more logical connection with the educational perspective on productivity, while attainment indicators fit better with the economic/sociological perspective. Yet, these functions are also, to some extent interchangeable, and the more recent interest in competencies and key-competencies may take away the edge of this contrast completely. This means that competencies can very well be seen as the “dependent” variables in “internal” school productivity models, and as the “independent” variables in “external” models on the societal impact of schooling. The educational perspective The predominant criterion in school effectiveness studies from various disciplinary origins is achievement. Attainment measures depend on formal levels in the school careers of pupils. Roughly speaking educational attainment scores express the level that individuals or groups of pupils have reached after a certain number of years of schooling. Examples of discrete attainment levels are the end of the primary school period and the end of the secondary school period. Particularly when an educational system consists of many school types, to which societal value is attributed in various degrees, attainment scales can become quite differentiated (see e.g., Bosker & Van der Velden, 1989). When discussing the option of either choosing attainment measures or achievement measures, various underlying dimensions for this choice can be discerned. First of all the choice may depend on different connotations of effectiveness (e.g., maximisation of output i.e. the quantity of educational attainment, vs. enhancing quality). Secondly, preferences concerning band-width vs. specificity of output measures could determine the choice, i.e., the question whether an overall output measure or a more narrowly defined performance indicator is to be preferred. Thirdly, the question of the

22

predominance of a more practical vs. a scientific interest in establishing effectiveness may lie at the background of this choice. Attainment measures are close to the economic notion of productivity as maximisation of output, where output is measured as the amount of products that results from a particular production process of schooling. Achievement, on the other hand, fits better in an interpretation of effectiveness in terms of quality. Achievement tests as effectiveness criteria capitalise on more fine-grained quality differences of the units of output. Attainment measures are cruder output measures than achievement tests, but at the same time they usually imply a broader coverage of the whole spectrum of educational objectives. The passing of a final examination (attainment indicator) depends on achievement in many subjects, whereas achievement tests in school effectiveness studies are often limited to arithmetic and language tests. School effectiveness is both a subject of scientific inquiry and an applied field of interest in educational policy and management. When issues of consumer demands, monitoring of schools and accountability are at stake one cannot do without attainment indicators. In the case of inquiry into determinants of school effectiveness, i.e., input-output, or input-process-output studies in order to define school improvement strategies, output indicators are required that differentiate more strongly between qualities of the units of output and achievement tests are likely to be chosen. In summary, attainment measures are called for when purely economic and applied perspectives of productivity and effectiveness predominate or in case one wishes to explore, in the tradition of sociology of education, the contribution of schools to a person’s status attainment. Achievement measures are more likely to be chosen when quality of education is at stake and when a more psychological interest in cognitive development (or an educational interest in schools as organisations) has the upper hand. Finally, it should be mentioned that the choice between attainment or achievement can be avoided in two ways: (a) by using both, and (b) in the case of a decision oriented use of achievement tests (as when performance standards in the form of cutting scores on tests determine further career options). Intermediate and 'ultimate' effect measuresAre attainment or achievement measures obtained at the end of a particular period of schooling to be considered as the ultimate productivity measures or would only more long term civil effects of schooling, such as employment or job level reached by graduates, qualify as such? Or, moving into the other direction on the scale of “ultimateness” of effectiveness measures, could we use intermediate effects like attendance and drop-out rates as substitute effectiveness criteria? (e.g., Rutter, Maughan, Mortimore, & Ouston, 1979). Searching for ultimate school effects is like looking for the holy grail, since one can go on and on in stating even more ultimate effects. The most likely, be it arbitrary,

23

points in the school careers of pupils to measure school effects are indeed when a particular period of schooling is terminated and transition to a higher school type or into the labour market takes place. Post-school effect measures could be seen as important in macro-level applications of educational indicators for purposes of monitoring national school-systems. Also, post-school effect-measures could be seen as important criteria to gain insight into the predictive validity of effect measures at the end of the period of schooling. Attendance and drop-out rates are better treated as process-measures in school-effectiveness studies, because they generally function as means rather than as desired ends of schooling. General vs. curriculum specific achievement testsWhen the decision is taken to use achievement rather than attainment output data, there is a further option in the choice of the type of test. Madaus, Kellaghan, Rakow, and King (1979) have provided arguments in favour of curriculum-specific tests and against the use of general achievement tests (like the Scholastic Aptitude Test). One of their arguments is that larger school (or class) effects are demonstrated when curriculum specific tests (exams in their case) are used. Before offering a few lines of thinking in determining the choice of output measure along this particular dimension, it should be remarked that general vs. curriculum specific achievement measures should be seen as a continuum with many discrete scale-points rather than a dichotomous choice between two extremes. Varying from curriculum specific to general aptitude measures one could discern the following types of measures: - trained test items; - content specific measures; - Rasch-scales of narrow content areas; - subject specific tests; - general scholastic aptitude tests; - intelligence tests. A general guideline to choose from these alternatives would be to use the more specific measures up to the degree that the application purpose is closer to the micro situation of classroom-instruction. A line of thinking which perhaps offers a more fundamental solution to this problem of choice, would be to choose the type of outcome measure that has the greatest predictive validity with respect to the more ultimate educational effects. To give an example: when measuring achievement at the end of a specific stream of vocational education, we might prefer content-specific measures, assuming a close connection between the curriculum and skills that are required in the job-situation. It should be noted here, however, that this latter kind of criterion choice further depends on the theory one holds about the relationship between education and the labour market. Departing from a credentials or screening theory, certification itself would be the best criterion for effectiveness, whereas achievement is more connected to the human capital philosophy.

24

Controlling for confounding of measures of effectSo far it was assumed that the measures on attainment or achievement scales can indeed be interpreted unequivocally as criteria of productivity or effectiveness. This is not so. One source of confounding of these output measures that is well-known is the initial or even innate ability of pupils. In school effectiveness studies it should be attempted to separate the contribution of innate characteristics of pupils from the net-effect of school characteristics. Although this source of confounding the interpretation of output measures is well-known, critical reviewers of effective schools research have noted that in many cases control for innate characteristics of pupils is inadequate (Purkey & Smith, 1983). A second source of possibly biased interpretations of school effects are selective policy- measures at the school level. Examples are: lenient vs. a strict policy in letting pupils pass from one grade to the next, a more or less conservative policy in allowing students in secondary education to go in for their final examination, a more or less reluctant attitude when it is to be decided to send pupils to special schools (primary education) or to lower categories of secondary education (when a country has a differentiated secondary education system). This type of selectivity bias is usually accounted for when economic measures of school output are used, for instance, number of graduates divided by the total number of students in the cohort that entered the school as many years back as the normal duration of the school-period. Usually in individual level measures of attainment selectivity will also be accounted for, because drop-outs will still receive a score on the attainment scale. When achievement data aggregated to the school level are used as the effect-criterion this type of selectivity bias is usually neglected, however. The consequence of this practice is that the corresponding estimation of school effects might well be confounded and corresponding policy- or managerial decisions consequently unfair. The general solution to this problem would be to obtain some kind of measure of the selectivity policy of schools and use this information as an additional independent variable (defined at the school level) to separate its effect from the other independent variables that stand for the more genuine determinants of school effectiveness. Statistical techniques to model attrition bias may help in solving the problem of separating selectivity effects from the influence of other independent variables (see e.g., Hausman & Wise, 1979). Although generally people use the school effect concept without any hesitation, there are, however, many different underlying operationalisations. Some, of course, have to do with the subject matter areas in which the performance or ability of the pupils was assessed (see also Chapter 3), others have to do with gross versus value added indicators. More important is that when using value added measures different corrections are being made before the actual school effects are assessed, i.e. in different research studies different covariates are used. When looking at the type of covariates used, four different approaches can be distinguished, which to be indicated as “gross school effects“,

25

“unpredicted achievement based school effects”, “learning gain based school effects”, and “unpredicted learning gain based school effects”. Gross school effects The first operational definition uses as the measure for school effect the mean (uncorrected) achievement score of pupils in a certain school. The value of this definition lies in its use within a criterion referenced framework: if a standard is set (or: if a growth continuum is specified) a priori this gross school outcome measure provides the information to judge whether the school performs above, at or below the standard. It does, however, not imply that all pupils within that school meet the standard. This definition can be labelled as the 'gross' school effect. In operational terms it is the mean achievement, i.e. it is averaged over the pupils within a school, possibly with a correction for sampling error. School effects based on the assessment of over- or under-achievement as compared to predicted achievement The second operational definition starts from predicted achievement. A prediction equation is estimated from the data, where achievement is predicted from aptitude, socio-economic status, age, gender, ethnicity-status, and other student variables. Next the parts of the distribution of student achievement scores that are above and below the predicted level are estimated. The reasoning behind this approach is that schools widely differ in their student populations, and that, since these background variables have a strong relationship with achievement, their effects on achievement should be partialled out. Most of these variables are static, and not subject to -much- change, though the aptitude of a child may. For this reason the aptitude assessment should - ideally - take place before or at school entrance. School effects based on learning gain The third operational definition can be seen as a specific case of the second one: achievement is predicted from prior achievement or -what is more appropriate- the difference between both is used. Once again the same argument applies as in the case of the aptitude assessment. If prior achievement is assessed at a later point in time than school entrance, the school effect transforms into the effect of a school on its pupils within a certain time-interval. School effects based on predicted learning gain The last and seemingly most strict definition combines the previous two: predicted learning gain forms the basis. Using a post-test score corrected for a pre-assessment score, this score is in its turn corrected for aptitude, socio-economic status, age, gender, ethnicity-status, and other student variables, since these are related to the learning progress that pupils make. In this case prior achievement as well as aptitude should - ideally - be assessed at school entrance.

26

In the literature on school effectiveness school effects are mostly referred to as value added measures, i.e. some kind of intake adjustment is applied. We have seen however, that in actual fact there are at least three different sets of value added based school effects. Except for the first definition of a school effect (gross school effect), where no adjustment for intake differences between schools is employed, the value added based definitions can be made more strict by not only correcting for the student level effect of the covariates but also for the potential extra effects of their aggregates. It is often observed that, for example, the average socio-economic status of the student populations of schools has an effect over and above the individual socio-economic status variable. This indicates that being a working class child has a negative effect on achievement, but being also in a school with a majority of working class children has a substantial extra negative effect on achievement as well. In all cases, however, causal attribution (showing that schools cause the effect) is difficult. Raudenbush & Willms, argue that, since randomisation cannot be achieved, assignment of students and schools to treatments should be "strongly ignorable". This qualification expresses that the different treatment outcomes for a student and a school "are conditionally independent of treatment assignment given a set of covariates" (Raudenbush & Willms, 1995, p. 312) which implies that value added measures should be preferred. But on the other hand, it is possible that value added based school effects underestimate true school effects. When "privileged" students are more likely to choose "good practice" schools, controlling for student's socio-economic status, for instance, results in over-adjustment. To complicate things further, it should be pointed out, that in compensatory systems, school effects may appear only after adjustments for covariates are made. In compensatory systems unfavourable scores on the covariates are compensated by (and thus correlated with) more intensive and adapted educational practices. And moreover, these school effects then can disappear again after context variables, that are negatively associated with good educational practices for the same reason, have been taken into account. A note on the size of school effects The question referring to school effectiveness and the size of school effects should be distinguished from questions about the practical relevance of school effects. There are at least two kinds of practical issues at stake when dealing with this latter question. First of all one should consider the situation where information on the value added to educational outcomes by each school is used for political or practical decision making. Examples are: rewarding well-performing schools by providing them with extra resources, and comparing schools among each other in so called league tables. The purpose of these league tables being the provision of information to parents to facilitate the choice of a school for their children. Goldstein and Thomas (1996) have shown that, at least in the UK, it is possible to discriminate significantly between schools only when extremely high and extremely low schools are compared. Goldstein (1996) provides an example in which "the confidence intervals for each

27

school cover a large part of the total range of the estimates themselves, with three quarters of the intervals overlapping the population mean ..." (p. 5). The implication being that it is generally not possible to discriminate in a precise and reliable way between schools for the practical concerns in question. A second type of practical implication of "value-added" school effects is by considering the consequences for individual pupils. It has been established repeatedly in the literature (Purkey & Smith, 1983; Brandsma & Doolaard, 1996) that relatively small differences between schools will nevertheless have important consequences for the educational career of individual pupils. Purkey and Smith, for instance, discerned an entire school year's difference between the average pupil in the most effective schools and the average pupil in the least effective school. Brandsma and Doolaard demonstrate that pupils from effective schools have better chances to be advised to pass on to "higher" types of secondary education in the Dutch educational system. These authors distinguish between four IQ-bands, and show that in highly effective schools pupils from the three lower bands score at a level that is equivalent to two curricular tracks (out of a total of 5) higher than pupils from ineffective schools. In the highest IQ-band the difference is equivalent to one track. A point that is made by Bosker and Witziers (1996) in this context is that school effects, small as they may be, have implications as described in the above, for not just one, but for all pupils of the cohort the effects were measured for. The economic perspective: theories and research findings The economist approach to education generally analyses the rational choice behaviour of actors in education. Particularly the question about the so-called utility of a person who follows education is studied. This means that all the costs a person has to make to follow a particular education program are counted on the one hand. On the other hand the benefits in terms of increased earnings (i.e. the difference in earnings between having obtained a particular qualification and not having obtained that qualification) are calculated. This is the basis of the best-known economic theory on education: human capital theory. Other theories also consider the utility of employers to engage persons with certain qualification levels. The research that economic theories have generated gets particularly interesting for the topic at hand, when the question is raised about the competencies that make qualified individuals more attractive to companies than unqualified individuals. In their turn these competencies could be taken as key targets for schools and training institutes to maximise the external relevance of their programs. Human capital theory treats education as a trade-off between costs and benefits. “ In this theory, individuals incur costs, including tuition and foregone earnings while in school, to acquire skills that increase their productivity and result in subsequent wages higher than they otherwise would have earned. A long tradition of research has

28

focussed on the magnitude of the return to educational investment, showing that human capital investments tend to pay off at least as well as investment in physical capital” (Levy and Murnane, 2001, p. 154). One of the research supported characteristics that was found in explaining this general result was that employees with more formal education have better ability to deal with unanticipated events (like non-routine problems at work or forced job changes), than do workers with little formal education (ibid, p. 154). Signalling theory concentrates more on the perspective of the employers. Employers use the information on formal levels of schooling as a “signal” that potential employees possess certain skills that are valued for the work in question. Recent research, the International Adult Literacy Study has shown that it is not just the formal qualification level that counts, but some competencies, in this case literacy, has an effect over and above the formal level of schooling. Principal agent theory presents an economic perspective on the utility functions of employees on the one hand and managers on the other. The theory assumes that the agent (the employee) has a utility function that may differ from what is valued by the employer. For example, a certain degree of leisure, that employees could permit themselves may be pleasant for the former but not so much for the principal (employer). In theory the employer would need to closely monitor the behaviour of the employee to minimize such non task-related priorities or activities. Since monitoring may be difficult and costly, relying on employees having acquired certain basic skills and attitudes that give some guarantee of reliability, education functions as a substitute for monitoring and supervision. It is interesting to note that the competencies in question are not just technical skills but also more attitudinal characteristics like perseverance and honesty. Levy and Murnane (2001) summarize research findings that are the result of various kinds of economic research that are driven by these theories and have generally been targeted to the question what competencies, related to formal schooling, could explain the attractiveness of qualified individuals to employers. They mention five “key competencies”: 1) Basic reading and mathematical skills are important in determining long-run

labour market outcomes, including the ability to adjust to changing circumstances. 2) The ability to communicate effectively, both orally and in writing, is important in

determining long-run labour market outcomes. 3) In modern firms it is increasingly important to possess the ability to work

productively in groups. 4) The latter condition also emphasises elements of “emotional intelligence”

including the ability to relate well to other people. 5) Familiarity with computers is of growing importance in the labour market.

29

They present their findings as tentative, however. The fact that employers are ready to pay wage premiums to employees with higher levels of formal schooling may be explained by the hypothesis that these employees have better communication and teamwork skills and or are more apt to adaptation and continuous learning, but the evidence is inconclusive. These authors also point at a number of institutional arrangements that blur the picture of a straightforward relationship between companies hiring and promotion policies and acquired key-competencies. They mention “the organization of work, the links between firms and educational institutions, and the nation’s distribution of income as modified by the welfare state” (ibid, p. 167). Identifying key competencies as a general approach to closing the gap between schooling and the demands of society The term “competencies” is used, as the magic word to express what is needed to function effectively in a given social and cultural context. Clearly this question goes beyond that of education responding to the demands of the labour market. Similarly the concept of “competency” is meant to have a broader meaning than the transmission of knowledge and skills in institutionalised education settings, like schools. “Key competencies are considered as structured around meeting demands of a high degree of complexity and are comprised of cognitive as well as motivational, ethical, social and behavioural components” (Weinert, 2001, cited by Rychen, 2001, p.11). It is recognized that acquiring these competencies also happens in other settings than formal schooling, e.g. in the home situation of children and in the work-place (learning on the job). As the economic approach already illustrates competencies are identified by analyzing valued “real life” situations, such as the functioning of highly schooled individuals in work situations. Taking a broader outlook this valued situation could be described as the “effective” or “competent functioning” of an individual in a given society, i.e. a given cultural, social and linguistic context (Haste, 2001). This author identifies five “competency domains” which she feels are essential in the immediate future and possibly universally relevant. These are: “technological competency; the ability to deal with ambiguity and diversity, the ability to find and sustain community links; the management of motivation, emotion and desire; and finally, the sense of agency and responsibility”. (Haste, 2001). Apart from these “comprehensive” characteristics (cognitive, motivational, social and ethical components), Rychen, ( 2001, p. 11,12) mentions two other core elements in current, multi-disciplinary based, definitions of competencies: - “Key competencies are seen as transversal or generic in the sense that they enable

individuals or groups to participate effectively in all relevant social fields with their specific power and social relations, challenges and capital at stake.

30

- Key competencies call for a higher order of mental complexity involving an active and reflective approach, which includes the capacity to distance oneself from one’s own socialising process and even one’s own values”.

When approached from the context of formal schooling, the idea of competencies can be placed on a continuum of types of educational outcomes, already introduced in a preceding section, that runs from specifically content oriented to “content free” personality traits. Discrete positions on this continuum are presented in Figure 4. - outcomes as measured by tests included in textbooks - outcomes as measured by implemented school curricula (teacher developed) - outcomes as measured by tests based on the intended national curriculum - outcomes as measured by international tests covering the common core of a

range of national curricula, e.g. TIMSS - “literacy” tests, aimed at measuring basic skills in reading, mathematical and

scientific reasoning, e.g. PISA - competencies as multi-facetted dispositions of individuals, including cognitive,

motivational and possibly other components - personality traits, like internally or externally determined locus of control,

independence, general intelligence

Figure 4: A continuum of educational outcomes, running from highly content bound to personality dependent

It remains to be seen whether all “ambitions” inherent in the idea of key-competencies can be met. A core question is whether the multiple component idea of competencies survives empirical measurement. For example in a current dissertation study an instrument was developed to measure “career developmental competencies”, involving cognitive, behavioural and motivational components. In analyzing the instrument it appeared that these three components were relatively independent, and thus could hardly be seen as forming one integrated construct.Perhaps the heuristic value of the term will be that attention is drawn to other than purely cognitive educational outcomes, whether these can be shown as an integral part of a particular competency or as independent dimensions. The practical meaning of the competency issue is that the answer to an effective linking of education to societal functioning should partially be sought in the teaching of more general skills and in paying more attention to motivational and attitudinal aspects. Also the idea of meta-skills, such as general problem solving skills and “learning to learn” could stimulate a more reflective and self-steering attitude of

31

future citizens. In this way “competencies” are considered as useful tools in serving the “employability” of future citizens.

Overview of the most important categories of outcome indicators In the table below, Table 3, an overview is given of the different categories of outcome indicators that could be used to monitor quality aspects related to the productivity, the effectiveness and equity of education. Table 3: Overview of educational outcome indicators Main categories of outcome indicators

Sub-categories Technical issues

Output indicators Achievement measures - subject matter based - literacy (reading,

mathematical, scientific) - competencies (e.g. learning

to learn)

- Value-added effect

measures; growth curves - Assessment methodology

(ranging from multiple choice tests to authentic assessment)

- Criterion versus norm-referenced testing

Outcome/attainment indicators

Attainment measures - graduation rates - proportion of students

graduated without delay - drop-out rates - class repetition rates

Controlling for selection oriented school policies

Impact indicators Social participation rates - (for each attainment level) %

of employed at a certain job level)

- % of unemployed - (for lower school levels: %

enrolled in follow-up education

- degree of social participation (social capital)

- adult literacy rates - average income, for each

attainment level; earning differentials

Availability of national educational and labour market statistics

Appropriate measures of social capital and adult literacy

32

- skill shortages and surplus

Process indicators Although indicators on educational processes can be use in a “stand alone” way, according to what was earlier described as a disjointed application of indicators, it makes more sense to see them as part of indicators systems, in combination with outcome, input and context indicators. In this section process indicators will be considered within the framework of macro-level, i.e. national level, indicator systems, and, secondly, as part of multi-level indicator systems, where transformation processes at school level are at the centre. System level indicator systems The OECD Education Indicators project (INES – see the Education at a Glance publications) uses the following categorization, which is evident from the table of contents in the Education at a Glance Publications (OECD, 1998). The main categories are: A) The demographic, social and economic context of education (e.g. Literacy skills

of the adult population) B) Financial and human resources invested in education (e.g. Educational

expenditure per student) C) Access to education, participation and progression (e.g. Overall participation in

formal education) D) The transition from school to work (e.g. Youth unemployment and employment

by level of educational attainment) E) The learning environment and the organization of schools (e.g. total intended

instruction time for pupils in lower secondary education) F) Student achievement and the social and labour-market outcomes of education (e.g.

Mathematics achievement of students in 4th and 8th grades, and Earnings and educational attainment)

These 6 categories can be classified in various ways. The context-input-process-outcome scheme, as used throughout this chapter, is the most likely way to do so. Accordingly category A is in the context domain, category B refers to inputs, categories C, D and E refer to different interpretations of the process dimension, and category F to an output/outcome dimension. See Figure 5.

Figure 5: Ordering of the Ocontext-input, proc

In Figure 5 arrows between sense, these categories are exis used in a descriptive sensbeen analyzed so far. A second way to look upon t“flow” types of indicators. Cacategories C and D, and to soA stock indicator describes quantitative and qualitative tex). A flow indicator refers toteacher, to a different part of profession in school year y). It should be noted that claindicators adds a third interprof process indicators around: a) malleable conditions of

Figure 1); b) checks on program implec) flows of units through the Since our subject is the quindicators will be predominanto issues of the quantity ofsurvival rates, can be seen asas was evident from the proutcome measures of educatiis considered as an issue of th

Context

demographic, social and economic context of education

Input

financial and human resources invested in education

Output/Outcomes achievement labour-market outcomes

Process

access, participation, progression transition school to work learning environment and organization

33

ECD-INES education indicator set, according to a ess and outcome scheme

the boxes have been omitted since, only in a very loose pected to be linked in a causal way. In fact each category e and interrelationships between indicators have hardly

he OECD indicator set is by distinguishing “stock” and tegories A and B are typically stock indicators, whereas

me extent F refers to flows. a relevant educational aspect at one point in time in rms (e.g. the number of qualified teachers in school year the transition of an educational unit, e.g. a student or a the system (e.g. the number of teachers that have left the

ssifying “transition” or “flow” indicators as process etation to process indicators. There are really three types

transformation processes (see the systems model in

mentation within the context of program evaluation; educational system (OECD).

ality of education, the first interpretation of process t. Indicators on access and participation are more related supply and demand of education. Success-rates and

belonging to the domain of participation indicators, but, evious section, these kind of indicators are treated as onal attainment. Similarly transition from school to work e societal impact of education.

34

In Figure 5, where the OECD indicator set is categorized, process indicators seen as malleable conditions may refer to two different types of information; to policy measures and structures that are defined at the national level on the one hand, and to aggregated school characteristics on the other. In Table 4 below, some examples of process indicators that are actually defined at the national system level are provided. Table 4: Examples of system level process indicators

Process indicators defined at the level of national education systems

• Teaching time per subject • Total hours of instruction per year, for specific grade levels in primary and secondary

education • Opportunity to learn, in terms of expert ratings of test curriculum overlap • The locus of decision-making in education, by education level

(This indicator shows at which administrative level decisions in sub-domains of education – curriculum, personnel management – instruction, resources – are made with a certain degree of autonomy)

• School autonomy (this indicator is actually included in the concept of locus of decision making)

• Education standards by level [e.g. targets like increased completion rates, percentage of students scoring at or above a particular achievement level].

• Whether or not formal examinations are taken at the end of each school category • The degree of categorization and formal streaming at secondary level • The evaluation capacity of the system (defined as a quantification of the occurrence and

intensity of various evaluation forms, such as national assessment programs, examinations, school inspection, an educational management information system etc.)

• The magnitude and diversification of an educational support structure in the country (possibly comprising a curriculum development unit, ICT services, school counseling, an educational assessment and testing unit etc.)

• The division of private, government dependent and public schools • Incentive based policies to stimulate school performance • The degree to which school choice is free.

It should be noted that system level process indicators may be interpreted as context indicators when one takes the perspective of the school.

35

comprehensive indicator systems including process indicators of school functioning aggregation levels

Educational systems have a hierarchical structure where administrative levels are “nested”. Indicator systems may ignore this hierarchical structure by using statistics that are defined at national level or formal characteristics of the system. Examples are: pupil teacher ratio computed as the ratio of all pupils and all teachers in a country and teacher salaries defined on the basis of nationally determined salary-scales. Even when considering use of indicators at national level only, there are two main advantages to use data at lower aggregation levels: • disaggregated data allow for examining variation between units, e.g. the variance

between schools in success rates on examinations; • disaggregated data allow for better adjustments and more valid causal inferences;

the best example in education is the use of so called “value-added” performance indicators based on achievement test scores adjusted for prior achievement and/or other relevant pupil background characteristics.

When it is the intention to relate, for example, school organizational characteristics to pupil achievement, disaggregate data at pupil level are required to carry out appropriate multi-level analyses. Particularly when indicators are used for program evaluation purposes the above mentioned advantages of disaggregate data are important, because they provide firmer ground to answer causal questions about program effectiveness. A final added advantage is that the relevance of indicator systems for lower administrative levels (e.g. school districts and individual schools) grows when disaggregate data is available. Time-frame For the purposes of evaluating large-scale educational development programs experimental and quasi-experimental designs have been recommended. (Ezemenari, Rudqvist & Subbarao, 1998). Although there is no question about it that (quasi-)experimental designs should be used whenever possible (compare Campbell’s famous idea of “Reforms as Experiments, Campbell, 1969), they are often not feasible. Using educational indicators in a longitudinal way, whereby the same units are measured at several points in time, is a viable alternative to experimentation. Function of process indicators In previous sections various interpretations of educational process indicators were referred to. In this section process indicators that reflect malleable conditions of basic transformation processes at school level will be placed central (see Figure 1). School organizational functioning and teaching and learning at classroom level are examples of such educational transformation processes.

In general it could be said that such process indicators shed some light on what happens in the “black box” of schooling. Process indicators are interesting from the point of view of policy and management since they refer to conditions that are malleable and thus the subject of active policies to improve education. In a later chapter the perspective of school effectiveness research will be presented as the most likely rationale for identifying and selecting process indicators. Accordingly those process indicators are selected that show positive associations with educational output and outcomes. Ideally such process indicators should be able to predict output (as in “education production functions”: increments in “process” conditions predicting increments in output according to an exact function). To the extent that such instrumental knowledge would be complete process indicators could rightly be used as substitutes of output indicators. Given the fact that the education production function is debated and, more generally, school effectiveness knowledge is “incomplete”, to say the least, such a strong instrumental interpretation is not realistic. This leaves two further possibilities for the use and interpretations of process indicators: • as “annex” to output indicators, whereby in each and every situation of their use

the association between process and output indicators would have to be explored with the intention to “explain” differences in outcomes between schools and between educational systems;

• a weaker interpretation of instrumentality, where process indicators are seen as instance of educational good practice, and, in this way, could lead to value-judgements about educational quality even in the absence of output data.

Within the context of program evaluation process indicators are sometimes defined as checks on the actual implementation of the program. This interpretation of process indicators is easily reconcilable with the one used throughout this section. Implementation checks are a more basic and administrative type of monitoring, whereas process indicators as defined above, are referring to more generic causal processes of organizational functioning and teaching and learning. When process indicators are used over and above implementation checks, they say more about why an (implemented) program works. Figure 6 illustrates this.

Figure 6: Use of proce

program inputs

outputs

degree of implementation of program inputs

36

ss indicators in the co

transformation processes following program implementation

ntext of program evaluation

37

School effectiveness research and the identification of process indicators The most likely rationale for selecting process indicators is to choose those variables that are manipulative predictors of school output. Research literature on school effectiveness can be used as a source to identify promising process variables. Generally speaking, school effectiveness research is aimed at discovering school characteristics that are positively associated with school output, usually measured as students' achievement. Various research traditions can be subsumed under this heading, including (in)equality of education (sociological), educational production functions (economical), school improvement and effective schools, and teacher- and instructional effectiveness (psychological). Apart from these, more theoretical and analytic contributions from organizational science and micro-economic theory of public-sector organizations can also be sources of inspiration in selecting process indicators (see Cameron & Whetten, 1983 and Niskanen, 1971, respectively). For reviews of the school effectiveness research literature see Scheerens & Bosker, 1997 and Scheerens, 2000. More details about the research approach and substantive results of school and instructional effectiveness research will be provided in Chapter 3. In the overview of examples of process indicators on school functioning this research literature has been the predominant source. Table 5 summarizes effectiveness enhancing factors at school level, and Table 6 shows variables at the level of teaching and learning which have been associated with effectiveness.

38

Table 5: Overview of examples of process indicators of school functioning

Process indicators defined at school level

Community involvement • the degree of actual involvement of parents in various school activities (the teaching and

learning process, extra-curricular activities and supporting activities) • the percentage of the total annual school budget that is obtained from the local

community • the amount of discretion local school boards have in the conditions of labour of teachers

[possible operationalizations in EDUCO project – El Salvador] School financial and human resources • average years of teachers’ experience per school • school level pupil teacher ratio • average class size per school • proportion of formally qualified teachers per school • school managerial “overhead” (principal and deputy-principal fte per 1000 students) Achievement oriented policy • whether or not schools set achievement standards • the degree to which schools follow (education) careers of pupils after they have left the

school • whether or not schools report achievement/attainment outcomes to local constituencies Educational leadership • the amount of time principals spend on educational matters, as compared to

administrative and other tasks • whether or not principal’s appraise the performance of teachers • the amount of time dedicated to instructional issues during staff meetings Continuity and consensus among teachers • the amount of changes in staff over a certain period • the presence or absence of school subject-related working groups or departments

(secondary schools) • frequency and duration of formal and informal staff meetings Orderly and safe climate • statistics on absenteeism and delinquency • ratings of school discipline by principals, teachers and pupils Efficient use of time • total instruction time and time per subject matter area • average loss of time per teaching hour (due to organization, moving to different rooms,

locations, disturbances) • percentage of lessons “not given”, on an annual basis Opportunity to learn • teacher or student ratings of whether each item of an achievement test was taught or not

39

Evaluation of pupils’ progress • the frequency of use of curriculum specific tests at each grade level • the frequency of use of standardized achievement tests • the actual use teachers make of test results Ratings of teaching quality • quality of instruction as rated by peers (other teacher) • quality of instruction as rated by students

Table 6: Overview of effective teaching and learning variables

Effective teaching variables

Main teaching factors • opportunity to learn • structuring and scaffolding (cognitive structuring) • stimulating engagement (motivational structuring) • climate aspects: - task orientation

- mutual respect - orderliness, safety

• monitoring and questioning • feedback and reinforcement • modeling learning and self-regulation strategies • “authentic” applications • adaptive teaching Learning strategies of students overt: engaged learning time student use of resources cooperative learning covert: self-regulatory capacity auto-control meta-cognitive “actions” learning styles

40

Input indicators According to our basic input-process-outcome-context framework inputs provide the material and immaterial pre-conditions for the core transformation processes in organizations. In the case of education, and taking the school as the level where teaching and learning as the primary transformation process take place, the following main categories of inputs can be discerned: - financial and material resources - human resources - background conditions of the students Household and community characteristics as well as guidelines and operational frameworks from administrative levels above the school might also be considered as inputs, when the school is chosen as the level of analysis. The latter have been described in this chapter under the heading of system level processes and system level process indicators. The former (household and community characteristics) appear to be better classified as “context” characteristics, particularly when the environment provides given antecedents and constraints rather than active control measures. These choices, however, are quite arbitrary. For out purposes the framework primarily serves a heuristic function aimed at providing a comprehensive overview of indicators that can be used to measure a broad range of aspects of educational quality. Input indicators contribute to assessing the quality aspects of effectiveness (which inputs “work”), efficiency (which inputs work at the lowest possible price) and equity (equitable distribution of material and human resources inputs to schools, the degree to which outcome levels depend on student background characteristics). Financial and material resources Financial and material resources indicators can be defined at system and at school level. Financial indicators are predominant at system level, while material resources indicators make more sense to be formulated at the level of schools. In Table 7 below examples of financial and material resources indicators, defined at system and at school level are given.

41

Table 7:

System level financial and material resources indicators

• proportion of Gross Domestic Product spent on education • educational expenditure per student • proportion of public and private investments in education • public investment in educational research and development • total expenditure on programs and special facilities for disadvantaged students • state provision of ancillary services • household expenditure and public subsidies to parents • percentage of spending on salaries for administrative personnel • percentage of spending on pensions for educational personnel • percentage of spending on salaries for teachers

School level financial and material resources

• proportion of the school’s budget that is acquired through other than public funding • school building facilities • classroom equipment (furniture, computers, etc) • school supplies like pencil and paper, chalk board, flipchart • availability of textbooks in the major school subjects • basic services like separate toilets for girls and boys, water, electricity, heating,

telephone, provision of ancillary services, regarding nutrition, health and transportation

Human resources A well- qualified and motivated teaching force is to be seen as one of the most vital assets for educational quality. Indicators on teachers as individuals or of the total stock of teachers in a country can be categorized in various ways. In the table below a distinction is made between descriptive background characteristics of teachers, knowledge and skills, attitudes and morale relative to general working conditions and attitudes with respect to the work situation at school and student staff ratios.

42

Table 8: Categorization of teacher indicators. Teacher background characteristics

• Age, sex, and ethnicity distribution • full-time/part-time distribution • certification/license status • formal qualifications • year of experience • language • health, specifically HIV • in-service training history Teacher professional knowledge and skills

• general knowledge • content knowledge • knowledge about pedagogical and didactic strategies • knowledge about students • beliefs and attitudes about teaching • flexibility in adapting teaching repertoire Teacher working conditions

• salaries (relative to other professionals) • working time • average class-size • merit based incentives • other incentive policies • career structures • teacher training/certification requirements • teacher autonomy • standards-based teacher appraisal • secondary working conditions (e.g. vacations) • exposure to external inspection Teacher morale and status

• opinions about career and job mobility • teacher morale • perception about being needed by society • perceived status as a teacher • appreciation of general working conditions • appreciation of the work situation at school of current employment • job mobility • sense of political efficacy Staff to student ratios

43

• system level student teacher ratio • school level student teacher ratio • support staff student ratio (system and school level) • school managerial “overhead” relative to the number of students

Only a part of these indicators is likely to be available on the basis of national statistics, and would depend on the availability of school or teacher surveys. Student background characteristics To some perceiving students as the “raw material of the education production process” may seem to stretch the economic metaphor a bit far. More psychologically inclined analysts might maintain that students are the main producers of learning and the attainment of learning results. For analytical purposes it makes nevertheless sense to recognize that the home background and intellectual capacities of students make a lot of difference. When effectiveness and productivity interpretations of quality are at stake it is usually considered relevant to construct value-added outcome indicators, that is indicators that show the effect of malleable conditions of schooling over and above the impact of background conditions. For equity interpretations of quality student background characteristics function as categorization criteria, to contrast groups with one another, for example, boys and girls, schools with a relatively small and a large proportion of students from minority groups, etc. As will be documented more fully in chapter 4, when presenting results of empirical school effectiveness research, student background variables do not only have an impact as individual student characteristics but, when considering the average achievement level of a class or school, also as a so-called compositional effect. This means that average achievement is influenced by the composition of the group in terms of intellectual capacity and socio-economic background. For our purposes a distinction is made in two categories of student background characteristics. The first is selected because of its more general association with educational performance, the second refers to specific circumstances, particularly in developing countries, that are likely to interfere with a student’s capacity to optimally benefit from schooling.

44

Table 9: General student background characteristics

- general intelligence or scholastic aptitude - socio-economic status - mother’s level of educational attainment - gender - ethnicity Student background characteristics associated with specific situational constraints

- discrepancy between language spoken at home and language at school - distance a student has to walk to school - the amount of out of school time a student has to spent on labor - whether the students has had a meal when arriving at school - place to study at home - number of books in the home - malnutrition - ill health/HIV

Contextual issues and context indicators What has already been covered Within the input-process-outcome-context framework specifying what is meant by the context depends on the level at which the central transformation process is defined. Throughout this chapter two interpretations have been used. Most of the time transformation processes at school level have been concentrated on. When transformation processes at school are further differentiated to distinguish primary teaching processes at classroom level and secondary, supporting management and organization processes at school level, a multi-level model results, in which everything “outside” the school is defined as the context. In this kind of conceptualization “context” could be further subdivided in the direct environment, local community and local/regional administration on the one hand, and the national context on the other. The second model interpretation that has been referred to now and then in this chapter is the one where education is considered at one level only, the national system level. According to this interpretation the context is defined as the relevant environment of the “education province” as a whole. As such the general affluence of a country, demographic tendencies, cultural aspects that impinge on values that are important in education and the institutional infrastructure of a nation could be seen as the context of education.

45

So far “malleable” conditions at the national level have been described already, and from the perspective of the school these form an important category of contextual conditions. Particularly the patterns of functional (de) centralization, which result when one determines the locus of decision-making for particular decisional domains, and the relative autonomy with which decisions are taken at a certain level, is quite fundamental for the functioning of schools. It includes for example the question of centralization and decentralization in the curriculum domain. Other important nationally malleable conditions that were mentioned were: the degree of development of evaluation capacity throughout the system, and the question of explicit standards and formal examinations. Having covered this important category of school context indicators, what are left are maybe a few antecedent conditions at national level, and a few antecedent conditions at the level of the school community. As stated before the, sometimes relative, distinction between “malleable conditions” and “given, antecedent conditions” expresses the fact that some conditions have a more permanent status and if at all changeable, this would be more in the sense of gradual evolution than by means of active steering operations. So, a closer look will be taken at these remaining areas of context indicators. Firstly, given conditions in the society at large that are relevant for the educational system as a whole, secondly, given conditions that are part of the national education system, and, thirdly, given conditions in the immediate environment of the school. When society at large is considered as the context of the “education province” Obviously when dealing with the whole of society it is totally beyond the scope of this presentation to do a comprehensive analysis. Instead some areas and dimensions will be mentioned that are believed to imply relevant contextual constraints for the education system. Demographic information concerning the magnitude of age cohorts and longer term projections of these indicate the scope of education in terms of the number of students that need to be serviced over time. Labor market statistics concerning employment rates, wage differentials and skill shortages and surpluses in different sections of the labor market. It should be noted that these kind of indicators can also be analyzed as impact indicators. The purpose and time dimension makes the difference. When such indicators are used pro-actively in the context of planning educational provisions they can be seen as part of the relevant societal context. In the case of evaluating the impact of the functioning of the education systems these indicators are used retrospectively and therefore can also be classified as a particular kind of outcome indicator. General information on the economy, such as indicators on the affluence/poverty of a country, is obviously relevant in indicating the likely margins of educational expenditure.

46

Aspects of the culture and cultural tradition of a country. The value that education has in a particular cultural tradition will vary as well as norms and values that bear upon the expected role behavior in education, e.g. the way authority of teachers and the participation of students are seen. Another example could be the degree of trust in others. As the World Value Study (1990) has shown, there are enormous differences in the percentage of people in a country that answer yes to the question whether or not most people can be trusted (in Norway this is 65%, in South Africa 18% and in Turkey 6.5%; source, OECD, 2001). Perhaps it would also be interesting to make more systematic study of the implications for education of Hofstede’s categorizations of national cultures Hofstede, (1980). Hallinger, (2001) provides a fascinating analysis of the way educational leadership and school improvement are interpreted in the culture of Thailand, which he characterizes in terms of Hofstede’s dimensions of “power-distance” “masculinity–femininity”, “uncertainty avoidance” and “collectivism”. Cultural values that express support and reverence for education and “translate” into a high appreciation of education and a high status for teachers are often seen as explanatory for the success in education of countries like Japan and Finland. The institutional infrastructure. New institutional economics stresses the importance of institutions. Institutions not just in the sense of organizational provisions, but also seen as “the rules of the game”, North (1990) defines institutions as the broad complexes of rules and norms that govern the behaviour of individuals, organizations and firms. They include legal, political and regulatory systems, the policing mechanisms for enforcing laws, financial and insurance systems, a free press, the market system and all the rules and formal and informal enforcement mechanisms that operate within it. (McMeeking, 2003, 11). The degree to which education in a country has become institutionalized is likely to depend on the institutionalization of the society at large. And the rights and liberties of teachers, for example, are likely to conform to those of personnel in the public service at large. Alternately the eventual lack of institutionalization in the society at large could also be reflected in the education system, and this could bear upon important disciplinary issues like the degree to which teacher absenteeism is tolerated. The general health situation in a country. When relative large parts of the population suffer from illnesses, like HIV/aids, this will clearly have a detrimental effect on the functioning of education as well. Disasters of nature and war. Finally there is the impact of disasters of nature, for example floods that happen more frequently as the probable consequence of global warming, and of countries being in a state of war. The table below summarises these types of societal conditions. Table 10 Types of societal conditions particularly relevant to education

47

Contextual conditions of education systems

. Demographic developments

. The labour market, e.g. shortages and surplus in certain sectors

. The general state of the economy

. Relevant cultural aspects

. The institutional infrastructure

. The general health situation in a country

. Disasters of nature and war

Antecedent conditions at national level within the education sector

Some of the societal dimensions mentioned in the section above have specific translations to conditions within the educational system. An overview of areas relevant for description and indicator development is given in the table below.

48

Table 11: Antecedent conditions within the educational system

Demographics

• the supply and demand of teachers in a country • the proportion of teachers over 50 years old • the gender composition of the teacher force per school level • percentage of students in school outside the age ranges for grade levels

Cultural aspects

• the status of teachers as perceived by the general public • appreciation of education and being educated • expectations about pedagogical functions of the school (e.g. educating for good

citizenship, moral education, teaching democracy) • cultural embedded interpretations relative to authority and educational leadership

Institutional infrastructure

• degree of formalization of teacher working conditions • formalization of teacher, student and parent rights (e.g. free school choice) • formal monitoring and inspection of schools • rules and enforcement of rules with respect to teacher absenteeism • regulations with respect to private tuition by teachers in the public service • anti corruption measures in education • framework for delivering and assessing the curriculum

Context indicators at the level of the local community In earlier sections the more active involvement of the community has already been referred to, by mentioning community involvement and parental involvement with a particular school. Again there is a vague line that separate theses “malleable conditions” from other conditions that are considered as given. As major categories to classify contextual conditions in the local school setting organizational infrastructure and (local) culture will be used.

49

With respect to the organizational infrastructure the factors summarized in the table below (Table 12) appear to be relevant: Table 12: The organisational infrastructure of the local community

The organisational infrastructure of the local community

- the existence of a school board in which the local community is represented - the availability of a local or regional educational resources centre (which, among others,

might offer ICT facilities to the schools in the community) - the “openness” of local companies and industry to work with schools and offer students

opportunities for site visits or specific training opportunities - the role of the community in financing the school; in-kind support

Cultural aspects that are manifest in the local community, are likely to reflect regional, national or even “world cultural” traditions. It is therefore somewhat arbitrary to deal with these conditions at national or local level. The reason to do so here, is that the local level is the one closest to the school, and the aspects to be dealt with are seen primarily as contextual constraints on the functioning of schools. Fuller and Clarke, (1994) have distinguished different types of cultural constraints relative to the effective functioning of schools. The first category of “local cultural conditions” distinguished by Fuller and Clarke concerns “prior effects of family processes an current effects from labor demands placed upon children”. They mention a range of variables that appear to be quite plausible for their relevance in developing countries. Examples are: ethnic membership, perceived legitimacy of the school by the parents, commitment of parents to literacy, willingness of parents to engage in modern child health and nutritional practices, gender, and distance to school. The second type of local condition treated by Fuller and Clarke pertains to the correspondence or rather the distinction between indigenous knowledge and “school knowledge”. They refer to the common finding that knowledge that is to a greater degree exclusively dealt with at schools usually shows larger effect of schools and school conditions as compared to knowledge that is also acquired at home As a third category, Fuller and Clarke refer to discrepancies between “participatory forms of pedagogy” which “Western policy mechanics” “would like” to show as being effective regardless of cultural setting. The problem they discern is that participatory forms of teaching may be at odds with hierarchical forms of teacher authority that coincide with cultural traditions.

50

The “culturally constructed meaning of classroom inputs”, which is the fourth type of local condition discussed by Fuller and Clarke, relates to the phenomenon that inputs of schooling, such as textbooks are interpreted and adapted in the ways they are actually used by teachers. This phenomenon is also well known in “Western” curriculum studies, where the distinction between an “intended” an “implemented” curriculum has similar implications. The additional element in developing countries being the interaction of traditional culturally embedded ways of teaching with the classroom inputs concerned. It makes sense to expect that these “mediations” of school inputs will be stronger to the degree that the intended use differs from the established teaching traditions. Fuller and Clarke provide examples from developing and industrialized countries of the pervasiveness of preferred ways of teaching e.g. expository teaching, despite the explicit hints and guidelines in textbooks to practice discovery learning. An example from education production function research in industrialized countries that also illustrate the different meanings and implications of school inputs in different cultures is the difference in the way class size is dealt with by teachers in Japan as compared to the US (Galton, 1999). In our quality framework the meaning of these cultural conditions, manifest in the local school community, is that they are likely to interfere with active measures to enhance school effectiveness, which might be inspired by an international common understanding of “what works” in education. The four types of cultural constraint, distinguished by Fuller and Clarke are summarized in Table 13, below. Table 13: Overview of local cultural conditions as examples of given local contextual conditions.

Local cultural conditions

- parents’ values concerning school participation of their children

- discrepancy between indigenous knowledge and “school knowledge”

- discrepancy between local perspectives on authority and ideas on active participation of

students during lessons

- culturally constructed meaning of school inputs

Mechanisms of responsiveness at system and at school level In our quality framework “context” has been interpreted as “provider of direct influence and control”, as a “provider of inputs” and as a source of more general “constraints” that interfere and interact with more direct control measures. In all of

51

these interpretations the direction of influence is from the context to the educational level, be it the school or the national educational system, that is being described or analyzed for its quality. However, in one of the quality perspectives that were described in the chapter, the direction is the other way around. According to the responsiveness perspective schools or national educational systems attempt to meet the demands of the relevant context, the local community or society at large. This means generally two things, first, that the intended outcomes or the goals and objectives are in line with the external expectations and second, that these goals are also actually being realized. The latter issue is the effectiveness issue, while the former is properly addressing the question of the responsiveness of educational organizations. The key question in the responsiveness interpretation of educational quality is therefore whether the “right” goals are chosen as a first step to delivering what is externally required. This is not a simple matter. In the most straightforward application, the issue of vocational education meeting the demands of the labour market, the complexity of the issue becomes painfully clear. Methods that seek to make explicit the skills and knowledge for a particular field of application and use such job profiles as a basis for establishing curriculum goals are indicated as “naïve”. For our purposes the main question is whether an educational system has an infrastructure and established mechanisms to deal with responsiveness questions. Not only with respect to the demands of the labour market, but also with respect to other kinds of societal demands to education, as for example developing good citizenship. At national level this focuses the attention on the availability of an institutional infrastructure for curriculum development, including procedures for controlling the actual implementation of nationally agreed curriculum priorities. In order to fulfill the responsiveness function well, such a curriculum development infrastructure should have established liaison functions with industry and other societal organizations. Sometimes the design of the school structure can include conditions that facilitate responsiveness, as in the case of so-called dual vocational education (forms of education where the students spent part of their school time in companies). At school level, an active approach of school management to maintain contacts with relevant local organizations, as well as with parents, could be seen in line with the responsiveness criterion. Areas for describing “responsiveness to context” at system and school level are summarized in Table 14 below.

52

Table 14: Areas of responsiveness to context at system and school level

Areas for describing responsiveness to context at system level

- the availability of an institutional infrastructure for curriculum development - enforcement mechanisms that monitor curriculum development and implementation - liaison functions of educational authorities and societal organizations - analysis and research units that try to predict the demands of the labor market - dual systems in vocational education

Areas for describing responsiveness of the school towards the local community

- external contacts of school management - “school marketing policies” - active role of the school in acquiring parental involvement - “authentic” teaching examples involving representatives from the local community

Summary and conclusion In this chapter the well-known input-process-outcome-context framework was used to define different perspectives on educational quality: productivity, effectiveness, efficiency, equity, responsiveness and a more eclectic use of quality indicators. The framework was critically analyzed with respect to its dynamic application of planning and controlling education. The flexibility and broadness of the framework was documented by describing modifications of the “rationality model” that increasingly address the complexity, situational contingency and interactive nature of social reality. The fact that the basic categories of the framework stand up as being useful descriptive categories for these modified interpretations as well was taken as a further encouragement to the use of the framework as a categorization scheme for indicators. Input, process, outcome and context education indicators that were further described and specified in the remaining part of the chapter. The indicator set has been summarized in the Table 14 below.

53

Table 15: Synthetic overview of educational input, process, outcome and context indicators

References Argyris, C., & Schön, D.A. (1974). Theory in practice: increasing professional

effectiveness. San Francisco: Jossey-Bass.

System level financial and material (see Table 7) and human resources (see Table 8) indicators

School level financial and material (see Table 7) and human resources (see Table 8) indicators

Student background characteristics (See Table 9)

System level process indicators See Table 4

Output indicators • subject matter based • literacy (reading,

mathematical, scientific) • competencies (e.g. learning

to learn

Outcome/attainment indicators

• graduation rates • proportion of students

graduated without delay • drop-out rates • class repetition rates

Impact indicators • (for each attainment level) %

of employed at a certain job level

• % of unemployed • (for lower school levels) %

enrolled in follow-up education

• degree of social participation (social capital)

• adult literacy rates • average income, for each

attainment level

Contextual factors Antecedent conditions Context indicators of

The societal context at national level the local community • demographic information • demographics • the organizational infrastructure • aspects of the culture and cultural • cultural aspects • local cultural conditions tradition of a country • economic aspects institutional infrastructure (see Tables 12 and 13) • the institutional infrastructure (see Table 11) • the general health situation in a country • disasters of nature and war (Table 10)

INPUTS PROCESS OUTCOMES

School level process indicators See Table 5

Effective teaching variables See Table 6

Areas for describing responsiveness See Table 14

54

Bosker, R.J., & Velden, R.K.W. van der (1989). Schooleffecten en rendementen. In: J. Dronkers & J. van Damme (Eds.), Loopbanen doorheen het onderwijs. Lisse: Swets & Zeitinger.

Bosker, R.J., & Witziers, B. (1996). The magnitude of school effects. Or: Does it really matter which school a student attend. Paper presented at AERA Annual meeting, New York.

Brandsma & Doolaard (1996) Cameron, K.S., Whetten, D.A. (eds.) (1983). Organizational Effectiveness. A

comparison of multiple models. New York: Academic Press. Campbell, D.T. (1969). Reforms as Experiments. American Psychologist, 24(4). Cheng, Y.C. (1993). Conceptualization and measurement of school effectiveness: An

organizational perspective. Paper presented at AERA annual meeting, Atlanta, GA.

Cohen, M.D., March, J.G. & Olsen, J.P. (1972). A garbage can model of organizational choice. Administrative Science Quarterly, 17, 1-25

Ezemenari, K., Rudqvist, A., & Subbarao, K. (1998). Impact evaluation: a note on concepts and methods. PRMPO World Bank, Washington D.C.

Fuller, B., & Clarke, P. (1994). Raising school effects while ignoring culture? Local conditions and the influence of classroom tools, rules and pedagogy. Review of Educational Research, 64, 119-157.

Galton, M. (1999). Class Size and Pupil Achievement. International Journal of Educational Research, vol. 29.

Goldstein, H. (1996). Multilevel statistical models. London: Arnold. Goldstein, H., & Thomas, S. (1996). Using examination results as indicators of school

and college performance. Journal of the Royal Statistical Society. Series A , Statistics in Society, vol. 159, no. 1, 149-164.

Hallinger 2001 Haste, H. (2001) Ambiguity, Autonomy, and Agency: Psychological Challenges to

New Competence. In: Rychen. D.S., & Hersch Salganik, L. Defining and selecting Key Competencies. Seattle, Toronto, Bern Goettingen: Hgrefe & Huber Publishers.

Hausman, J.A., & Wise, D.A. (1979). Attrition bias in experimental and panel data: the Gary Income Maintenance Experiment. Econometrica, 47, 455-473.

Hirsch, D. (1994). School: A matter of choice. Paris: OECD/CERI. Hofstede ,P., (1980) Culture’s consequences: International differences in work-

related values. Beverly Hills, CA:Sage Hoogerwerf, A. (1978) Overheidsbeleid (Public Policy). Alphen aan den Rijn:

Samson Leune, J.M.G. (1994). Onderwijskwaliteit en de autonomie van scholen. In B.P.M.

Creemers (red). Deregulering en de kwaliteit van het onderwijs. (pp. 27-48) Groningen: RION.

55

Levy, F., & Murnane, R.J. (2001) Key Competencies Critical to Economic Success. In: Rychen. D.S., & Hersch Salganik, L. Defining and selecting Key Competencies. Seattle, Toronto, Bern Goettingen: Hgrefe & Huber Publishers.

Lockheed, M.E. (1988). The measurement of educational efficiency and effectiveness. AERA paper. New Orleans.

Luhmann, N. (1995). Social systems. Stanford, Ca.: Stanford University Press. Madaus, G.F., Kellaghan, Th., Rakow, E.A., & King, D.J. (1979). The sensitivity of

measures of school effectiveness. Harvard Educational Review, 49, 207-230. March, J.G. & Simon. H. A. (1958) Organizations Jon Wiley: New York McMeeking, R. M., (2003) Incentives to improve education. Cheltenham UK,

Northampton MA, USA: Edward Elgar Meuret, D. & J. Scheerens (1995). An International Comparison of Functional and

Territorial Decentralization of Public Educational Systems. Paper presented at AERA 1995, San Francisco.

Morgan, G. (1986). Images of organizations. Beverly Hills: Sage. Niskanen, W.A. (1971). Bureaucracy and representative government. Chicago: Aldine-

Atherton. North, D.C., (1990) Institutions, Institutional Change and Economic Performance.

Cambridge UK: Cambridge University Press OECD (1998). Education at a Glance. Paris: OECD. OECD (2001). Knowledge and Skills for Life. First results from PISA 2000. Paris:

OECD. Purkey, S.C., & Smith, M.S. (1983). Effective schools: a review. The Elementary

School Journal, 83(4), 427-452. Raudenbush, S.W., & Willms, J.D. (1996). The estimation of school effects. Journal

of Educational and Behavioural Statistics, 20, 307-335. Reneman, D., (2000) Self-Reference and Policy Success Dissertation: University of

Utrecht, the Netherlands Riley, D.D. (1990). Should market forces control educational decision making?

American Political Science Review, 84, 554-558. Rutter, M., Maughan, B., Mortimore, P., & Ouston, J. (1979). Fifteen thousand hours.

London: Open Books. Rychen, D.S. (2001) Introduction. In: Rychen. D.S., and Hersch Salganik, L. Defining

and selecting Key Competencies. Seattle, Toronto, Bern Goettingen: Hgrefe & Huber Publishers.

Scheerens, J. (1990). School effectiveness research and the development of process indicators of school functioning. School Effectiveness and School Improvement,1(1), 61-80. Lisse: Swets & Zeitlinger.

Scheerens, J. (1992). Effective Schooling, Research, Theory and Practice. London: Cassell.

Scheerens, J. (2000). Improving school effectiveness. Paris: UNESCO, IIEP, Fundamentals of Educational Planning series no. 68.

56

Scheerens, J., & Bosker, R.J. (1997). The Foundations of Educational Effectiveness.Oxford: Elsevier Science Ltd.

Scheerens et al 2002 bequad? Simon, H. A. (1945) Administrative Behavior. New York: Mc. Millan Weinert, F.E. (2001). Concept of competence: a conceptual clarification.. In: D.S.

Rychen, & L.H. Salganik (Eds.), Defining and selecting competencies, 45-64. Seattle: Hogrefe & Huber Publishers.