save us from the time of trial

Centre for Educational Evaluation (CEE), DCU

NIJESH SOLK MWIL

START GYAMYATH (Save us from the time of trial)

From Austerity to Prosperity and back again

ESAI CONFERENCE 2013

M.Brown, G.McNamara, J.O’Hara


FOCUS OF THE STUDY

This study examined the changing landscape of evaluation policy and practice in the ROI and NI since the inception of the Stanley letter 1831 where evaluation in the form of school inspection and self evaluation aims to promote and by turn of phrase improve the quality of education provided in schools.

From this, a framework for the mutual terms of co-existence between IE and EE is proposed.

Theoretical Assumption

‘Both systems should exist because we need both and because they might even benefit from each other’

(Nevo p.6, 2002)


Phase 1 :- Tentative Schemata

Review of the literature on the rise of evaluation in education and the different international models of evaluation that exist, coupled with an analysis of principal and inspector interviews in the 19th and 20th century.

Phase 2: - Quantitative Analysis

All Island survey of school Principals perceptions of internal/external evaluation in the Republic of Ireland and Northern Ireland. Analysis was carried out using parametric and non parametric techniques.

Phase 3: - Qualitative Analysis

Interviews with 46 Principals and Inspectors from both regions.

Phase 4: - Final analysis (March 2013)

Proposed Framework: for the mutual terms of co-existence between Internal and External Evaluation.

OUTLINE OF THE STUDY: MAY 2009 - MARCH 2013

A mixed methods way of thinking rests on assumptions that

there are multiple legitimate

approaches to social inquiry and that any given approach to

social inquiry is inevitably partial

(Greene, 2007 p.20).


Initial findings: Principals perceptions of Internal Evaluation


ASSESSMENT (THE PAST)


ASSESSMENT (THE PRESENT)


VALUE ADDED ASSESSMENT(THE GLOBALISED FUTURE)

OECD REVIEW OF EVALUATION AND ASSESSMENT (Australia)

The development of “value-added” models represents significant progress as they are designed to control for the individual student’s previous results, and therefore have the potential to identify the contribution an individual teacher made to a student’s achievement. (OECD p.96, 2011)

OECD REVIEW OF EVALUATION AND ASSESSMENT (Portugal)

In value-added models, students’ actual test scores are often compared to the projected scores, and classroom and school scores that exceed the projected values are considered as positive evidence of instructional effectiveness. (OECD p.132, 2012)

OECD REVIEW OF EVALUATION AND ASSESSMENT

Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Portugal, Slovak Republic, Sweden and the United Kingdom (Northern Ireland).


JUNIOR CYCLE FRAMEWORKThe DES will provide each school with a Data Profile…The Data Profile will also provide schools with information on their patterns of achievement relative to schools with a similar school context…These data will help schools to refine their assessment and moderation practice. They will also be a valuable source of information for schools’ self-evaluation processes…In the event of an unusual pattern of achievement, the Inspectorate of the DES will be advised, and support and evaluation measures will be provided for the school (DESROI, 2012a, p.27).

VALUE ADDED ASSESSMENT THE IRISH FUTURE


SELF EVALUATION: THE PRESENT

Over a four-year period from 2012, all post-primary schools should engage in school self-evaluation and produce three-year improvement plans for numeracy, literacy and one aspect of teaching and learning across all subjects and programmes (Department of Education and Skills 2012, p.2).

The performative culture is so deeply ingrained in schools and education systems that I can foresee a game of permanent artifice, where schools squeeze their individual circumstances into a self-evaluation document designed solely to impress inspectors, and hold themselves in a state of perpetual readiness to live up to their claims, the model prisoner. In this context, ‘bleak indeed is the desire for perfection’ (Marshall 1999, 310)’(Perryman 2009, p.629)


OUR FRAMEWORK FOR THE CO-EXISTENCE OF INTERNAL AND EXTERNAL EVALUATION

AT A CONCEPTUAL LEVEL Evaluation needs to be perceived as a means of understanding rather than judgement’ (Nevo 2002, p.10).

AT A METHODOLOGICAL LEVELEvaluation should be based on the continuous dissemination of information, enabling mutual learning. However, Ryan et al (2007, p.208) ask ‘what should we expect from novice, school-based evaluators? Should we have the same standards and expectations for the school teams (i.e., internal evaluators) as we do for evaluators conducting external evaluations?’

AT A COMMUNICATION LEVELThere needs to be mutual respect and trust between internal and external evaluators. However, within the social fabric of many regions, such as the ROI, where the economic downturn has led to cut backs in education, it would be reasonable to suggest that very little trust in government policy and practice in general actually exists.


OUR FRAMEWORK FOR THE CO-EXISTENCE OF EVALUATION

AT AN INFLUENTIAL LEVELIn the past, many Evaluation systems tended towards, Reward, Coercive and Legitimate power.

Table 2.3:Ravens (1965) typology for the six bases of power Positional vs transformational bases of power

POWER BASE

POSITIONAL

DEPENDENT ON CHANGE

IN TARGETS INTERNAL BELIEFS

REQUIRES SURVEILLANCE

Reward Yes No Yes

Coercive Yes No Yes

Legitimate power Yes No Yes

Referent power No Yes No

Expert power No Yes No

Informational power No Yes No

However, in order for internal and external evaluation to mutually and beneficilally co-exist:Evaluation frameworks should centre on the Referent, Expert and Informational aspect of Evaluation.


USE OF THE INFLUENTIAL ASPECT OF EVALUATION

During my time as Principal the emphasis was on schools evaluating themselves with the Inspector as the "critical friend“ [Expert Power]. For the approach to be successful it required a good working relationship between the Principal and the District Inspector… [Referent Power]. By meeting frequently we grew to trust and respect one another with the result that I never felt threatened or worried when [Dr.Shevlin] (DI) would ask probing questions… With [Dr. Shevlins] support we tried new approaches to things like literacy, discipline, behaviour and then wrote up our policies [Informational Power]. It was this professional relationship between the Inspector and the Principal which greatly assisted school improvement in the case of St Colm's…

(Hansard Archive, Northern Ireland Assembly, 2011)


Evaluator Influence(St. Colm’s High School

Belfast)


AT A CULTURAL RESPONSIVE LEVEL

I can find no logical explanation as to why our evaluations should not be culturally responsive or that we should not behave in culturally responsible ways in our work as evaluators. (Hood, 2001)

Indeed:Evaluation efforts have typically failed to consider cultural background and context in their design, implementation, analyses, and recommendations.

However:It is not possible to effectively derive evaluative meaning from educational programs, designed to serve culturally diverse students, unless the evaluator themselves are more culturally responsive. (Hood, 2001)



ASSESSMENTEvery time you use a measure of pupil attainment for some extraneous purpose you risk creating a perverse incentive. So every time you find yourself doing that or participating in a system that requires you to do that; ask what incentive is this creating? Whom is it damaging? Pupils, head teachers, schools, employers and I think that would be a very useful first step (O’Neill, 2011).

EVALUATIONThose of us who are proponents of external evaluation should find ways to empower schools and teachers to participate as equal partners in the evaluation process and make use of it; and those of us who believe in internal evaluation as a means for school autonomy and teacher professionalisation must admit the legitimacy of accountability and the right of the public to know. They, in their turn, should seek external evaluation as a partner for dialog rather than an object for rejection. (Nevo 2010, p.784)

CONCLUSION


RegionDisagree Strongly

Disagree Indifferent AgreeAgree

StronglySouth Count 9 58 64 210 22

% 2.5% 16.0% 17.6% 57.9% 6.1%North Count 1 17 25 52 4

% 1.0% 17.2% 25.3% 52.5% 4.0%U(1) = 16805.500, Z =-1.100 , p = .271Table 4.4.7: External evaluation results in better teaching and learning

RegionDisagree Strongly

Disagree Indifferent AgreeAgree

StronglySouth Count 16 57 78 187 25

% 4.4% 15.7% 21.5% 51.5% 6.9%

North

Count 1 17 23 53 5% 1.0% 17.2% 23.2% 53.5% 5.1%

U(1) = 17885.500 , Z = -.077 , p =.939

Table 4.4.3: External evaluation results in better Management

Initial findings: Principals perceptions of External Evaluation

save us from the time of trial

Documents