tec 2013 hyderabad the role of programme evaluation in ... · potential in curriculum and...

TEC 2013 Hyderabad

The role of programme evaluation in curriculum development

OR‘How are we doing?’

Richard KielyUniversity of St Mark & St John, UK

My life in programme evaluation

1. Medium of instruction evaluation (Pupil)

2. Native‐speaker teacher evaluation (Teacher)

3. Evaluation for development consultancy (Consultant)

4. International CLIL project (Evaluator)

5. Teacher education programme impact study (Researcher)

Some principles

1. Evaluation cannot just test the theory

2. Evaluation has to understand learning processes as well as outcomes

3. Evaluation has to engage with all stakeholders

4. Evaluation has to facilitate action

5. Evaluation should make sense of the programme for everyone.

This presentation

• Overview of language programme evaluation and its potential in curriculum and professional development.

• Evaluation purposes, designs and methods

• Identities and roles of programme stakeholders

• Using the processes and outcomes of evaluation for improvement of programmes, and

• Management of change issues.

Overview

• Programme evaluation is ‘a set of strategies to document and understand the programme. It involves research activity (conventional studies or action research by which teachers learn about and transform aspects of their practice) and assessment data (conventional measures of outcomes). In addition to these, evaluation has to engage with the social, cultural and historical identity of the programme, as a product of the institution, as a phase in the biographies of participants, and as a context of personal investments of individual stakeholders’.

Kiely 2009: 114

Programme Evaluation

TYPE 1 TYPE 2

Experimental/comparative

Case study

Probability Possibility

Policy‐oriented Practice‐oriented

Programme theory (explicit)

Programme theory (implicit)

Product‐focus Process‐focus

External Internal

Programme Evaluation

TYPE 1 TYPE 2

Experimental/comparative

Case study

Probability Possibility

Policy‐oriented Practice‐oriented

Programme theory (explicit)

Programme theory (implicit)

Product‐focus Process‐focus

External Internal

Discuss: Any of these

features new to you?

Evaluation purposes, designs and methods

• Purposes: accountability, development and quality assurance

• Designs: the links between data, theory and action

• Methods: ways of getting data.

Evaluation designs – Type 1Template 1 Template 2

Measurement of outcomes (language

tests; teacher performance; teacher

qualifications;

Surveys of attitudes; preferences; aspirations;

wants; needs

Evaluation designs – Type 2Template 1 Template 2 Template 3

Quality Assurance

Programme support

Teacherresearch

Evaluation designs – Type 2Template 1 Template 2 Template 3

Quality Assurance

Programme support

Teacherresearch

Buzz group discussion:What kind of activities do you think would contribute to these designs?

Evaluation designs – Type 2

Template 1 Template 2 Template 3

Quality Assurance

Programme support

Teacher research

End of course questionnaires

Mentoring Action research

Surveys Peer observation Reflective Practice

Focus Groups Reading circles Exploratory practice

Audits Assessment workshops

Study for research degrees

Assessment data Meetings

Evaluation designs

Template 1 Template 2 Template 3

Quality Assurance

Programme support

Teacher research

Design issuesHow do we find the right amount of activity?How do we get a focus on learning?How do we add value to learning opportunities?

Identities and roles of programme stakeholders

Teachers


Students

Teachers


Students

Teachers

Teacher educators


Students

Teachers

Teacher educators

Leaders andmanagers


Students

Teachers

Teacher educators

Leaders andmanagers

Remote stakeholders

Identities and roles

Immediate stakeholders Remote stakeholders

Students Teachers ManagersCustomers Transformers Sponsors Learners Transmitters ParentsParticipants Advisors EmployersPractitioners Assessors

Student role ‐ learning

• Using the processes and outcomes of evaluation for improvement of programmes

WORKSHOP


• Using the processes and outcomes of evaluation for improvement of programmes

WORKSHOP

Buzz group discussion:

How can teachers workshop

evaluation process with students?


• Processes

• Awareness raising• Focus groups• Structured discussions• Reflections as part of assessment


• Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta‐communicative discourse – all these are considered indispensable learner activities in ELT.

Legutke and Thomas (1991:65)




Autonomy




Autonomy

Agency




Autonomy

Agency

Motivation

Using the processes and outcomes of evaluation for improvement of programmes

Teachers in a community of practice


• Collaborative development (peer observation; mentoring; coaching)

• Workshops on materials; ICT; assessment formats; test data


• Collaborative development (peer observation; mentoring; coaching)

• Workshops on materials; ICT; assessment formats; test data

Buzz group discussion:Do these happen in your context? If not,

why not?

Teacher role – professional learning

• Sponsored professionalism: based on qualification and recognition

• Independent professionalism: ‘a commitment to careful and critical examination of the assumptions and practices embedded in sponsored professionalism’

(Leung 2009:53).

Teachers and Change

• Int: So after the evaluation […..] Will you do it differently next time?

• Millie: I don’t think I’ll do it differently. […] we have developed some good strategies and some quite good materials, that will continue to develop. So it is not a change in direction, but perhaps going further in the same direction.

Kiely 1998: 194

Teachers and Change (Kiely 2001:257)Feedback Resistance Reflection Innovation

CYCLE 1Anna gets feedback from students suggesting less attention to explaining words in class

CYCLE 2

Teachers and Change (Kiely 2001:257)



Anna interprets this as selfish thinking on the part of students

CYCLE 2





Anna rationalises that this might not be the best way of using classroom time

CYCLE 2






Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words

CYCLE 2







CYCLE 2Anna gets feedback from students suggesting more attention to explaining words in class








Anna resists suggestions that her focus on ‘broad swathes of meaning’should change









Anna:‘This group say they want it, so I try to do it for them’, and includes a short activity in Week 6









Anna:‘This group say they want it, so I try to do it for them’, and includes a short activity in Week 6

Anna spends more time on vocabulary in Weeks 9 & 10, and provides vocabulary tasks to texts in Weeks 11 & 12


Teachers and Change

• Change is gradual, perhaps not always visible to the naked eye;

• Change occurs at the teacher’s pace;

• Change is negotiated;

• Change happens.

Key guidelines from principles

1. Evaluation cannot just test the theory

2. Evaluation has to understand learning processes as well as outcomes

3. Evaluation has to engage with all stakeholders

4. Evaluation has to facilitate action

5. Evaluation should make sense of the programme for everyone.

Use evaluation to ….

1. …. explain what is going on

2. … answer the ‘How are we doing?’ question

3. … allow all voices to be heard

4. … act for improvement

5. … raise awareness and explain.

ReferencesKiely, R. (2012) Designing evaluation into change management processes. Overview chapter in Tribble, C. (Ed) Managing Change in Language Education. London: The British Council pp 75‐91

Kiely, R. & P. Rea‐Dickins (2009) Evaluation and learning in language programmes. In Knapp, K. and B. Seidlhofer with H. Widdowson (eds) Handbooks of Applied Linguistics: Volume 6: Handbook of foreign language communication and learning. Amsterdam: Mouton de Gruyter. Pp. 663‐694

Kiely, R. (2011) Understanding CLIL as an innovation. Studies in Second Language Learning and Teaching. Vol 1, No 1, pp 153‐71.

http://ssllt.amu.edu.pl/images/stories/volume.1/SSLLT_11_153‐171_Kiely.pdf

Kiely, R. (2009) Small answers to the big question: Learning from language programme evaluation. Language Teaching Research Vol 13, No 1: 99‐116

Kiely, R. (2006) Evaluation, innovation and ownership in language programs. Modern Language Journal, Vol 90, No 3 pp: 597‐602

Kiely, R. & P. Rea‐Dickins (2005) Program Evaluation in Language Education. Basingstoke: Palgrave Macmillan (Series editors Chris Candlin and David Hall) [Second edition – 2013 – in preparation].

Legutke, M. & H. Thomas (1991) Process and Experience in the Language Classroom. Harlow: Longman

Thank you

[email protected]

tec 2013 hyderabad the role of programme evaluation in ... · potential in curriculum and...

Documents