EVALUATION FRAMEWORK &

APPLICATION Group B

Alice Sheridan

Katherine Holland

Beatrice Segura Harvey

WHAT WE DID

The reading •  1970s to 2008

•  High frequency words

•  Checklists and criteria

•  Categorising and grouping

•  Gathering quantitative and qualitative data

•  Types of questions to gather data

•  Clarity and flexibility within the framework

Developing the framework •  Accidentally devised a checklist

that is similar to 1990s frameworks.

•  Everyone is different.

•  Macro and micro environment

•  Checking each question is an evaluation question.

•  Used categories and questions.

Applying the framework •  Applied the same framework to

unit 6 of the chosen textbook.

•  Worked separately

•  Compared our results

•  Compared our choice of weightings for each category.

•  Discussed the limitations we found when applying the framework.

WEIGHTING EACH CATEGORY •  Difference of opinion

•  Different experience

•  Different contexts (present and past)

•  Different institutions

•  Different class demographics and dynamics

•  Different teaching preferences and styles

•  Different approaches and methods

Pedagogic approach

Flexibility

Topic content

Syllabus

Teachability

Voice and instruction Design

and illustration/ images

Reading and

listening texts

SLA research

Extra resources

WHY USE QUALITATIVE AND QUANTITATIVE DATA

•  The quantitive data was an attempt to •  reduce the amount of details in the process •  Offer an holistic overview of the evaluation (in categories, as a whole and as a

way to deduce some understanding of the evaluators prioritisation of the categories)

•  The qualitative data was generated to attempt to •  Offer detailed insight into areas of the evaluation. •  Highlight key areas, points.

APPLICATION TO THE COURSE BOOK

70% sufficient 62% sufficient

Category Alice Beatrice Katherine Pedagogic Approach 12% 12% 32% Flexibility 10% 10% 16% Syllabus 8% 12% 7.5% Topic Content 5% 15% 10% Voice and Instructions 3% 5% 4% Teachability 17% 15% 7.5% Design and Illustrations/ Images 15% 10% 4% Reading and Listening tasks 10% 10% 7.5% SLA research 2% 1% 7.5% Extra resources 18% 10% 4%

64% sufficient

0%

5%

10%

15%

20%

25%

30%

35%

Pedagogic Approach

Flexibility Syllabus Topic Content Voice and Istructions

Teachability Design and Illustrations/

Images

Reading and Listening tasks

SLA research Extra resources

We

ght

in %

Evaluation category

Distribution of weightings per evaluator

Alice

Beatrice

Katherine

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Pedagogic Approach

Flexibility Syllabus Topic Content Voice and Istructions

Teachability Design and Illustrations/

Images

Reading and Listening tasks

SLA research Extra resources

Suffi

cie

ncy

ratin

g in

%

Evaluation category

Sufficiency rating total per category

Alice

Beatrice

Katherine

LIMITATIONS •  SLA considerations •  First draft of questions •  No first level analysis •  Didn't distinguish between analysis and evaluation questions •  Questions were to vague and large •  No reliability and validity testing – would it even be possible? •  Words/questions are subject to interpretation •  Too many categories and questions •  Could an inexperienced evaluator use this framework? •  No retrospective or in use evaluation has been built into the framework – not possible

to triangulate data. •  No internal and external evaluation •  No trailing and user feedback taken into consideration

WHAT WE LEARNT FROM THE PROCESS

"We are swamped in a sea of details"

Cunningsworth, A. (1995), Choosing Your Coursebook. Oxford : Heinemann.

REFERENCES Chambers, F. (1997) Seeking consensus in coursebook evaluation ELT Journal 51 (1): pp.29-35.

Ellis, R. (1997) The empirical evaluation of language teaching materials. ELT Journal 51 (1): pp.36-42.

Johnson, K., et al. (2008) A step forward: investigating expertise in materials evaluation. ELT Journal 62 (2): pp.157-163.

Littlejohn, A. (2011) The analysis of language teaching materials: inside the Trojan Horse. In: Tomlinson, B. (ed). Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press. pp.179-211.

Masuhara, H. (2011) What do teachers really want from coursebooks? In: Tomlinson, B. (ed). Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press. pp.236-266.

Masuhara, H., et al. (2008 ) Adult EFL courses. ELT Journal 62 (3): pp.294-312.

McDonough, J., Shaw, C. & Masuhara, H. (2013) Materials and Methods in ELT: A Teacher's Guide. 3rd ed. Oxford: Wiley-Blackwell.

McGrath, I. (2002) Materials Evaluation and Design for Language Teaching. Edinburgh: Edinburgh University Press.

McGrath, I. (2013) Teaching Materials and the Roles of EFL/ESL Teachers: Practice and Theory. London: Bloomsbury.

Mukundan, J. & Touran, A. (2013) A Review of Textbook Evaluation Checklists across Four Decades (1970-2008)In: Tomlinson, B. & Masuhara, H. (ed). Research for Materials Development in Language Learning. London: Bloomsbury.

Pryor, S. (2013) The Developments and Trialling of Materials for Second Language Instruction: A Case Study. In: Tomlinson, B. & Masuhara, H. (ed). Research for Materials Development in Language Learning. London: Bloomsbury.

Roberts, J. T. (1996) Demystifying materials evaluation. System 24 (3): pp.375-389.

Tomlinson, B. (2011) Materials Development in Language Teaching. (2nd ed) Cambridge: Cambridge University Press.

Tomlinson, B. (2012) State-of-the-Art Article: Materials development for language learning and teaching. Language Teaching 45 (2)

Tomlinson, B. (2013) Developing principled frameworks for materials development. In: Tomlinson, B. (ed). Developing Materials for Language Teaching. London: Bloomsbury.