1 approaches to evaluation advisor: mavis shang kelly 9710001m dora 9710011m
Post on 20-Dec-2015
220 views
TRANSCRIPT
2
Approaches to evaluation* Curriculum evaluation:• Is the curriculum achieving its goals?• What is happening in classrooms and schools where it is
being implemented?• Who can affect the curriculum (e.g., teachers,
administrators, students, parents, employers) ?•Does the curriculum compare favorably with others of its
kind?
3
Approaches to evaluation
* Evaluation focuses on many different aspects :
• curriculum design• the syllabus and program content• classroom processes• materials of instruction• the teachers• teacher training• the students
4
Approaches to evaluation
* Evaluation may focus on many different aspects :
• monitoring of pupil progress• learner motivation• the institution• learning environment • staff development• decision making
5
Approaches to evaluation
* The scope of evaluation :
1. It focuses from test results to the need to collection information
2. make judgments about all aspects of the curriculum, from planning to implementation
6
Purposes of evaluation
*Weir and Roberts (1994):
• Program accountability :
to examine the effects of a program or a project
• Program development:
to improve the quality of a program or a project
7
Purposes of evaluation
1. Formative evaluation
To find out what is working well, and what is not,
and what problems need to be addressed
Ongoing development and improvement of the
program
Some typical questions (p. 288)
To address problems and to improve the program
8
Purposes of evaluation
1. Formative evaluation
Example 1:
* situation: task-oriented communicative methodology
* problem: teachers are resorting to a teacher-dominated
drill and practice mode of teaching during the
implementation
=> it can’t match with the course philosophy.
* solution: provide videos to model teaching strategies
9
Purposes of evaluation
1. Formative evaluation
Example 2:
* situation: to implement integrated skills
* problem: teachers spent time on different components
of the course and emphasize different things
=> a series of meeting review teachers’
understanding of the course objectives
* solution: peer observation as a way for teachers to
compare teaching styles and help them
10
Purposes of evaluation
1. Formative evaluation
Example 3:
* situation: implement conversation skills for low-level,
and the pronunciation is not a major element
of the course
* problem: students have serious pronunciation problem
(after four weeks)
* solution: refocus one section of the course which
includes pronunciation component
11
Purposes of evaluation
2. Illuminative evaluation
To find out the different aspects of the program and to
provide a deep understanding of the processes of the
teaching and learning without any change
Some typical questions (p. 289)
12
Purposes of evaluation
2. Illuminative evaluation
Example 1:
* situation:
1) teach many reading skills for students
2) teachers are interested in what is the students’
major focus of the course
* evaluated way: conduct a short questionnaire
=> understand students’ opinion and need
13
Purposes of evaluation
2. Illuminative evaluation
Example 2:
* situation: teacher is interested in teacher-student
interaction learning
* evaluated way: to invited colleague to do the
classroom observation => assess what happened
during the course
14
Purposes of evaluation
2. Illuminative evaluation
Example 3:
* situation: teacher wants to know how students carry
out group work and prepare students for
group-work task
* evaluated way: record and review the recordings
=> know what kind of strategy assigns for
members in a group
=> make sure students participate actively
15
Purposes of evaluation
3. Summative evaluation
To determine the effectiveness of a program, its
efficiency, and to some extent with its acceptability
Used “after” a program
Some typical questions (p. 292)
16
Purposes of evaluation Different measures of a course’s effectiveness and
each one can be used for different purposes
a) Mastery of objectives:
“How far have the objectiveness been achieved?”
ex: speaking skills
Objective: In group discussions students will listen to and
respond to the opinions of others in their group
Disadvantage: it does not provide the whole picture of
them effectiveness of a course
17
Purposes of evaluation b) Performance on tests:
◎ such tests might be unit texts given at the end of
each unit of teaching materials, quizzes or class
tests
◎ Weir: achievement have washback effect on
teaching
◎ help to make decision about needed changes
18
Purposes of evaluation c) Measures of acceptability:
◎ satisfactory achievement of the objectives and
good levels of performance on exit tests
◎ should be considered some factors: time-tabling,
class size, choice of materials, or teachers’
teaching styles
19
Purposes of evaluation
d) Retention rate or reenrollment rate:
◎ whether students continue in the course
◎ the percentage of students who reenroll for
another course at the end
20
Purposes of evaluation
e) Efficiency of the course:
◎ How to develop and implement the course efficiently
◎ some problems may occur during the course:
* the time (planning and course development),
* the needs (material and teacher training),
* the time needed (consultations and meetings)
21
Issues in Program Evaluation Weir and Robert (1994, 92) proposed:
a need for both insider and outsider commitment and involvement to ensure adequate evaluation
The demonstration of the “product value” of a program or project or their components
An associated commitment to a deeper professional understanding of the processes of educational change
22
Issues in Program Evaluation Weir and Robert (1994, 92) proposed:
Systematic documentation for evaluation purposes both during implementation and at the beginning and end of a program or project’s life
A willingness to embrace both qualitative and quantitative methodology appropriate to the purpose of the evaluation
23
Issues in Program Evaluation1.The audience for evaluation In planning an evaluation it is important to identify
(1) who are the audiences
(2) what kind of information they are most interested in
Ex: developing a new textbook (1) Officers in the ministry: money (2) Teachers: the sufficient materials (3) Outside consultant: the design of the materials, interaction and language practices (4) Vocational training centers: school leavers’ English
Evaluation has to satisfy all interested parties.
24
Issues in Program Evaluation1.The audience for evaluation (questions)
Students:
(1)What did I learn?
(2)How will the course help me in the future?
Teachers:
(1) What did my students learned?
(2) How well did I teach?
Curriculum developers:
Is the design of the course and materials appropriate?
25
Issues in Program Evaluation
1.The audience for evaluation (questions)
Administrators:
(1) Was the time frame of the course appropriate?
(2) Were clients’ expectations met?
Sponsors:
(1) Was the cost of the course justified?
(2) Was the course well managed?
26
Issues in Program Evaluation
1.The audience for evaluation Three audiences are identifiable for all summative evaluation of language courses (Shaw and Dowsett, 1986):
other teachers in the program -- for course design and planning purposes managers of the institution or program -- determining course offerings and placement the curriculum support or development -- monitoring the curriculumIn planning an evaluation , we should carefully identified
different audience and the result should be appropriate for each audience
27
Issues in Program Evaluation
2.Participants in the evaluation process Two types of participants:
(1) insiders:
teachers, students, anyone else closely involved in the
program
Ex: formative evaluation: teachers
summative evaluation: students
Successful program evaluation are often involve key
insiders in the process of designing and carrying out
evaluation
28
Issues in Program Evaluation
2.Participants in the evaluation process Two types of participants:
(2) outsiders:
consultants, inspectors, or administrators provide teachers with some perceptions with independent observation and opinion
29
Issues in Program Evaluation
3.Quantitative and qualitative evaluation
Quantitative evaluation:
the measurement of something can be expressed
numerically
Quantitative data:
collect information from a large number of people and
analyze statistically
ex: checklists, surveys, self – rating Advantage: objective Disadvantage: some limitations
30
Issues in Program Evaluation
3.Quantitative and qualitative evaluation Qualitative evaluation:
1) something can not be expressed numerically
2) depend on subjective judgment or observation
=> collect information from classroom observation,
interviews, journal, logs, case studies… Merits: more holistic and naturalistic
Defects: less rigorous
31
Issues in Program Evaluation
4.The importance of documentation
Relevant documentation: (p.297)
1) Course statistics
2) Relevant course document
3) Course work
4) Written comment
5) Institutional documents
32
Issues in Program Evaluation
5.Implementation In order to make decisions based on the evaluation, it
is first necessary to review the process of evaluation
Questions : (p.298) 1) Scope 2) Audience 3) Reliability 4) Objectivity 5) Representativeness 6) Timeliness 7) Ethical considerations
33
Procedures used in conducting evaluations 1. Tests
1) institutionally prepared tests
2) international tests
3) textbook tests
4) student records Advantage:
provide a direct measure of achievement or performance
Disadvantages:
1) hard to make sure the tests are a direct of teaching or
are linked to other factors
=> further investigation
2) “reliability and validity” problems
34
Procedures used in conducting evaluations
2. Comparison of two approaches to a course
Comparative approach:
compare the effects of two or more different teaching
conditions Advantage:
control all relevant factors and investigate rigorously
Disadvantage:
impose artificial constraints on the teacher
35
Procedures used in conducting evaluations
3. Interviews:
a) could get many views on any aspect of the course
b) structured interview is more useful
Advantages: obtain in-depth information
Disadvantages: 1) time-consuming
2) the representativeness of their views
may be questionable
36
Procedures used in conducting evaluations
4. Questionnaire:
elicit teacher’s and students’ comment
Advantages: 1) easy to administer
2) representative information
Disadvantages: 1) it may elicit unbiased answers
2) difficult to interpret the information
37
Procedures used in conducting evaluations
5. Teachers’ written evaluation:
use structured feedback form Advantages: 1) good position to report on a course
2) provide information quickly
Disadvantages: information may be impressionistic and
biased
38
Procedures used in conducting evaluations
6. Diaries and journals
1) provide a narrative record of things
2) problems encountered
3) critical incidents
4) time allocation Advantages: provide relatively detailed and open-ended
information
Disadvantages: 1) impressionistic and unsystematic
2) need cooperation and a time commitment
39
Procedures used in conducting evaluations
7. Teachers’ records
a) reports of lessons taught b) material covered
c) attendance d) students’ grades e) time allocation Advantages:
provide a detailed account of some aspects of the course
Disadvantages:
1) some of the information will not be relevant
2) some information may be subjective (teachers’ view)
40
Procedures used in conducting evaluations
8. Student logs:
ask students to provide their opinion of the course Advantages:
1) provide students’ opinion of the course
2) give insights that teachers may not be aware of
Disadvantages:
1) need cooperation and time commitment
2) no benefit for students to attend this activity
41
Procedures used in conducting evaluations
9. Case study: how student made use of lesson plans throughout course, or trace the progress of a particular learner Advantages: 1) provide detailed information 2) provide a rich picture of different dimensions of courses Disadvantages: 1) the subject may not be representative 2) time-consuming
42
Procedures used in conducting evaluations
10. Student evaluation:
written or oral feedback on a course
ex: the teachers’ approach, the materials used, and their
relevant to the students’ needs Advantages:
easy to obtain information which contains a wide range
of topics and large numbers of learners to be involved
Disadvantages:
1) subjective and impressionistic
2) difficult to interpret or generalize
43
Procedures used in conducting evaluations
11. Audio- or video-recording:
provide some examples of different teaching styles and
lesson formats Advantages:
provide a rich account of teaching in real time and record
information
Disadvantages:
good recordings are hard to set up
44
Procedures used in conducting evaluations
12. Observation by other teachers or a supervisor ex: 1) give a specific task 2) provide procedures to the observer Advantage: 1) observers can focus on many things in class 2) provide an objective viewpoint Disadvantage: 1) the observer’s presence may be intrusive 2) require preparation and explicit guidance