1 tesl 4340 -- evaluating call packages:curriculum/pedagogical/lingui stics dr. henry tao guo email:...
TRANSCRIPT
1
TESL 4340 -- Evaluating CALL Packages:Curriculum/Pedagogical/Lingui
stics
Dr. Henry Tao GUO Email: [email protected]: B 418
2
Outline
• Part one: Language program evaluation (Lynch 1996: 1-9)
• Part two: Evaluating CALL(Chapelle 2001:44-94)
3
Part one: Language program evaluation
• Critical issues
• Context-adaptive model (CAM)
4
Evaluation
• Evaluation here is defined as the systematic attempt to gather information in order to make judgments or decisions.
5
• As such, evaluative information can be gathered through different methods such as
ObservationQuestionaireInterview …
6
Why CALL evaluation?
• Internally, to better a program
• Externally, to justify program funding
7
8
Context Adaptive Model (CAM)
• Who needs the information? And what are the goals of evaluation? What are the roles of evaluators?
• Contextual questions• Which aspect of the program should the
evaluation focus on?• Data collection and analysis• Evaluation report
(Adapted from Lynch 1990a)
9
Who needs the information?
Audience
• Who is requesting the evaluation? • Who will be affected by the
evaluation?
Stakeholders (or clients)& Peripheral audience
10
What are the goals of evaluation?
• Why is the evaluation being conducted?
• What information is being requested and why?
The goals of evaluation may be determined by audience
11
What are the role of evaluator?
• The role of evaluator is determined by the particular audience and their goals
Someone providing consultation An expert standing in judgment A collaborator in program development A decision-making facilitator
12
• Hiring external evaluators for greater objectivity or internal evaluators for improving the curriculum?
13
Contextual questions
1. Is there a comparison group?2. Are there reliable and valid measures of
language skills?3. Are there various types of evaluation
expertise?4. Timing of the evaluation5. How students are admitted into the
program?
14
6. What the program students are like7. What the program staff are like8. Size and intensity of the program9. Instructional materials and resources10. Perspective and purpose of the program11. Social and political climate
15
Preliminary thematic framework (PTF)
• Where should the evaluator begin?
• What aspects of the program should evaluator investigate in detail?
16
An example of PTF developed for EST reading program
1. Effects of focusing instruction on reading only
2. Effects of focusing instruction on reading skills and strategies
3. Effects of using authentic reading texts
4. Feasibility of using students’ mother tongue versus target language for instruction
17
5. Availability of classrooms
6. Feasibility of using a ‘modified adjunct model’ approach
7. Feasibility and effects of conducting classroom-centered research
8. Levels of student proficiency in English upon entering the program
(Adapted from Lynch 1990a)
18
Data collection design/system
• What type of data need to be gathered-quantitative, qualitative or both?
• What will be the best method for gathering the data?
• The contextual questions can be used to determine which types of data collection design is possible.
19
Data collection and analysis
• How is data collection to be conducted?
• How is the results to be interpreted?
20
In the case of quantitative designs, have the assumptions of the design and statistical models been met?
In the case of qualitative design, have the procedures for data gathering been portrayed accurately, and have alternative interpretations of the data been pursued?
21
Evaluation report
• The evaluator must be sensitive to the audience and goals of the evaluation
• The social and political climate dimension of the context needs to be considered carefully at this stage
22
• The critical issue is how to communicate the findings of the evaluation honestly and successfully.
• Prepare multiple reports or express the findings in different ways, depending on the intended audiences.
23
Part two: Evaluating CALL
• The problem of instructed SLA
• Principles for CALL evaluation
• Judgmental evaluation of CALL
• Empirical evaluation of CALL
24
The problem of instructed SLA
• Cognitive conditions for SLA
• Social-affective conditions for SLA
• Other factors
25
Cognitive conditions for SLA
Choose a range of target structures Choose tasks which meet the utility
condition Select and sequence tasks to achieve
balanced goal development Maximize the chances of focus on form
through attentional manipulation Use cycles of accountability
26
Social-affective conditions for SLA
Willingness to communicate (WTC)
The desire to communicate with a particular person
Communicative self confidence at that particular moment
Interpersonal motivation Intergroup motivation
27
Self confidence Intergroup attitudes(e.g., integrativeness)Social situation(i.e.,features of context
affecting communication)Communicative competence Intergroup climatepersonality
28
Other factors
Individual differences in cognitive characteristics of learners due to, for example, age or cognitive style.
Learning situation such as the effects of task choice on teachers and learners and others who may be involved with the task
Practical factors such as the available resources
29
Principles for CALL evaluation
• Evaluation of CALL is a situation specific argument
• Judgemental anlaysis of software and planned tasks and empirical analysis of learners’ performances
30
• Criteria should come from theory and research on second language teaching
• Criteria should be applied in view of the purpose of the task
• Language learning potential should be the central criterion in evaluation of CALL
31
Judgmental and empirical analyses
32
Criteria from theory and research on SLA
• Language learning potential• Learner fit• Meaning focus• Authenticity• Positive impact• practicality
33
34
Criteria applied based on task purpose•These criteria for CALL appropriateness
need to be applied in view of the purpose of a CALL task.
•Moreover, tasks may have different purposes at various stages of instruction (Doughty & Williams,1998).
•Whatever the goal of the CALL task, however, evaluation of the task requires that it have a stated purpose.
35
The centrality of language learning
• Even though the importance of each of the six criteria may vary depending on the purpose of the tasks, language learning potential should be considered the most critical for CALL activities
36
Judgemental evaluation of CALL
37
38
Cases(Chapelle 58-67)
• Computer-assisted classroom discussion• Microworld• Text analysis• Storyboard• Concordancing
39
Summary
40
Empirical evaluation of CALL
• Why?
‘Students are often doing something very different from what [ language teachers ] assume they are doing’ ( Hosenfeld,1976:123). In other words, it is necessary to identify the observable data that provide evidence of CALL qualities.
41
How? Look for evidences
42
43
Language learning potential
• Focus on form• Modified interaction• Modified output
44
Learner fit
• Level of linguistic difficulty
• Individual difference
45
Meaning focus
• Effects of meaning-based instruction
• Assessing engagement with meaning
46
Authenticity
• Comparing CALL with non-CALL activities
47
Positive impact
48
Practicality
• Given the role of resources for the success of CALL, 1) some formal mechanism needs to be in place to monitor adequacy, and 2) an argument about CALL appropriateness should include a statement about sufficiency of resources.
49
Conclusion
• Evidence concerning the six ideal qualities of a particular CALL task needs to be combined to form an evaluative argument about the appropriateness of a CALL task for particular learners at a given point in time.
50
• Integrated research is needed to examine the types of CALL activates to seek evidence about CALL appropriateness in particular settings.
51
• From such research, implications can be drawn for specific contexts, and eventually research on particular types of activities might be generalised in a way that can inform general principles of CALL.