Computer-Assisted Language Learning (CALL)

Assessment Issues: The Good and the Bad

by Hannah Foy, Said Hamideh, Demaris Kenwood, and Monica Niespodziany

What is CALL?CALL is the future!

● Includes a variety of different softwares and “apps,” straying away from traditional teaching and shifting into a technology-based approach

● Different approaches to teaching and learning language, heavily based in online connection

The basicsWhy does CALL matter?

● CALL has exploded with user friendly, gamified apps that are “sticky” or addictive. Many can be used on mobile phones, making them easily accessible

● Apps that complement learning are pushing their way into the classroom as the new norm

CALL in the classroom

✔ Students

✔ Parents

✔ Curriculum Specialists

✔ School Administrators

✔ Teachers

✔ EdTech Bloggers

✔ Politicians

✔ Policy Makers

✔ Academics

So, how exactly does call fit into the language classroom? We asked....

CALL as an assessment toolWe asked several students, professionals and parents…

● Q1: What should be the role of CALL in the classroom based on the benefits?

● Q2: What are the limitations of CALL in language learning?

● Q3: What are the ethical issues surrounding the rapidly increasing usage of CALL in language learning?

3 Main Approaches of CALL

● Tutorial

● Engagement with authentic materials○ radio stations, tv programs, newspapers, etc.

● Communication

CALL as a tutorial

Example: duolingo

CALL with authentic resources

Example: twitter

CALL as a communication tool

Example:busuu

Online Learning 2.0:The mobile learning “revolution” at a glance

● Most apps built to reduce/eliminate the need for teachers

● Apps are more interactive

● Apps tap into innate human motivations for learning

● Improvements in audio/recording and interactive design make these assessment tools increasingly multi-modal

○ able to assess your writing, reading, listening comprehension, speaking, and conversational skills

Assessing listeningBrown and Abeywickrama ask:

● “Can you assess one skill in isolation, without the participation of at least one other skill?”

● Can we directly observe the performance of all four skills?

Duolingo sort of does it...Duolingo can test two skills in the same exercise: listening to the voice, and then having the user speak the sentence.

Cons:

● Voice is monotone, sometimes distorted and never varies.

● This exercise is never a substitute for conversation.

● Computer doesn’t give feedback on test taker’s pronunciation. It’s strictly pass/fail

● Teacher must be present to observe the student performing the skill when pass/fail scores not sufficient

Busuu relies on humans

Cons:● Feedback is not immediate ● Assessors are members of

community/receive zero compensation for their labor/ and no barter exchange driving feedback

● Lack of clear compensation model casts doubts on reliability and consistency of feedback

Livemocha assesses listening without AI

Livemocha will assess listening/writing and listening/ speaking without “artifical intelligence”. Humans do the assessing driven by a bartering system.

Cons:

● Feedback costs money or barter exchange

● Feedback is not immediate● User interface is not gamified like

Duolingo

Assessing speakingBrown and Abeywickrama remind us there are five different types of speaking (p.184):

1. Imitative2. Intensive3. Responsive4. Interactive5. Extenstive

All technology in question tested users on their ability to perform “imitative” speaking

Brown and Abeywickrama (p. 184): “Imitative speaking is the ability to simply parrot back a word or phrase..this is purely phonetic level oral production..no inferences are made about the test taker’s ability to understand or convey meaning or to participate in an interactive conersation

Assessing readingBrown and Abeywickrama ask (p. 224):

“Is reading so natural and normal that learners should simply be exposed to written texts with no particular instruction?”

“Quoted!”One interviewee says…

“With Duolingo, there is no instruction on how to learn anything whatsoever! None. I haven’t come across any instruction on how to learn things quicker except of course in the main forum and even there rather not too precise which is easy to understand as those are not too wide spread which in the end is to my advantage but to any learner's disadvantage. In German such a behavior is called 'Unterlassene Hilfeleistung' and is punishable.” -Denk Muskel

Bottom up vs. Top downFrom Brown and Abeywickrama (p. 224):

● Bottom up: strategies for processing separate letters, words, and phrases

● Top Down: conceptually driven strategies for comprehension. Readers must develop appropriate content and formal schemata to carry out correct interpretation

CALL on reading strategies ● It is easy for CALL to offer reading assessments that

measure bottom up reading strategies.

● It’s more challenging to test top-down reading on a massive self-service website like Duolingo.

● But that doesn’t mean they don’t try..

Assessing reading….through crowdsourced translations

Example: duolingo

How it affects reading assessment (in Duolingo)

● Crowdsourcing means that anyone with minimum of experience on Duolingo is allowed to assess your translations

● Assessors may only give you a reductive “looks good” or ”looks wrong”

● Detailed feedback on translation is no guarantee

● While translation requires reading comprehension, feedback on written translation is not a direct observation of reading comprehension. It is an observation of writing.

● Duolingo does not have a direct assessment on any type of reading skill.

Livemocha tackles “bottom-up” reading assessments in a direct, but limited way

Livemocha uses a short-answer reading task.

The user reads questions that must be answered in written form in a sentence or two.

Cons

● Software provides zero automated feedback. Not a simple correct/incorrect

● Human feedback has a cost and is not immediate

● No comprehension questions available

So what did interviewees think? Two themes arise

from interviews

People liked to discuss learning technology in terms of its:

● Capabilities● Appropriateness

the good,the bad...

the good● Students enjoy the versatility, availability, flexibility,

authentic materials and convenience of CALL.

● Teachers consider CALL another tool in their toolkit for differentiation, variety, independent work, authentic up-to-date materials and fun class activities.

● Administrators, Districts and School Boards are eyeing the cost-cutting benefits of CALL in an era of budget cuts.

● Parents see CALL as a supplement to school which can help with additional practice to assist and challenge their children.

● School and community libraries provide CALL as a way to engage children in learning and reading.

● Politicians and policy makers are looking at ways to innovate the schools of tomorrow-which may not have walls.

the good

● Student teachers of today have grown up with technology and integrate CALL into lessons to engage students and provide extended learning opportunities such as Skype chats, epals, Facebook pages, video exchanges and interactive games and activities.

● Designers of CALL are constantly working to improve their products to bring them to a wider audience.

the good

● Students get bored with the same CALL and move on to new CALL without learning in depth or with live human interaction.

● Teachers have trouble monitoring the use and learning done with CALL.

● Administrators, Districts and School Boards can’t provide enough technology or the updates required to provide all students access.

the bad

● Parents worry about students becoming “zombies hiding behind a screen” instead of engaging in human interactions.

● Schools and community libraries have trouble providing enough devices for all students and keeping them current.

● Politicians and policy makers focus on computerized testing for accountability, diminishing the role of teachers in instructing and assessing students.

the bad

● Student teachers are frustrated with out of date or insufficient technology to use new CALL in the classroom.

● Designers recognize there are still many bugs to work out to make CALL accurate and interactive.

● Computers recognize only one correct answer vs. different lexical variants.

the bad

All of the stakeholders agree that CALL…

● is a valuable supplement to language learning that is convenient and self-paced

● provides a wealth of authentic material and opportunities for cultural and linguistic exchange

the conclusions

but...

CALL cannot stand alone nor replace the human

interaction of the live, in person, language classroom.

... human interaction and communication iskey to language learning!

CALL-The conclusions