developing a multidimensional checklist for evaluating language-learning websites coherent with the...

8/18/2019 Developing a Multidimensional Checklist for Evaluating Language-Learning Websites Coherent with the Communic…

1/37

Interdisciplinary Journal of e-Skills and Lifelong Learning Volume 12, 2016

Cite as: Moncada Linares, S., & Díaz Romero, C. (2016). Developing multidimensional checklist for evaluating lan-guage-learning websites coherent with the communicative approach: A path for the knowing-how-to-do enhancement.

Interdisciplinary Journal of e-Skills and Life Long Learning, 12, 57-93. Retrieved from

http://www.ijello.org/Volume12/IJELLv12p057-093Moncada2161.pdf

Editor: Gila Kurtz Submitted: November 22, 2015; Revised: February 16, 2016; Accepted: February 20, 2016

Developing a Multidimensional Checklist forEvaluating Language-Learning Websites Coherent

with the Communicative Approach: A Path for theKnowing-How-To-Do Enhancement

Sthephanny Moncada Linares and Andrea Carolina Díaz RomeroUniversidad La Gran Colombia, Colegio Cristo Rey,

Bogotá, Colombia

[email protected]; [email protected]

[NOTE: A shorter version of this paper was presented at the Eighth edition of the International

Conference ICT for Language Learning (Florence, Italy, 2015).]

AbstractAs a result of the rapid development of Information and Communication Technology (ICT) and

the growing interest in Internet-based tools for language classroom, it has become a pressing needfor educators to locate, evaluate and select the most appropriate language-learning digital re-

sources that foster more communicative and meaningful learning processes. Hence, this paperdescribes a mixed research project that, on the first hand, aimed at proposing a Checklist for eval-

uating language websites built on the principles of the Communicative Approach, and on thesecond hand, sought to strengthen the teachers’ Knowing-how-to-do skill as part of their digitalcompetence. To achieve these goals, a four-phase research procedure was followed that included

reviewing relevant literature and administering qualitative and quantitative research methods to participants (i.e., language teachers, an expert in the Computer-Assisted Language Learning

(CALL) field and a college professor) in order to gain insights into problematic issues and, there-after, to contribute to the creation and validation of the Checklist model and the Study Guide. Thefindings revealed that: (a) evaluating language websites leads to the enhancement of the teachers’

practical skills and their knowledge of the technological language; and (b) having an assessmentinstrument allows educators to choose the materials that best meet their communicative teaching

purposes.

Keywords: Language-learning website, communicative approach, CALL evaluation, checklist,digital competence, knowing-how-to-do

skill

IntroductionThe great influence of the Internet on allareas of society (e.g., politics, economy,leisure, relationships, education) has

managed to redefine and extend the boundaries of interpersonal communica-tion, entertainment, information transfer,teaching practices, and so forth. There-

Material published as part of this publication, either on-line or

in print, is copyrighted by the Informing Science Institute.Permission to make digital or paper copy of part or all of theseworks for personal or classroom use is granted without fee

provided that the copies are not made or distributed for profitor commercial advantage AND that copies 1) bear this noticein full and 2) give the full citation on the first page. It is per-missible to abstract these works so long as credit is given. To

copy in all other cases or to republish or to post on a server orto redistribute to lists requires specific permission and paymentof a fee. Contact [email protected] to request

redistribution permission.

http://www.ijello.org/Volume12/IJELLv12p057-093Moncada2161.pdfhttp://www.ijello.org/Volume12/IJELLv12p057-093Moncada2161.pdfmailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://www.ijello.org/Volume12/IJELLv12p057-093Moncada2161.pdf


2/37

Checklist for Evaluating Language-Learning Websites Based on the Communicative Approach

58

by, in the education realm, especially in the language teaching and learning context, the integra-tion of the Internet as a supporting pedagogical means has brought new and innovating opportuni-

ties for students to encounter an endless number of authentic and motivating materials, to engagewith subject content, to interact in real-time and exchange meanings with people from other cul-

tures, and to become autonomous, intercultural and collaborative learners (Warschauer, Shetzer,& Meloni, 2000). These positive effects have been highly appreciated by language teachers seek-

ing to approach pupils in a more innovative and meaningful learning environment beyond thetraditional four-wall classroom, where free-access online resources such as language websiteshave become an increasingly appealing option to achieve this goal.

However, it should also be noted that when resorting to websites for enhancing language learningoutcomes, teachers must be aware of its limitations in regards to its internal architecture and itsimplications on a certain language teaching approach. Thus, in the first place, its possibilities of

engaging users in real communicative exchanges appear to be slightly minor, unlike e-learningspaces (e.g., Blackboard, WebCT) that are capable of simultaneously providing a multiplicity of

management, communication and interaction features. Therefore, websites may exhibit somedifficulties in terms of: (a) integrating asynchronous and synchronous communication tools that

facilitate spontaneous language production; (b) centering more on discourse, contexts and culturerather than solely focusing on forms; and (c) giving less limited feedback that surpasses the au-tomatic treatment of errors (Cruz, 2003; E. Fernández, 2007; García, 2000; Kartal, 2005; Lozano

& Ruiz Campillo, 2009; Magnúsdóttir, 2010; Torat, 2000).

Consequently, the majority of websites tend to present language grounded on a behaviorist ap-

proach, by promoting deductive grammar explanations as well as structuralist and audio-lingualexercises (e.g., multiple-choice and close-ended questions, true or false items, and fill in the

blank, drills) that do not account for the authentic use of the target language. This has been main-ly facilitated by the ease, attractiveness and immediate feedback delivery that authoring programssuch as Hot-Potatoes and WebPractest commonly offered to create language content (Cruz, 2002;

Kartal, 2005; Magnúsdóttir, 2010), which in turn has led to a significant increase and never-ending supply of free-access websites that do not necessarily guarantee its overall quality and

pedagogical potential.

Hence, and unlike what everyone might think, web-based teaching undeniably constitutes a diffi-cult and demanding task for educators, as they might probably do not know to what extent web-

sites support an effective communicative language learning process and which sites to choosefrom the wide array of possibilities offered by the Internet (Hassan & Fakulti, 2011; Son, 2005;Susser, 2001; Yang & Chan, 2008). This is the reason why teachers need to be discerning andthoughtful when deciding on the most reliable, practical and valid online resources, which should

be ideally built on the principles of a teaching approach that encourages free and creative expres-

sion of the target language in realistic communicative contexts.

In this sense, as 21st century citizens, educators are being asked to acquire the Knowing-how-to-do skill as part of their Digital Competence (European Parliament and European Council, 2007)

to be able to evaluate the instructional purposes and language teaching objectives of any online

resource just before taking it to the class (Hubbard, 2006; Nomdedeu, n.d.). Thus, the need arisesfor a website evaluation model (Checklist) that initially guides them to efficiently find and judgethe overall appropriateness of any language site in terms of its technical and pedagogical (i.e.,

learning potential driven by the Communicative Approach) properties and capacities, so that theygradually gain expertise in this field while growing in their Knowing-how-to-do skill, as it is thecase of the target population that participated in this study.

Therefore, with the purpose of addressing this issue, the present paper follows this general struc-ture: first, it situates the reader in the literature review section that encompasses topics such as


3/37

Moncada & Díaz

59

website, the Communicative Approach, CALL evaluation and Digital Competence, and then itdisplays the four-phase methodology used to conduct this research along with the findings, dis-

cussion and conclusion sections.

Approaches to Language-Learning Website Evaluation

The increasing implementation of websites to supplement language learning instruction has driv-en teachers and researchers to undertake the task of evaluating its value, effectiveness and quality.The growing literature in this field has provided multiple checklists models that encompass a

global set of features related to the interface and aesthetics of the site, its functionality, usability,efficiency, navigation, reliability, authority, unity and some other technical elements that facili-

tate users’ experience when surfing online and which should be reflected in general types of sites.Additionally, a few of these checklist proposals integrate a pedagogical dimension, but mostlyfocus on measuring linguistic variables that include controlled practice exercises of receptivelanguage skills, grammar explanations, vocabulary lists, drilling, as well as aspects comprisinginstruction delivery, learning objectives, target audience and the difficulty level of exercises. Ta-

ble 1 shows a brief description of some of these models.

It is important to indicate that evaluating a language website is not just a matter of judging its

general layout and the elements that solely enhance learners’ linguistic competence; it should alsoattempt to look for those features that promote a more authentic and creative use of the targetlanguage. Thus, autonomy, language contextualization, integration of the four language skills,

interculturality, interaction and evaluation (Cruz, 2002; Hita, 2004; Higueras, 2010; Luzón &Soria, 1999; Richards & Rodgers, 1986) become principles capable of promoting within websitecommunicative language-learning processes directly dependent on the level of capacities and

functionalities these kinds of online resources possess. Moreover, these evaluation models focusspecifically on the sites’ intrinsic properties but are not concerned with, for example, what learn-

ers will be able to do, in terms of language learning cognitive processes, when surfing its con-tents.

Table 1: Existing evaluation criteria for websites and online materials

Author(s) Evaluation Criteria

Stoker &

Cooke (1994)

1. Authority2. Genealogy or origin of the source3. Scope and treatment (purpose, coverage, currency and methods of revision, ac-

curacy, objectivity, audience)

4. Format5. Arrangement6. Technical considerations7. Price and availability8. User support

Marquès

(1999)

1. Functionality and usability2. Technical and aesthetic features3. Pedagogical and psychological features (motivation capacity to be attractive and

interesting, adaptation to users’ characteristics and needs, didactic resources po-tentiality, self-learning and initiative encouragement)

Lázaro &

Fernández

(2000)

1. Identification of the material (recipients of the information, level of activities,learning objectives in relation to the content, timing, index and required skills)

2. Authorship3. Navigation4. Interface5. Functionality and usability


4/37


60

Author(s) Evaluation Criteria

Son (2005)

1. Purpose2. Accuracy3. Currency4. Authority5. Loading speed

6. Usefulness (convenient information and language activities)7. Organization8. Navigation9. Reliability10. Authenticity (language learning authentic materials)11. Interactivity12. Feedback13. Multimedia14. Communication (Bidirectional communication among learners)15. Integration of the online material into the curriculum

Aly(2008)

1. Authority2. Purpose3. Coverage4. Currency5. Objectivity6. Accuracy7. Technical aspects8. Usefulness for EFL Teachers and Students (usefulness and helpfulness in devel-

oping learners’ four language skills, grammar, vocabulary and pronunciation)

Higueras

(2010)

1. Technical aspects2. Reaping the benefits given by the online environment3. Profitability of internet access4. Reliability5. Amount of information provided for the teacher and student

Kartal &

Uzun (2010)

1. Physical characteristics2. Contextual characteristics (features of the site content such as testing tools,

software, lesson plans, exercises)3. Pedagogical characteristics (features that contribute to the learning andteaching process)

Liu, G., Liu,

Z., & Hwang,

(2011)

1. Web usability2. Learning materials3. Functionality of assisting language learning (offers multi-channel and hyperm

promoting language learning),

4. Technology integration (inclusion of technological resources to enhance the websiits users)

Therefore, the present study contributes to the literature of language website evaluation by providing a multidimensional checklist based upon the theoretical framework of the Communica-

tive Approach, which, additionally, seeks to augment the teachers’ sense of self-confidence andcritical thinking when selecting the language sites that best match their language teaching purpos-es.

Theoretical FrameworkThis research study was guided by a review of relevant scholarly literature on the topics of Web-site, Communicative Approach in virtual environments, CALL evaluation and Digital Compe-


5/37

Moncada & Díaz

61

tence. The contributions given by the authors provided background information to support theconstruction of the Checklist model and the Study Guide derived from this project.

WebsiteThe large supply of available free-access online resources has become, in recent years, an increas-

ing trend in language education. Specifically, Websites, as innovative digital settings and as partof new technology, have been highly recommended and used to supplement and enhance lan-guage learning. Authors such as E. Fernández (2007) have defined these resources as a collection

of webpages interrelated to each other by hyperlinks hosted in a same URL (Uniform ResourceLocator) that are created for specific purposes (García, 2000; Higueras, 2010; Marquès, 1999;

Nomdedeu, n.d.); in this case, for promoting language learning practices by including several

activities, exercises, links and extra tools (Mangenot, 1998). In relation to this, it is important toclarify that Websites or Sites are not the equivalent of Webpages (Codina, 2000; Higueras, 2010)

since, unlike what people might commonly think, these latter consist of a single page informationor document contained within a website, characterized for being written in a HTML (HyperTextMarkup Language) format (E. Fernández, 2007) that enables hypertextuality and hypermedia

(Rodríguez, 2004) to take place.

Thus, the technological potential of Websites and what make them suitable supporting instruc-tional materials come from its internal architecture design that encompasses useful features suchas Hypertext, Multimedia, Hypermedia and Interactivity (Area, 2009). First of all, Hypertext re-fers to a non-linear system of interconnected documents that allow users to decide on the path-

ways to follow through the site. Multimedia integrates several forms of information content suchas text, audio, graphics and animations. On the other hand, Hypermedia is related to the combina-

tion of hypertext and multimedia used for instruction, entertainment and information manage-ment. And finally , Interactivity comprises the possibility for users to receive any kind of automat-ic feedback as a response of a given action.

Apart from these elements, if the aim is to maximize the user’s experience when surfing a qualitysite, it should also provide: (a) an easy accessible and intuitive interface design that encompassesattractive and motivating colors, illustrative, explanatory or decorative images, icons as well asframes that enable the site to be divided into independent areas; (b) an efficient and understanda-

ble navigation system design (i.e., dropdown menus, toolbars, sidebars, etc.); and (c) a tree designstructure that hierarchically indicates the site’s content in order to simplify the browsing paths(Calero, cited by García, 2000). These attributes directly affect users’ disposition towards learn-

ing a target language as they make websites more user-friendly, appealing and reliable environ-ments, just as has been considered by authors such as Aly (2008), Higueras (2010), Kartal and

Uzun (2010), Lázaro and Fernández (2000), Marquès (1999), and Stoker and Cooke (1994), whowere presented in the previous section.

Finally, it is worth noting that websites can be classified in relation to its general structure, hyper-

link patterns and the editors’ purposes (Area, 2009; Gonzalo, 2006; E. Fernández, 2007; Marquès,1999; Nomdedeu, n.d.). In this sense, when browsing the net, users might encounter: search en-

gines created to crawl, index and search for online information; web directories of categorizedwebpages interconnected by hyperlinks; corporate sites of commercial nature with a limitednumber of links to other sites; hosting services where several websites are housed (Gonzalo,

2006); personal sites designed to display the editor’s curriculum; and finally, sites for education,specifically created to facilitate learning and provide didactic resources (Marquès, 1999). This

latter, within the language teaching context, consists primary of a series of linguistic exercisesthat take advantage of hypertextuality, multimedia and interactivity elements to promote language

practice (Cruz, 2004), also known as “self-access CALL materials” (Mishan, 2013) intended forindependent study and as a complement to classwork (Chuo & Kung, 2002).


6/37


62

The Communicative Approach in Virtual EnvironmentsIn the present research study, the Communicative Approach will be considered as the theory oflanguage learning that advocates for the development of the communicative competence bywhich students are able to use the target language appropriately and effectively in real communi-cation contexts. To accomplish this goal, authors such as Canale and Swain (1980), Chamorro

(2009), Higueras (2010), Hita (2004), Luzón and Soria (1999), Melero (2004), Richards andRodgers (1986), have pointed out that language education should meet fundamental communica-

tive principles such as: (a) place students at the center of the process to allow them greater auton-omy; (b) encourage exchange and negotiation of meanings; (c) foster interaction and the integra-tion of the four language skills; (d) make emphasis on learning and its formative assessment; and

(e) always contextualize the language.

In this regard, and with the growing interest in technology potentials, online resources have been

considered instructional materials par excellence to meet the above principles (Adell, 1995; Cruz2002; Higueras, 2010). Thus, for example, language websites are capable of promoting to a great-er or lesser extent the communicative and meaningful use of the target language by integrating

the following properties.

On the first hand, there is the Hypertextuality that allows language learners to manage the site’s

content and choose the order of discourse; an action that activates their rational thinking by favor-ing the interconnection between prior and present knowledge and, consequently, arousing theircapacity for reflection, analysis and interpretation (Cruz, 2002). One of its greatest advantages is

that it gives students the choice to take different pathways in relation to their interests, communi-cative needs, learning styles and pace as well as their language level, a feature that promotes an

active and more autonomous role rather than a passive stance (Higueras, 2010; Mao & Kung,2001).

Besides, hypertext websites enables users to gradually approach and explore new cultures wherethe linguistic component is embedded into the cultural one and vice versa (García, 2000), so that

by practicing the target language they are able to conduct intercultural exchanges and understand-

ing (Luzón & Soria, 1999).

Then, there is the Multimedia that favors knowledge contextualization (Cruz, 2002; Higueras,

2010; Luzón & Soria, 1999), enables practice on receptive language skills (Caridad, 2008), poten-tiates understanding by simultaneously activating perception senses, and boosts motivation tolearning (Mao & Kung, 2001). Moreover, it not only encourages unidirectional communication(i.e., user-machine) but also interaction among users and provides a vast amount of authentic(Cruz, 2002) or pseudo-authentic materials that enhance the website’s attractiveness and the stu-

dents’ engagement.

For its part, the Hypermedia promotes learning-by-doing in which students are given the possibil-ity to interact with different multimodal presentations (e.g., text, audio, animation, videos, etc.) as

it immerses them in a living and dynamic context where language is embedded in culture, so thatthey can learn within an enriched environment of diverse communicative situations (Cruz, 2002;

Liu, 1994). Additionally, it involves learners in an active role whereby through linking and ma-nipulating images, text and videos they become creative users and strengthen their language skillswhile being motivated to solve specific tasks.

There is also the Interactivity, commonly understood in terms of unidirectionality, by which thecommunicative exchange between user and machine places the student in the role of a transmitterand receiver (Haack, cited by Caridad, 2008; Màrques, cited by Higueras, 2010; Mantilla, n.d.),constituting a basic level of interaction. However, owing to the internal design limitations of most

websites, bidirectionality (i.e., user-user), and thus authentic communication, cannot be easily


7/37

Moncada & Díaz

63

achieved, being the reason why it is advisable that sites link users to external resources or at least, propose a series of activities to be conducted in a face-to-face classroom setting.

In addition to the above, interactivity has been as well defined in relation to the feedback provid-ed by websites as a response to learners’ input after solving any language exercise or activity.Commonly, language sites offer an automatic close-ended feedback that confirms whether an

answer is right or wrong by matching it with predetermined patterns (Caridad, 2008) or givingshort explanations. This is what make sites good at strengthening receptive skills through the

integration of controlled practice exercises but unable to provide a more flexible feedback systemthat comprises productive language skills as, unlike e-Learning environments, they would requirehaving an online tutor to guide the learning process (S. Fernández, 1988). Furthermore, despitethe feedback immediacy and its possibility of stimulating students’ self-correction (Lobin &Rösler, 2012), the errors treatment is still very limited and does not account for learners’ ability to

communicate in the target language.

Finally, there are the CMC (Computer-Mediated Communication) tools that represent essential

resources to foster effective exchange of meanings and language practice, as they encompasssynchronous and asynchronous communication. Thus, the former enables real-time interpersonal

communicative exchanges and collaboration among users, encouraged by the implementation of

chats, audio-conferencing, instant messaging, among other services that contribute to enhancinglearning motivation, the integration of the four language skills and the promotion of collaborative

learning and group work (Cruz, 2002; García, 2000). On the other hand, the latter allows thetransmission and interchange of information over a period of time via e-mail, listservs, discussion

boards, and so forth. These tools encourage students to focus on revising the formal aspects oftheir messages (i.e., grammatical correctness), the discourse order, the needed vocabulary and thesociolinguistic competence (Rodríguez, 2004), with the aim of gaining self-awareness on theaccuracy of the language.

It is worth noting that both these types of online communication tools, especially the synchronousone, are particularly favored in an e-Learning environment (Luzón & Soria, 1999) due to its in-trinsic architecture supported by learning management systems and a wide range of applications,which, unlike language websites, do promote to a higher extent personal telematic communica-

tions and thus authentic contexts to perform communicative activities.

CALL EvaluationWith the advent of the Internet and its implications in the language teaching and learning context,

educators are compelled to possess knowledge and expertise in the CALL field. It involves ac-quiring practical skills to implement several online materials (e.g., websites) as well as the ability

to take a critical and reflective stance for evaluating them in the light of their privileged languageteaching approach. This is why, during the past years, researchers and teachers have providedseveral evaluation proposals in the form of methodological frameworks (Chapelle, 2001; Hub-

bard, 1996; Plass, 1998), surveys (Hubbard, 2003; Knowles, 1992; Robb & Susser 2000), check-lists (Chapelle, 2001; Cruz, 2002; Higueras, 2010; Hubbard, 2006; Lázaro & Fernández, 2000;

Marquès, 1999; Son, 2005; etc.) and some other assessing methods to alleviate the languageteachers’ complex and time-consuming labor of judging the learning potential of any virtual envi-ronment.

In this sense, CALL evaluation has focused on conducting a systematic process to establish thequality, efficiency, overall value and suitability of a given technological material (e.g., software,online course, website, etc.) in order to make a decision on its possible integration into a particu-lar language learning setting (Chapelle, 2001; Hubbard, 2006; Levy & Stockwell, 2006; Robb &


8/37


64

Susser, 2000). This in turn has also implied building evaluation models following three specificaspects that should go hand-in-hand: an approach, a purpose and a method (Duignan, 2003).

To embark on this task, the first step is to think of an approach, which refers to a “way of lookingat or contextualizing evaluation” (Duignan, 2003, para.1). This can be done by centering on im-

plicit or explicit theoretical assumptions or a set of values that judge the level of congruency be-

tween the material’s intrinsic properties and its capacity to support learning from a specific lan-guage notion (e.g., Communicative Approach) (Hubbbard, 2006).

Then, it is important to determine the purpose, which aims at “identifying what evaluation activi-ty is going to be used for” (Duignan, 2003, para.1). It entails deciding on two different modalities

of CALL evaluation: the formative and the summative one. The former takes place during thedevelopment of a new material and seeks to identify its strengths and possible drawbacks in orderto improve it before being approved for use, whilst the latter is done after the completion of the

material and aims at determining whether it works well or not for the intended purpose; that is, to boost language learning (Chapelle, 2001, Colpaert, 2004; Levy & Stockwell, 2006; Ma, 2008).

In addition to the above, it is worth noting that the summative evaluation type comprises threelevels of analysis: two judgmental-based and one empirical (Chapelle, 2001). On the first level,the CALL material, is decontextualized as the teacher has not used it within a language class andhe/she is in the initial process of choosing the most appropriate one from a wide range of possibil-ities (also conceived by Squires & McDougall (1996) as predictive or selection evaluation and by

Reeder et al. (2004) as introspective). Meanwhile, the second level implies the way in which theteacher plans the material’s application for a specific context. And the third one reflects on theuser’s learning outcomes from the material.

Finally, there is the need to think of an evaluation method or instrument (e.g., consultations, par-ticipant observations, interviews, key informant surveys and questionnaires, checklists, etc.) that

leads to the collection, disclosure and subsequent analysis of data from the evaluated object (e.g,.learning material) (Duignan, 2003, para.1). Thus, for example, within the CALL evaluation field,

the checklist has been considered as one of the most common and practical instruments highlyused by language teachers and researchers to gain “insights on how well a particular program,

Web site, or online course is working” (Levy & Stockwell, 2006, p. 45), especially when con-ducting a first level judgmental evaluation (Chapelle, 2001; Reeder et al., 2004).

This assessment tool is frequently built on a set of evaluation criteria and descriptors presented in

the form of questions, categories, features, statements or explanations that aim at issuing a quanti-tative or qualitative measure over a specific level of accomplishments met by the evaluated object(Levy & Stockwell, 2006; Rodríguez, 2012; Vela, n.d.). In this sense, evaluation descriptors un-der each criterion are advised to be formulated based on three different elements: action, contentand condition (Campoverde, 2006), which, despite being used for judging students’ performanc-

es, may also account for the CALL materials intrinsic properties to foster quality language learn-ing processes (Díaz & Moncada, 2014).

Therefore, the action part of descriptors denotes an achievement expressed in an active verb (i.e.,

third-person singular) that responds to the question “what language learners will be able to do as aconsequence of the CALL material’s features?” Verbs can be written following the revised

Bloom’s taxonomy by Anderson and Krathwohl (2001), who shifted the thinking skills fromnouns to verbs, repositioned them and listed them in the form of cognitive processes (Low Order

Thinking Skills – LOTS – to Higher Order Thinking Skills – HOTS). Thus, for example, the verb“remembering” implies an information retrieval cognitive activity that can be easily promoted bylanguage-learning materials such as websites.


9/37

Moncada & Díaz

65

Additionally, the evaluation descriptor is composed by the content to which the action is relatedto, as well as the condition under which this action is expected to be performed (i.e., resources,

facilities, tools, etc.). Table 2 exemplifies how these elements join together to become possibleevaluation descriptors within a checklist.

Table 2: Examples of evaluation descriptors (HOTS/LOTS)

Cognitive process

CALL material (e.g., website) allows learners to: Action Content Condition

Creating

Generate attitudes of openness and em-

pathy towards the target cul-

ture

by including texts, readings and/or vid-

eos that address issues such as immigra-

tion, political debates, human rights,

among others

EvaluatingCheck their oral comprehension by clicking on the transcription button

for video or audio

Analyzing

Compare cultural traits of various speak-

ing regions of the target lan-

guage

by integrating topics of cultural interest

that have influenced people’s lives (e.g.,

popular culture, politics, history, etc.)

Applying

Carry out oral and written activities that

imply a communicative pur-

pose

by offering rich input (e.g., journals,

videos, recordings, etc.) and encourag-

ing input elaboration

Understanding

Inductively

Infer

the uses of certain grammatical

forms

by providing input enhancement (e.g.,

capital letter, boldface, underlining, ital-

ics, etc.)

Remembering

Identify in a particular exercise or ac-

tivity their right answers and

correct the errors

by clicking on the auto-feedback button.

It is worth noting that when exploring any learning material, language learners can be stimulatedto activate cognitive skills of different levels depending on the provided input and the intrinsiccapabilities of the material.

Digital CompetenceIn this emerging knowledge society, Digital Competence has been regarded as one of the eightlifelong learning competences every person must possess in order to effectively use and take ad-vantages of the digital tools at hand (European Parliament and European Council, 2007). It has

been even considered as a transversal area that enables people to acquire the other seven funda-mental competences, which are communication in the mother tongue, communication in foreignlanguages, mathematical and basic competence in science and technology, learning-to-learn, so-

cial and civic competence, sense of initiative and entrepreneurship, and cultural awareness andexpression (Ferrari, 2013).

Within the educational field, particularly in the language-teaching context, Digital Competencehas been gaining a lot of attention due to the increasing learning opportunities associated with therapid development of new technologies and its innovative uses. Consequently, language teachers,as “21st century citizens of the world,” have been expected to be more than competent to knowwhy and how to implement technology into their classrooms, to evaluate its appropriateness and

suitability for their intended teaching objectives as well as to conscientize students about its edu-cational potential (Fotos & Brown, 2004; Instituto Cervantes, 2012). This has implied masteringand applying basic technology-related ICT instrumental and cognitive abilities resulting from theeducators’ own Knowledge, Attitudes and Skills, three constituent learning domains of any com-

petence (Lara, 2012).


10/37


66

Thus, the first domain involves the understanding of the digital and technical language, the means by which it is materialized as well as its role and the opportunities it provides for users (Know-

ing). The second one entails an active participation and responsible citizenship to be functional ina digital environment, where values, aspirations and priorities are key elements to act and interact

appropriately (Knowing how to be). And the last one refers to a practical knowledge by whichindividuals posit a critical and reflective mindset when searching, analyzing and selecting suitable

digital tools or materials (e.g., websites) and their ability to use them to perform specific tasks( Knowing-how-to-do skill ).

This latter domain suggests that language teachers become third reviewers or “third-party evalua-tors” (Levy & Stockwell, 2006, p. 46), responsible for judging the overall quality and appropri-ateness of a given online learning material based on their specific teaching approach, the learningcontext, the curriculum and their students’ needs. Nonetheless, this is not an ability that can be

gained overnight; it requires having background knowledge and expertise in CALL evaluation(Tecnologías Educativas, cited by Lion, 2012), what initially implies making use of helpful as-sessing methods (e.g., checklists) before being critically prepared to evaluate instinctively.

Methodology

ParticipantsPopulation of this research study encompassed a convenience small group of language teacherswho were taking the MA in Applied Linguistics to the Spanish as a Foreign Language at a PrivateUniversity (Bogotá, Colombia). Six participants were voluntarily chosen to whom a needs analy-sis data-collection instrument was administered in order to determine their expertise when select-ing CALL materials coherent with their class’ language teaching approach. Additionally, a semi-

structured interview was applied to a language college professor, which sought to explore anddeepen on her experiences and practices on this same issue. Finally, a single participant of thetarget population was selected to review and pilot test the resulting products of this project.

Research DesignThe methodological approach was guided by a mixed qualitative and quantitative research para-digm (Hernández, Fernández, & Baptista, 2006; Tashakkori & Teddlie, 2003) that made use of a

four-phase sequential procedure to have a better comprehension of the research problem and,thereafter, to develop and validate the resulting products of this project (i.e., the websites’ evalua-

tion checklist and the study guide).

This implied starting with a qualitative stage that aimed at exploring, identifying and understand-ing the initial variables of the study in order to gain insights into the problem and propose a pos-

sible solution (Hernández et al., 2006). Thus, in the first place, a semi-structured interview wasdone and audio-recorded, then it was transcribed so that researchers could read through the raw

data, recognize recurring themes and find relations among them, to finally interpret them and setcore categories or variables. Later on, an open-ended questionnaire was conducted and analyzed

following the same procedure applied to the interview. The last step was executing a data triangu-lation process (Patton, 1990; Tashakkori & Teddlie, 2003) by which findings collected from these

both methods were compared and verified, and subsequently confronted with relevant literature in

order to corroborate its validity and seek for possible convergences and divergences among them.This resulted in a series of problematic issues that allowed researchers to have a general compre-

hension of the nature of the research problem and thus to set an appropriate research question aswell as the objectives that would lead this study.


11/37

Moncada & Díaz

67

Afterwards, this project followed three quantitative stages that mainly focused on measuring andcontrolling the variables (Fernández & Baptista, 2006) resulting from judging the quality and

suitability of the proposed Checklist model and the Study guide so as to improve them.

Therefore, during the first stage, a survey adapted from Yang & Chan (2008) was applied to theChecklist model by one expert in the CALL field which, through a content and construct validity

assessment (Kelley, 1999; Rubio, Berg-Weger, & Tebbs, 2003), researchers sought to revisewhether the evaluation criteria and descriptors corresponded to each other and if these latter wereappropriate, representative, important and clear for the criteria they stood for (i.e., technical, ped-agogical-contextual and communicative approach-related characteristics) (see Appendix A).Then, after having refined the checklist proposal based on the expert’s judgments, in the secondstage, the instrument was pilot tested by one of the participants who applied it to a language web-

site; this same was conducted by researchers in order to verify its inter-rater reliability (Seliger &Shohamy, 1989). Finally, in the third stage, the same participant evaluated both products bymeans of a survey that inquired about the quality of the study guide as an instructional materialand for the checklist structural properties (see Appendix B).

Findings

Major outcomes from the four-phase procedure conducted in the present research are reported.

Phase I (qualitative stage)

The first stage was an application to the target population of an open-ended questionnaire and asemi-structured interview to a language college professor, both in the form of a needs analysisthat focused on identifying and gaining an in-depth understanding of the research problem. Thus,

the gathered information from these collecting instruments served to establish four core catego-ries that became the following orienting problematic issues: (a) the steadily increasing number of

websites grounded on a behaviorist language-learning theory; (b) its interface design that, unlike paper-based language materials, should integrate features such as hypertextuality, multimedia,

hypermedia and some others, so as to promote communicative achievements; (c) the mismatchfound between the teachers’ choice of apparently communicative-based language websites and

the ones that truly work for this purpose; and finally, (d) the lack of an evaluation instrument thatallows language teachers to obtain a true understanding of what a communicative language web-site should look like. Table 3 displays a sample of the triangulation matrix used to correlate data

resulted from both collecting instruments and its confrontation with literature.

The emerging research issues presented led researchers to draw the conclusion that there was a pressing need to propose an evaluation instrument for websites that comprises the principles ofthe Communicative approach, and that additionally focuses on what students will be able to dowithin this virtual environment as active participants of their own learning process. This would be

with the aim of guiding language educators to make a reliable and informed decision on the mostappropriate online material to integrate into class and as a brief training for them to progressivelydevelop their Knowing-how-to-do skill. Thus, right after conducting the need analysis and inter-

preting the gathered results, researchers started to review literature with the purpose of targetingthe first immediate need, which was the development of an initial checklist model. They analyzed

and synthesized various checklists evaluation proposals (Table 1) that accounted for what have been done and said about assessing language-learning websites and its intrinsic properties (e.g.,structure, organization, functionality, multimedia, content, language learning focus and son on).Based on this review process, it was found that most of the striking features included within these

proposals were directly linked to technical aspects, followed by some pedagogical considerations

and by a few language approach statements still limited to the linguistic competence, as it can beseen in Table 4.


12/37


68

Table 3: Triangulation matrix sample

Category

Direct responses General overview of the

revised literature Participants of the

target population

(survey responses)

Language college

professor

(interview responses)

Behavior-ist-based

Websites

Teacher 5: “considerar la retroali-

mentación inmediata como fuente deconocimiento” (to consider imme-diate feedback as a source of kno-

wledge).

Teacher 1: “ejercicios de completar

los espacios en blanco” (fill in thegaps exercises).

Teacher 6: “ generalmente se encuen-tran ejercicios de rellenar huecos, ya

sea con información dada de maneraescrita o de archivos de audio”(generally, it can be found fill in thegap exercises to be solved with

written or audio file input).

Teacher 4: “en Internet se puedenencontrar una variedad de ejercicios

para las habilidades receptivas y la

mayoría son ejercicios gramaticalesmecánicos. Casi no encuentro ejer-cicios para la producción oral ” (On

the Internet you can find a variety ofexercises for the receptive skills and

most of these are mechanicalgrammar exercises. I hardly findexercises for oral production).

Teacher 1: “hay mucha informaciónen la Red, sin embargo, está seencuentra desorganizada” (there is a

lot of information on the Web,however it is disorganized).

“ se llamaban ejercicios

interactivos pero ya con eltiempo me di cuenta de queno eran tan interactivos

como pensaba, se brindabala respuesta correcta y ya”

(these were called interac-tive exercises but I eventu-ally realized that they were

not as interactive as I

thought, they only providedthe correct answer).

“lo que había eran paginasenfocadas a las formas

lingüísticas…no hay un reto

más allá y el estudiante nove la lengua como instru-mento para comunicarse”

(there were websites fo-cused mainly on linguisticforms…these did not implyany challenge and the stu-

dent could not regard thelanguage as a tool for com-munication).

“el profesor de lenguas…se

siente como bombardeado por tanta información quehay en la red ” (the language

teacher…feels over-whelmed by so much in-

formation on the Web).

García (2000) claims that a wide

number of websites focus on providing grammatical explana-tions through structuralist exer-

cises, which ultimately relegatesthe language communicative

component.

Likewise, Kartal (cited by Kartal& Uzun, 2010) outlines that agreat number of language learn-

ing websites have narrowed their pedagogical approach to struc-tural exercises with immediatefeedback response. Moreover,

Martín (cited in Cruz, 2003)

indicates that most of the lan-guage exercises with an interac-tive look still being traditional,

structuralist-based.

Thus, the majority of those kindsof online materials offer exercis-es that enhance a deductive

language learning approach,favored by the ease of program-ming of authoring programs suchas Hot-potatoes and Web-

practest (E. Fernández, 2007).

Additionally, the growing offerof free-access website on the Netdoes not directly means quality(Susser, 2001; Yang & Chan,

2008; Son, 2005; Hassan &Fakulti, 2011).

Interface

design

qualities

Teacher 6: “los criterios que tendríaen cuenta para seleccionar un sitioweb serían la interactividad y la

lecturabilidad ” (the criteria I would

take into account when selecting awebsite will be interactivity andreadership).

Teacher 1: “que los sitios web notengan problemas de plugins” (web-sites must not have plugin failures).

Teacher 2: “un sitio web debe tener

practicidad” (A website must entail

practicality).

Teacher 5: “ No sabe / No responde”

(do not know /do not answer – thiswas given as a response to the ques-tion that inquired for the criteria

participants that take into accountwhen choosing a language website).

(Nothing was mentioned inthis respect).

Internet-based materials can becommonly evaluated under thesame criteria used for paper-

based resources, due to theapparent familiarity between

both of them. In this regards,Levy & Stockwell (2000) arguethat, unlike textbooks that imply

a linear reading, websites for itshypertextual nature need to be

evaluated based on its intrinsiccapabilities, which is why teach-

ers must be third-party evaluatorsof this kind of material.

A learning material cannot belimited to the format; instead, it

must be adapted to the audiovis-ual culture supported by multi-media and hypertextuality online

properties (Cruz, 2002).


13/37

Moncada & Díaz

69

Category

Direct responses General overview of the

revised literature Participants of the

target population

(survey responses)

Language college

professor

(interview responses)

Teacher’s

mis-

matching

choice of

websites

Teacher 2: “ Me encanta Duolindo, esinteractivo y comunicativo” (I loveDuolingo, it is interactive and

communicative).

“Recuerdo muy bien una página que a mí me parecía perfecta…entonces me di

cuenta que realmente losestudiantes no estaban

haciendo ningún tipo denegociación ni un intercam-

bio significativo” (I re-

member a website thatseemed to be perfect…thenI realized that students

weren’t actually making

any sort of communicativeexchange).

Warschauer (cited by Chapelle &Hegelheimer, 2004) argues thatthere is a need to rethink the

communication competence invirtual environments.

It is worth noting that Duolingois a crowdsourcing platform thatencourages students to learn the

target language by translatingwords and phrases from theirmother tongue and vice versa.

Lack of

an as-

sessment

instru-

ment

Teacher 1: “una herramienta de

evaluación permitiría facilitar milabor como docente de lengua” (An

assessment tool would make easiermy work as a language teacher).

Teacher 3: “sería bueno contar conuna herramienta…que me ayude aescoger un buen sitio web, dado quehasta el momento no conozco esos

sitios muy bien” (It would be nice tohave a tool… that helps me to pick

a good website, given that so far Idon’t really know those sites that

well).

Teacher 5: “nunca me he detenido aevaluar la concordancia de las

páginas que uso con lo que hago

diariamente en mis clases, entoncessería bueno tener una herramienta deevaluación para este propósito” (Ihave never considered evaluating thecoherence between the websites I use

daily in my classes, so it would begreat to have an assessment tool for

this purpose).

“ yo creo que es bastante

importante que se pudieratener un recurso…para que

encuentre una página querealmente le permita pre-

sentarle a sus estudiantesun material de apoyo decalidad… en donde lasinstrucciones sean claras,

sencillas y que sobre todo permita el uso real de lalengua” (I think it’s quiteimportant to count on with a

resource…to find a websi-te that truly allows teachersto provide students with aquality supporting mate-rial…where instructions are

clear and overall, leads tothe real use of the langua-ge).

According to Higueras (2010)

and Magúsdóttir (2010), thelarge amount of free-access

websites possess a problem ofdiscrimination for language

teachers, being this the reasonwhy, there is a need to count onwith a clear set of criteria thatallow educators to make valuable

judgements on the most appropriate websites for the intendedteaching purposes.

Fotos & Brown (2004) note that

“language teachers are increas-ingly required to possess CALLexpertise that includes both

practical skills and a thorough

understanding of informationtechnology theory. Teachers mayneed to design, implement andevaluate CALL activities in theirclassroom” (p.3). The same is

considered by authors such asSon (2005); Levy & Stockwell(2006), Chapelle (2001).

Table 4: Website evaluation proposals

Dimension Websites features / items Authors

Technicalaspects

- Authority- Authorship

- Accuracy- Navigation- Interface

- Usefulness- Edition settings

-Genealogy or source origin- Loading speed

Stoker & Cooke (1994), Marquès,(1999), Lázaro & Fernández (2000),

Son (2005), Aly (2008), Yang &Chan (2008), Higueras (2010), Kartal& Uzun (2010), Dogoritti & Pagge(2012)

Pedagogical

considera-

tions

- Learning materials- Content- Difficulty level of activities

- Evaluation information- Learners’ preferences- Contextual characteristics

Marquès (1999), Trenchs Parera(2001), Yang & Chan (2008), Higue-ras (2010), Kartal & Uzun (2010)

Language

approach

principles

- Language skills - Feedback delivery- Communication setting

Son (2005), Yang & Chan (2008),Lozano & Ruiz Campillo (2009),

Higueras (2010)


14/37


70

Once the literature has been reviewed and researchers have chosen the specific items to includewithin the Checklist proposal, it was necessary to conduct an additional exploratory search on

current Spanish learning websites (e.g., Español para extranjeros http://www.aurora.patrick-nieto.fr/ , RedMolinos http://www.redmolinos.com/ , ver-taal http://www.ver-taal.com/ , La ruta de

la lengua española http://www.larutadelalengua.com/index2.htm , etc.) in order to observe theircapabilities and limitations and thus to derive extra features. This whole process resulted in a

preliminary Checklist model that encompassed nine evaluation criteria and a total of 40 de-scriptors included within three evaluation dimensions, as referred as follows:

1. Technical characteristics related to the ease of use (7 descriptors) and reliability criteria(3 descriptors).

2. Pedagogical contextual characteristics referring to the content quality criterion (5 de-scriptors) and the material potentiality criterion, this latter in terms of the provided ad-

vantages and benefits (2 descriptors).

3. Communicative Approach-related characteristics that reflected on criteria such as au-tonomy (4 descriptors), language contextualization (5 descriptors), language skills inte-gration (6 descriptors), interculturality (4 descriptors) and interaction (4 descriptors).

Finally, it is important to mention that, during this first stage, a theoretical framework concerningthe Communicative Approach principles, the language websites’ features and Digital Competencewas also taken into account with the purpose of: (a) conducting the triangulation procedure (see

Table 3); and (b) establishing the orienting checklist dimensions (i.e., evaluation criteria) alongwith its descriptors; in particular, the ones implying the Communicative Approach-related char-acteristics.

Phase II (quantitative stage)

A four-part validity survey that sought to improve the quality of the preliminary evaluation crite-ria and descriptors was applied to an expert in the CALL field. This data collection instrument,adapted from Yang & Chang (2008), asked to carefully revise, rate and provide comments or

suggestions for each of the evaluation components that comprised the preliminary proposal. This

included judging the appropriateness (i.e., the suitability of a descriptor in relation to the criterionit was proposed for), representativeness (i.e., the descriptor stands for the criterion it was pro-

posed for), importance (i.e., the relevance of the descriptor for the respectively criterion), andclarity (i.e., avoid wording ambiguity) of descriptors, as well as checking their belonging to the

proposed criteria (see Appendix A).

The results from this survey (see Figure 1) showed that the evaluation dimension regarding the

Technical aspects received an overall assessment of 99.4%, since the descriptors encompassedwithin the “ease of use” and “reliability” criteria were 100% representative, important and appro-

priate, and 97.5% clear. However, the reviewer recommended that there should be more de-

scriptors concerning the hardware-software technical specifications (ease of use criterion) andone item that inquires for the author(s) of the website (reliability criterion).

On the other hand, the Pedagogical contextual dimension obtained an 85.7% valuation, as itsdescriptors were 96.4% representative, 89.2% important, 85.7% clear and 71.4% appropriate.This revealed that it was necessary to make descriptors more comprehensible and shorter, this by

including exemplifications, clarifications, etc. (content quality criterion); likewise, descriptorsimplying the advantages and benefits of the website could be deleted as they are taken up in sub-

sequent descriptors (material potentiality criterion).

Finally, concerning the Communicative Approach dimension, it could be observed that it got an

82.6% of accomplishment, as its descriptors showed to be 79.3% representative, 82.6% important, 90.2% clear and 78.2% appropriate. In this respect, the expert asserted that: (a) the de-

http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/http://www.redmolinos.com/http://www.redmolinos.com/http://www.redmolinos.com/http://www.ver-taal.com/http://www.ver-taal.com/http://www.ver-taal.com/http://www.larutadelalengua.com/index2.htmhttp://www.larutadelalengua.com/index2.htmhttp://www.larutadelalengua.com/index2.htmhttp://www.larutadelalengua.com/index2.htmhttp://www.larutadelalengua.com/index2.htmhttp://www.ver-taal.com/http://www.redmolinos.com/http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/


15/37

Moncada & Díaz

71

scriptor about evaluation should be formulated in a way that does not purely imply test taking buta more formative assessment process (autonomy criterion); (b) some of the descriptors tend to

make emphasis on linguistic and lexical properties and do not take into consideration other aspects that could promote authentic communication exchanges (language contextualization criteri-

on); (c) three descriptors do not account for the language skills integration (language skills inte-gration criterion); (d) descriptors should include the notion of attitude as one of the important

aspects involved in the learning of the target culture (interculturality criterion); and (e) some ofthe descriptors focused mainly on the machine-student interaction and did not express real ex-change of meanings, so for this reason they should be integrated into a new criterion that embrac-

es feedback-delivery (interactivity criterion).

These outcomes were of great help to researchers for gathering useful and important feedback onhow well-founded and accurate were the initial evaluation criteria and descriptors. Thus, from

this validity survey, 11 evaluation descriptors were revised and improved, and other 19 were in-corporated. Even though the findings showed that some of them had wording problems and need-

ed to be rewritten again, they were still appropriate, representative and important.

Figure 1: Expert rating over preliminary Checklist proposal

In light of the above, researchers were able to refine the preliminary Checklist model (see Appen-dix C) and obtain an evaluation tool that comprised a total of 49 descriptors integrated into the

three already known dimensions and eight evaluation criteria. This instrument can be used byreviewers or third-party evaluators to judge the value and quality of any language-learning web-site by simply checking off “Yes” or “No” in the given grid as well as adding some personal

comments based on the degree to which the online material meets the intended communicativeteaching objectives. Table 5 explains in detail each of the final checklist’s components.

Apart from the Checklist model, the other goal of this research project was to strengthen theteachers’ Knowing-how-to-do skill as part of their Digital Competence. This was attempted to beaccomplished by designing a Study Guide

(https://issuu.com/sthephannymoncadalinares/docs/guiapracticafinal) , as a training and informa-tional instructional material that could provide a conceptual and practical framework along withmethodological suggestions for educators to become familiar with language-learning websites

just before evaluating them.

https://issuu.com/sthephannymoncadalinares/docs/guiapracticafinalhttps://issuu.com/sthephannymoncadalinares/docs/guiapracticafinal


16/37


72

Table 5: Checklist evaluation model

Technical features dimension

Allow users to recognize at first sight, whether thewebsite is worth exploring and if it serves as a com-

plementary material for the classroom use.

Pedagogical context features dimension

The opportunities offered by the website to facilitatelearning and how it adapts to complement the learners’needs and interests in relation to the contents, activities

and extra resources.

14 Descriptors:

Ease of use criterion (9 descriptors): It covers

the website qualities in regards to a friendly,clear, simple, consistent, attractive and motivat-ing graphic interface, aspects that directly affect

the user’s browsing experience. Therefore, web-sites for itself should be self-explanatory and not

possess any difficulties when browsing it, re-trieving information, obtaining learning materials

or following hyperlinks.

Reliability criterion (5 descriptors): It showsthat a website is credible when it is possible to

identify its developers, it does not present anysystematic errors that hinder navigation and hy-

perlinks do not connect to other dubious sites.

5 Descriptors:

Quality of contents criterion (5 descriptors): it

encompasses a variety of integrated content, activi-ties and resources while indicating its level of com-

plexity. Additionally, it should present the learning

objectives to be achieved with the completion ofany activity or exercise, as well as the correspond-ing explanations and clear instructions that guideusers without the need to require an external super-

vision.

Communicative Approach theoretical dimensionThis criterion aside from being proposed based an approach-driven framework (i.e., Communicative Approach), itsdescriptors target at describing what students are able to do within the website in terms of Low or High Order think-ing skills (Anderson & Krathwohl, 2001).

30 descriptors:

Autonomy criterion (4 descriptors): It should promote a self-learning, self-monitored and self-regulating

space by the inclusion of hypertextuality and rich contextual hypermedia.

Language contextualization criterion (7 descriptors): It should introduce language input inductively and

additionally, enhance it (e.g., capital letter, boldface, underlining, italics, etc.). Activities and exercise musthave an implicit communicative intention and should be supported by various multimedia and hypermedia re-sources that contextualize knowledge.

Language skills integration criterion (7 descriptors): In real communication exchange the four language

skills are integrated, that is why a language website should ideally encompass authentic materials, multimedia,hypermedia and CMC tools to achieve this goal. However and differing from an e-Learning platform, re-searchers are aware that most websites have some intrinsic limitations in this regards, being the reason why

they commonly integrate a wide range of receptive skills activities.

Interculturality criterion (8 descriptors): through the integration of hypertext, multimedia, hypermedia and

authentic materials, websites are able to recreate a living and dynamic context where culture is present.

Evaluation criterion (4 descriptors): the possibility to receive automatic feedback that allows users to self-monitor and observe their performance when solving different exercises or activities. Some websites apart

from the right/wrong remark, place allusive phrases about what students will be able to do or include assess-ment rubrics.

The main structure of this created material consists of four sections plus a glossary and a refer-ence part:

• Section 1 - What do language teachers need to know? Fundamental concepts related tothe notion of Digital Competence, Communicative Approach, websites and its main fea-tures (hypertext, multimedia, hypermedia, and feedback tools CMO) are presented.

• Section 2 - What are we looking for in language learning websites? The CommunicativeApproach principles (i.e., autonomy, contextualization of language, integration skills, in-

tercultural interaction and assessment) are displayed in the form of evaluation criteria anddescriptors, as they appear in the checklist. This with the aim of illustrating what might

possess a certain language website if it is highly communicative. Thus, each descriptor is


17/37

Moncada & Díaz

73

accompanied by a screenshot that represents the available features or resources that meetthe teaching and learning expectations, so it can give a visual idea of what evaluators are

looking for.

• Section 3 - It is time to evaluate: The assessment instrument (checklist) is presented alongwith its corresponding instructions.

• Section 4 - Let's talk about design: In order to help direct the work of the reader to a prac-tical exercise, a number of methodological suggestions for the creation and design of awebsite are provided as an opportunity to put into practice the knowledge acquiredthroughout the material.

• Glossary: The definitions of eleven terms that may be new to the reader are organized inalphabetical order, giving conceptual clarity of the technicalities referred in the material(e.g., Web search engine, web page, URL, hyperlink, interface, among others).

• References.

This Study Guide was also reviewed by a professional in the language-teaching field, to whom anevaluation survey was administered consisting of 20 questions (see Appendix B). This evaluationwas performed in a space of two hours, where the college professor revised the whole instruction-

al material and then completed the survey format. The results showed that most of the material’ssections met the objectives for which they were developed, reaching a total score (i.e., over four

points - 100%) on aspects such as: (a) impact on arousing and sustaining the reader’s curiosityand attention by providing attractive presentation, appealing content and activities that boost theiractive thinking; (b) encouraging readers to feel at ease when facilitating the understanding ofcontents by including exemplifications, illustrations and other contextualizing resources; (c) de-veloping readers’ self-confidence by pushing their skills and knowledge beyond their existing

proficiency through activities with a progressive level of difficulty; and (d) promoting self-instruction. On the other hand, it obtained 3 points (87.5%) on its capability of offering usefuland relevant contents. It is important to mention that these criteria used for evaluating the StudyGuide was adapted from Tomlinson’s (2011) principles to design effective language-teachingmaterials.

Furthermore, as the checklist was also part of the Study Guide (section 3), the language professorwas asked to judge and evaluate its overall content, from which it received a rating from three to

four points in items such as: (a) its structure (three points); (b) the clarity of its instruction deliv-ery (four points); and (c) the quality of its descriptors (four points). The evaluation results from

both products can be seen in Table 6. The outcomes from this second phase of the study allowed

researchers to make improvements to the Checklist and the Study Guide before implementingthem in a real setting.

Table 6: Language college professor rating on the Study guide and the Checklist model

Evaluation criteria Given rating General remarks from the participant

STUDY GUIDE

Materials’ impact 100% (4 points) “en la guía se presentan imágenes, glosario, explica-

ciones. Hay llamados o sección “para pensar”. Seobserva una secuencia lógica en la presentación de

los contenidos. La presentación de información teóri-

ca le ayuda al docente a desarrollar la autoconfianza”

(the guide displays images, a glossary, explanations.

There is a section called “to think.” There is a logic

sequence in the exhibition of contents. The presenta-

tion of theoretical information helps teachers to de-

velop self-confidence).

Teacher’s development of

self-confidence100% (4 points)

Teacher’s feeling at ease 100% (4 points)

Perception towards the

displayed contents as rele-

vant and useful

87.5% (3 points)

Teacher’s self-instruction 100% (4 points)


18/37


74


CHECKLIST MODEL

Structure91.6%

(3 points)

“En general, la lista de chequeo presenta los criterios

de evaluación de forma y concisa. Por su parte, los

descriptores son fáciles de entender y están organiza-

dos de manera secuencial” (In general, the checklist

presents contents in a clear and concise way. For its part, descriptors are easy to understand and are orga-

nized in a sequential fashion).

Clarity of instructions100%

(4 points)

Quality of the evaluation

descriptors

100%

(4 points)

Phase III (quantitative stage)

With the purpose of applying and testing the quality and efficacy of the proposed Checklist mod-el, it was necessary to pick one language-learning website. The selection was conducted by typingin the Google search engine entries like “Spanish language-learning websites”, “the best language

websites to learn Spanish”, etc.; then, from the wide array of possibilities and after skimming andscanning some of the retrieved sites that were free access no-cost, reliable and useful as they inte-grated a variety of contents, activities and resources, the “ Español para extranjeros”(http://www.aurora.patrick-nieto.fr/) website was chosen.

During this third stage of the research, one participant of the target population was asked to usethe Checklist in order to evaluate the previously selected online environment. First, she received acopy of the Study Guide for her to read thoroughly; then, she was requested to familiarize with

the site, surfing and exploring its various options and pathways; and finally, she was encouragedto apply the evaluation instrument to this site.

Findings regarding this assessment process showed that, in general, the website accomplished in60.7% the descriptors encompassed within the three evaluation dimensions (i.e., technical, peda-gogical-contextual and the Communicative Approach-related characteristics), being ranked as a

“Good (appropriate for use)” complementary material for meeting 29 out of 49 evaluation items.

Thus, within the dimension of technical features, the site met in 59% the evaluation criteria, as it

proves to have free and no-cost access, it is easy to use as it displays a friendly and attractiveinterface that provides navigation guidance to users, makes use of a clear and precise languageand, in addition, has an appropriate balance between the textual and multimedia content. Howev-

er, browsing can be a bit difficult because it lacks permanent frames that allow the use of its in-dependent controllable sections. On the other hand, the site seems to be reliable as it is claimed to

be the result of a teaching project developed by its authors, who pointed out that it is founded onthe constructivism and notional-functional theoretical framework; nonetheless, information relat-ed to its permanent updating is not presented.

Some of the participant’s comments that support her rating over this category include: “no veo las pestañas” (I don’t see the tabs), “ yo le añadiría otros colores un poco más vivos” (I would add

more vivid colors), “los enlaces son muy llamativos” (the hyperlinks are too eye-catching), “veoque en algunas ocasiones el azul y el morado se confunde en las instrucciones y en los títulos” (I

see that sometimes blue and purple blends into the instructions and the titles), “no lo veo ni esvisible para mí la actualización” (the uptading option isn’t visible for me), and “ se basa en lasteorías de Piaget y Vigotsky, tal como se señala” (it is based on Piaget and Vigotsky’s theories,

just as it is indicated).

In regards to the dimension of pedagogical-contextual characteristics, the website reached a suc-

cess rate of 60%, since it includes a variety of contents, activities and resources, displays shortand understandable instructions and introduces grammar points inductively. Nevertheless, it doesnot present the learning objectives pursued by the realization of the different activities and exer-

http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/http://www.aurora.patrick-nieto.fr/


19/37

Moncada & Díaz

75

cises nor indicates its language level of difficulty, as the participant underlines: “ no hay objetivos planteados al inicio de cada tarea ni su nivel de lengua” (neither the goals nor the language level

are presented at the beginning of each task).

Finally, in relation to the Communicative Approach principles dimension, the website got a 63%of success, meeting 17 out of 30 evaluation descriptors. It proves to have a stronghold in promot-

ing autonomy, language contextualization and evaluation, as it allows students to freely chooseand make decisions on the pathways they are interested in following within or outside the site, it

offers different types of resources (e.g., multimedia, authentic materials, etc.) that contextualizeknowledge and facilitates the understanding of the target language, and additionally provides bothexplicit and implicit feedback that favor the recognition of the students’ strengths and weakness-es. Nonetheless, it fails to provide tools (e.g., evaluation rubrics) for monitoring their own lan-guage learning performance in terms of productive skills.

It is worth noting the website has some disadvantages when trying to successfully foster intercul-turality and interaction as, for example, it relies on a single sociocultural perspective coming fromthe peninsular Spanish and does not take into consideration other Spanish speaking regions; like-wise, it does not integrate or link users to other sites that encompass CMC tools that could ap-

proach them to interact with people and thus promote authentic and meaningful learning situa-

tions, as evidenced by the participant who asserted that “en los ejercicios que exploré, yo no en-contré herramientas CMO” (within the exercises that I explore, I couldn’t find CMC tools). Italso has some weaknesses with respect to the integration of language skills, because it does notinclude authentic sounds, such as songs and everyday dialogues, that could enhance the oral com-

prehension and, on the contrary, it mainly focuses on receptive skills. However, this site does provide learners with a series of printable complementary activities that offer practical applicationof the language communicative skills and, despite being paper-based, they can successfully en-hance oral communication by means of teamwork activities in class.

Table 7 portrays the results obtained from evaluating this site and compares the participant andthe researchers’ results, who also evaluated it with the purpose of testing the inter-rater reliabilityof the Checklist model in relation to the consistency and accuracy between both responses.

Table 7: Results of evaluation of the website Español para extranjeros

Dimension CriterionParticipant’s results Researchers’ results

Items Percentage (%) Items Percentage (%)

Technical features

Ease of use 7/9 77,7% 8/9 88,8%

Reliability 2/5 40% 3/5 60%

Total 9/14 59% 11/14 74,4%

Pedagogical contex-

tual characteristicsContents quality 3/5 60% 3/5 60%

Total 3/5 60% 3/5 60%

Communicativeapproach-related

characteristics

Autonomy 4/4 100% 4/4 100%

Language contex-

tualization 6/7 85,7% 6/7 85,7%Language skills

integration3/7 42,8% 3/7 42,8%

Interculturality 1/8 12,5% 1/8 12.5%

Evaluation 3/4 75% 3/4 75%

Total: 17/30 63% 17/30 63,2%

Final score: 29/49 60,7% 31/49 65,8%


20/37


76

Phase IV (quantitative stage)

After the application (pilot testing) of the Checklist and the corresponding review of the StudyGuide conducted by one of the participants of the target population, a 24-question evaluationsurvey was administered to her as a final stage of this research project (see Appendix B). This

data-gathering instrument contained some of the questions that were also asked in the previous

stage to the expert, so, it asked about the material’s capacity to achieve impact on learners, tomake them feel at ease, to develop their confidence, to be relevant and useful as well as to pro-mote self-instruction. There were also 9 questions regarding the Checklist’s overall structure, theclarity of its instructions and the quality of its descriptors and an additional set of 4 open-ended

questions, which asked for the contributions, advantages, disadvantages and possible future use ofthis evaluation instrument.

The results from the survey indicated that both materials – the Study Guide and the Checklist –met the objectives for which they were created, receiving a score of three or four points in each ofthe listed items (over a total of four points) (see Table 8). Thus, the former can contribute withaugmenting the teachers’ Knowing-how-to-do skill by providing theoretical and practical inputfor them to be aware about the several possibilities websites can offer for language students, as

well as to know what to look for in these online materials when selecting the most appropriate

one. Indeed, the respondent stated that the Guide serves as a starting point to venture in the digitalworld and bring it to the classroom setting, since she just started gaining knowledge and skills inthe CALL evaluation field. On the other hand, the latter was regarded as a useful instructive toolthat leads to reflect on one’s own beliefs and opinions about the nature of the target language,

language teaching and learning and the capacity of virtual spaces to support and enhance this process. It was highly valued (four points out of four) for not only focusing on the structural fea-tures of a website but also comprising aspects related to language learning and the students’ ac-tive and constructive involvement while surfing on it, as reflected on the evaluation descriptorsthat account for the Communicative Approach principles.

Table 8: Participant’s rating on Study Guide and Checklist


STUDY GUIDE

Materials’ impact100%

(4 points)

- “como docente de inglés y que no he tenido la experi-

encia de enseñar español, la exploración de este sitio me

brindó herramientas para comenzar mi experiencia como

profe digital y desde luego, evaluar objetivamente los

recursos virtuales” (As an English teacher and I have not

had the experience of teaching Spanish, the exploration of

this site provided me with tools to start my experience as a

digital teacher and certainly, to objectively evaluate virtual

resources).

- “algunas de las ventajas de este material son: los con-

ceptos y la aclaración de dudas referentes al sitio web”

(some of the advantages of this material are the conceptsand clarification of doubts concerning the website).

- “las desventaja son: falta de tiempo para explorar este

sitio a cabalidad y así emitir un mejor juicio valorativo”

(the drawbacks are lack of time to fully explore the web-

site and thus, to provide a better value judgement).

Teacher’s development

of self-confidence

100%

(4 points)

Teacher’s feeling at ease100%

(4 points)

Perception towards the

displayed contents as

relevant and useful

100%

(4 points)

Teacher’s self-

investment

100%

(4 points)


21/37

Moncada & Díaz

77


CHECKLIST MODEL

Structure100%

(4 points)

- “ sus fortalezas son: claridad de los descriptores, infor-

mación precisa y concisa. Volvería a utilizar este instru-

mento de evaluación con base a los criterios de este mate-

rial podré formar ideas and bases para mejorar mi proce-

so de evaluación de materiales de lengua” (its strengthsare the clarity of descriptor, the information is precise and

concise. I would use this evaluation instrument again and

based on the criteria displayed on this material, I will be

able to build ideas and fundaments to improve my process

of assessing language materials).

- “definitivamente los descriptores presentados son repre-

sentativos para esta medición porque nos orientan en el

proceso de evaluación de cada principio del enfoque co-

municativo” (definitely, the displayed descriptors are rep-

resentative for this measurement because they guide us in

the evaluation process of each principle of the Communi-

cative approach).

Clarity of instructions87.5%

(3 points)

Quality of the evaluation

descriptors

100%

(4 points)

It is noteworthy that the main disadvantage found by the participant when evaluating the Spanishlanguage-learning website was that it is time consuming exploring it as a whole, because in orderto get an accurate judgment on its worth and quality it is advisable to follow every pathway and

even solve most of its activities and exercises. Table 8 displays the overall results of both evalu-ated materials.

DiscussionThe present study has given a first step toward providing an approach-based Checklist model

proposal that guides teachers to make an informed decision on the most suitable Communicative

language-learning websites, while at the same time they strengthened their Knowing-how-to-doskill. This was accomplished by conducting a four-phase research procedure from which findingssuggested the following.

Primarily, it can be stated that the application of the Checklist to a selected language website andthe usage of the Study Guide by one of the participants of the target population shed some light

on the enhancement of her digital competence, mainly on the Knowing-how-to-do skill. Thus, itcould be observed that the participant was able to make use of her critical and reflective abilitiesto appropriately addresses the evaluation of a website by judging its worth and suitability bothquantitatively (i.e., numerical judgements) and qualitatively (i.e., open-ended commentaries). Infact, comparing the score she gave to the site, it can be evidenced that her ranking only differed in

5.1% from the researchers’ appraisal, which demonstrated her capability to become a third-partyevaluator (Levy & Stockwell, 2006; Nomdedeu, n.d.) despite her lack of experience conducting

this type of assessment and the short time given to do it.On the other hand, the participant recognized how the theoretical insights provided by the StudyGuide were of a great support not only to do the practical task (i.e., evaluate a website) but also to

developing a multidimensional checklist for evaluating language-learning websites coherent with the...

Documents