yoshida thesis

1

University of Aizu, Graduation Thesis. March, 2012 s1170173

Abstract There is scant literature in language studies

suggesting the efficiency with which EFL learners with low English language proficiency are able to process English website content, and their opinions on the same. Structured website analysis has the potential to be a stimulating exercise because it addresses various types of questions that are very unique to the medium and might not be immediately applicable to other contexts. This article reported on an experiment where 41 junior-level Japanese EFL readers were asked to study an English tourism website closely, and then answer a set of questions which focused entirely on readers’ ability to excavate information by navigating through the website content. Further, several survey questionnaires (QUIS, CSUQ, and MPRC) were used wherein readers were asked to self report their levels of comfort with the website, and the words they would choose to describe their feeling about the website. Results suggested relatively higher levels of proficiency ordering the steps required to navigate and search for specific information from the given website. Moreover, readers were mostly comfortable searching for information from all over the website. Self-reports suggested relatively moderate levels of comfort with different tasks and access features (overall reaction to the website; webpage design, terminology and website information; learning; website capabilities) related to the website. Finally, data shows that the number of positive words chosen to describe the tourism website is way more than the negative words chosen.

1. Introduction Readers of a traditional printed text read in order

of ascendency. However, website readers must choose between hyperlinks which could possibly take them to different related or unrelated sections of the broader webpage, and often to other web pages as well. Carrell (1987) suggests that organization of a text affects reading comprehension as well as recall. She said that signaling devices in the text help

readers recall the information in the hierarchical structure of the text and improves text comprehension. The signaling devices help connect one part of the text to another and improve content comprehension (Kintsch & van Dij, 1978). However, such conclusions on reading ability and comprehension might be completely lost when reading hypertext-rich websites are accessed. In this context the hierarchical structure of the text is completely lost because of the skipping around to different parts of the text that is involved (Charney, 1994).

The efficiency in reading depends on readers’ schema which helps detect the structure of a text. The readers’ schema functions as an abstract script and the reader expects the script to match the text he/she reads (de Beaugrande, 1980). The readers’ schema is often influenced by the website’s graphic manifestations that lead to a shift from the centrality of text (Burbules, 1998). However, the author suggested that image does not replace the need for text. Rather, the idea behind use of graphics is to summarize the importance of the message that text communicated, but in a quick glance. Graphics tend to serve a pointing function. Boardman (2004) suggested that webpage creators often choose short, dense phrases, rather than full sentences to summarize the core of the message.

However, we are not entirely sure of how this experience translates to an EFL context where readers’ English proficiency is at a very low level. On a very specific level the question is the efficiency with which EFL learners with low English language proficiency are able to process English website content, and what is their opinion about the website?

Structured website analysis with open-ended feedback as a tool for foreign language learning has the potential to be a stimulating exercise because it addresses various types of questions that are very unique to the medium and might not be immediately applicable to other contexts. Very little is known about user’s ability to provide open-ended feedback on website content (Elling et al., 2012). Moreover, in an EFL context, website analysis might deal with a range of issues related to learning difficulties resulting from lack of language proficiency, variable

EFL Student’s Ability for Website Information Comprehension and Perceptions on Website Usability

Makoto Yoshida s1170173 Supervised by Prof. Debopriyo Roy

2


use of cognitive and meta-cognitive strategies for information comprehension on the web (Lomicka, 1998) or in an intensive English learning context (Hong-Nam and Leavell, 2006), individual differences in learning with hypermedia (Knight, 1994), besides complexity in the web content itself. These above-mentioned areas of research directly or indirectly contribute to our general understanding about how L2 learners perceive English website content and design, and how specific reading strategies (self-reported) might impact design decisions.

As part of the experiment reported in this article, Japanese EFL readers were asked to study a website closely, answer design questions related to the website and how it is structured, and then answer a set of questions which focused entirely on the content of the website. The purpose behind this questionnaire was to judge whether readers actually have read through the website, and could identify where the information is placed in the website. Answering the questions in this questionnaire successfully demanded both reading ability and identifying links that takes the reader to the required information. Further, readers were asked to self report what they thought specifically about the website with different questionnaires, and the words they would choose to describe their feeling of the website. This paper is solely based on exploring readers’ ability to identify and read website content successfully, and self-reports of their feelings of the website. The purpose behind this activity was to identify what bothered them about the website information comprehension, what they liked about the navigation, their levels of motivation and feelings about the website etc. This will allow L2 researchers to get some sense of how EFL readers with low language proficiency approach the task of reading an English website.

2. Method Participants (N=17) for the first stage of the

analysis (Week # 1) are junior level students (age group: 18-20 years) in their third year undergraduate program specializing in computer science in a Japanese technical university.

First Week: During the 1st week of the actual experiment, a randomly selected section of the students analyzed the Belize tourism website based on the 8 open-ended design questions about the website. However, the analysis of how efficiently students

could answer the design questions is beyond the scope of this study.

Second Week: During the 2nd week, readers (all students in the class - N = 59) were given a set of questionnaires to answer on the same Belize tourism website. The websites focused on three different aspects related to information comprehension and online reading strategies for an EFL context. The questionnaires focused on three different aspects. • Their ability to navigate through and look for

information from the website. • Their ability to meaningfully self-report on the

usability of the website in terms of navigation, content, and organization.

• Their ability to meaningfully self-report on their reading strategies when analyzing the Belize tourism website.

All the questionnaires with the above-mentioned focus were completed in class.

Instruments:

The Belize tourism website was chosen with the following reasonable conditions in mind. • The content in the website is not text heavy and

clear navigation is possible. • Information could be searched directly from the

home page. • Attractive pictures are available to keep the reader

engaged in the task of finding information. • Japanese version of the webpage is NOT available

so that readers are forced to look for information from the English version alone.

The instructions for the first week of the assignment (where readers had to respond to 8 open-ended questions) were all in English, largely because readers already had practice from the weeks before regarding what is expected of them. For the second part of the experiment (held during the 2nd week), all the questionnaires were administered in Japanese, to ensure proper understanding of the content and instructions. Readers were handed out an instruction sheet in both Japanese and English explaining what is exactly expected of them for the task. The second part of the experiment comprised of several tasks. - 1st Task: To check for readers’ ability to navigate through and look for information from the Belize tourism website, a website information comprehension questionnaire was designed specific to the Belize tourism website. The questionnaire had to answer 8 questions. For the 1st question, readers were given a specific scenario where they were asked to find out a

3


hotel in a specific location in a given price range. Readers were given a list of steps they had to follow in order to find the information from the Belize tourism website. For all other questions except one, they had to choose the correct option regarding information that is available in the website. They can only choose the correct option when they have actually tried to look for information from the website. This should not be called a questionnaire in the true sense, because the questions were geared towards searching for information from the website and were completely unrelated from each other, with no pattern or similarity or were not part of any specific sub-group of analysis. - 2nd Task: A set of three usability questionnaires was handed out as part of this task. The questionnaires are named as QUIS, CSUQ, and Microsoft Product Reaction Card respectively. The first two questionnaires used a Likert scale for each question. The review of the literature has more details about these questionnaires. These questionnaires are designed not only to elicit readers’ impression of the Belize tourism website in terms of content, navigation and usability, but also judge the extent to which their impression on items matches or deviates from the judgement made by the more experienced coders. Assessing Website Usability: According to the IBM technical report, most usability evaluations gather both subjective and objective quantitative data in the context of realistic scenarios-of-use, as well as descriptions of the problems representative participants have trying to complete the scenarios. Subjective data are measures of participants' opinions or attitudes concerning their perception of usability. Objective data are measures of participants' performance (such as scenario completion time and successful scenario completion rate) (Lewis, 1993). Usability is recognized as an important quality factor of any modern website (Avouris et al., 2003). Literature on web usability has reported the use of variety of questionnaires for reporting perceived usability of interactive systems (Tullis & Stetson, 2004). As part of this study, several questionnaires were used, based on the study performed by Tullis and Stetson (2004) on comparison of questionnaires for assessing website usability. QUIS (Questionnaire for User Interface Satisfaction) was developed at the University of Maryland, and composed of 27 questions. However, for website context, we adopted 24 questions that are appropriate (http://www.isr.umd.edu/news/news_story.php?id=4099). This questionnaire was used for obtaining self-

reported impression of the website on five categories of information. They are (1) Overall Comprehensive Evaluation of the website (2) Website design (3) Terminology and website information (4) Learning from the website and (5) Website capabilities. CSUQ (Computer System Usability Questionnaire) also used for this study was developed at IBM and was originally developed for computer systems. However, it was adopted for websites in this context (http://drjim.0catch.com/usabqtr.pdf). The purpose of this website is to get self-reports on overall ease of using the website, and related efficiency of use. Finally, the Microsoft Product Reaction Card was used for this study, to obtain qualitative reviews on the impression about the website. A modified list, based on originally developed 121 words, was presented to the user and was asked to choose the words that best represented their interaction with a website. Readers were free to choose any many or as few words as they deemed appropriate (Benedek and Miner, 2002). The above questionnaires were used for the study as reported in this article. For the Website Information Comprehension Questionnaire, QUIS questionnaire, CSUQ and Microsoft Product Reaction Card: The three coders were independently asked to grade the questionnaires that readers completed during the second week analysis and reporting. For the website information comprehension questionnaire, grading was straightforward in terms of whether the reader could correctly identify the correct option after searching for information from the website. However, for each question, I asked the coders to consult amongst themselves as to whether there is some information that is truly difficult to locate, or if there are alternate ways of finding the same information. However, for each reader, and for each question there was only one grade assigned (2=correct or 0= incorrect). The project coordinator offered coders with expert advice as to how they can search for answers to the questions asked in the questionnaire, and then grades the responses for accuracy. For this questionnaire, each response was graded only once by three coders who worked together to decide on a grade. For the QUIS, CSUQ and Microsoft Product Reaction Card questionnaires, coders recorded the self-reports by the readers coming together as a group. This group activity was important to make sure that no wrong data was entered into the system during data collection.

4


3. Findings 3.1. Website Information Comprehension Questionnaire

The website information comprehension questionnaire showed relatively moderate levels of accuracy, indicating the fact that text comprehension from the web pages, understanding the navigation styles of the Belize website, understanding the linked pages, etc. were moderately successful. However, since the questions in this questionnaire were designed to test whether readers are actually capable of searching through the information in the website, the different questions in the questionnaire can’t be considered to be testing anything specific. Rather, different questions tested searching ability for different types of information from the website, and in no order of complexity or similarity of content. This was evident from the reliability test done for the questionnaire, with a Chronbach’s alpha value of .086 (from Q2 ~ Q9). So, it probably won’t make sense to term this as a questionnaire. Instead the purpose would be to see the accuracy with which individual questions were answered and relate it to the self-reports in other questionnaires.

For most questions, students scored in the range of 1.19 ~ 1.97 showing some variability in the scores. This indicates that some questions were relatively easier to answer, while some were more difficult depending on how much the information had to be searched from the linked pages in the website. Table 1 shows the details of the website information comprehension data. On an average, for questions 2 ~ 8, the data shows accuracy of searching for information from the website at around 83%. For the first question which asked readers to sequence steps when searching for specific information from the English website, the accuracy could be pegged at 86.6%. Table 1 shows the mean accuracy for the different questions asked in the website information comprehension questionnaire. The mean accuracy scores from Q2 ~ Q8, shows high values for Q2 and Q3, but a drop in the mean score for Q3, and Q4, and more overall fluctuations.

Data shows significant correlation vales between accuracy scores for the different questions asked. Results indicate statistically significant correlation values involving Q2, Q7, and Q8. This suggests comparable amounts of accuracy between the questions, indicating searching for some kinds of information and/or levels of difficulty when searching for information from the website were

comparable for the questions concerned. We also wanted to test if there is any statistically

significant difference between the accuracy scores in the website information comprehension questionnaire. Results of the Friedman test suggested that responses are indeed significantly different (χ2 (2) = 78.120, P = 0.000). Since p-value = 0.00 ≤ 0.01 = α, we rejected the null hypothesis that there is no difference between the mean ranks for the accuracy scores.

Following the website information comprehension questionnaire, participants self-reported their impression of the software on different categories using the CSUQ questionnaire.

3.2. CSUQ and QUIS Questionnaire Results

The purpose of the CSUQ Questionnaire was to explore self-reported ease of using the Belize tourism website. Results show a significant correlation in most cases between all questions asked in the questionnaire, with a handful of exceptions. QUIS questionnaire had 5 different categories namely (1) overall reaction to the website (2) screen (3) terminology and website information (4) learning and (5) website capabilities. Table 5 shows the mean and standard deviation values for all 5 categories in the QUIS questionnaire. For this questionnaire, in this case of reliability testing, a Chronbach’s alpha of .687 was obtained for category 1, .809 for category 2, .637 for category 3, .785 for category 4 and .731 for category 5. Data also reported the percentage levels of agreement with the different categories in the QUIS questionnaire. 100% agreement shows a mean value of 5 on a category in a 1 ~ 5 Likert scale whereas a mean value of 1 shows 0% agreement with a statement. Results here suggest 59.4% agreement on the” overall reaction to the website” category, which goes up to 64.8 for “web page design” category, but then slides back to around the 44 ~ 54% level for the other three categories. Data also show the number of cases in various categories where responses to questions show significant correlation with responses to other questions either in the same category or in a different category. Except for question 2, 23, and 24, all questions demonstrated significant levels of correlation with all other questions across the questionnaire. This result suggests significant similarity in what website readers think about the different features of the website, ease of learning from the website, ease of reading and using the website interface, navigating through the website, website accessibility and so on.

5


Next, we wanted to explore if there is significant similarity or difference between the self-reported scores on the Likert scale in the CSUQ questionnaire. Null Hypothesis: Mean Ranks for all the self-reported scores on the CSUQ questionnaire are equal; Alternative Hypothesis: Not all the mean ranks are equal. Table 7 shows us a Friedman analysis for responses to all the questions combined.

The Friedman non-parametric statistical analysis was performed to find out if there is an overall statistically significant difference between the mean ranks of the self-reported scores in the different questions asked. This test tells us whether there are overall differences between self-reported scores but does not pinpoint which questions in particular differ from each other. To do this we need to run post-hoc tests, but post-hoc analysis was not considered as part of this study. Results suggest that there was a statistically significant difference in self-reported sores for a combination of all the questions asked in the CSUQ questionnaire, depending on the overall weight of how readers felt about accessing the English website, χ2 (2) = 126.626, P = 0.000. Since p-value = 0.00 ≤ 0.01 = α, we reject the null hypothesis.

Next, data explored if there is significant similarity or difference between the self-reported scores on the Likert scale in the QUIS questionnaire for the combined responses on the 5 different categories studied. Friedman analysis was done for each of the 5 categories in the QUIS questionnaire and was considered separately. This test tells us whether there are overall differences between self-reported scores within a category, but does not pinpoint which questions in particular differ from each other. To do this we need to run post-hoc tests, but post-hoc analysis was not considered as part of this study.

Results suggest that there was a statistically significant difference in self-reported sores for a combination of all the questions asked in the “overall impression of the website” (χ2 (2) = 21.532, P = 0.000), “web page design” (χ2 (2) = 24.818, P = 0.000), “terminology and website information” (χ2 (2) = 60.670, P = 0.000) and “website capabilities” (χ2 (2) = 75.683, P = 0.000) categories. However, data shows there was no significant difference between responses when it comes the “learning” (χ2 (2) = 5.769, P = .217) category. Since p-value = 0.00 ≤ 0.01 = α, we reject the null hypothesis.

3.3. Microsoft Product Reaction Card Results

For the MPRC questionnaire that was handed out

to participants in the final stage, following interaction with QUIS and CSUQ questionnaire, we obtained the following results. In total, 116 words were selected by the readers (out of 121). Readers chose 77 positive words and 41 negative words. Also, the total frequency of positive words chosen was way more than the total frequency of negative words chosen. For the 10 most frequently chosen words, 8 of these words are positive ones and 2 of these words has a negative sense.

4. Discussion

Question 1 in the website information comprehension questionnaire intended to explore the efficiency with which readers are able to follow and confirm the sequence of activities that is necessary for finding certain information. A score of 8.66 in a scale of 10 for all correct answers, suggest high levels of efficiency in following the steps correctly from an English webpage. Also, scores on other questions (related to searching for information from Belize website) in this questionnaire suggest moderate mean scores. However, there is probably hardly any literature to suggest that ability for search in an English context, from an English website might also indicate higher levels of English content comprehension ability.

Further, not a significant number of correlations were observed between accuracy scores on the website information comprehension questionnaire and self-reported scores on the CSUQ and QUIS questionnaire. This goes on to suggest that self-reports on the questionnaires had a general outlook about the website and might not have been specific to the accuracy scores. But, the Friedman test of accuracy scores did suggest a significant variation between the results of accuracy scores. This goes on to show that in significant number of cases, some readers could and some could not search through the required information and locate the answer in this multiple-choice website information comprehension questionnaire. This possibly indicates towards the fact that individuals with high reading and content comprehension ability might have a higher statistical probability to search for and locate answers from English website more successfully.

The CSUQ questionnaire made positive statements in different categories and asked readers to respond to what extent they agree with the statement. Self-reported scores in the range of 3 ~ 4.3 suggest moderate agreement with the statements. This also

6


indicates towards the fact that readers were not highly comfortable with the website as for its features, design, etc. Further, we see a very high number of significant correlations between the self-reported scores in the CSUQ questionnaire, suggesting similar range of responses for most questions. However, Friedman test suggests a different result because this non-parametric analysis considered the mean rank for each question, and all the results were considered together, and not in a bivariate context.

Finally, we could clearly see that the number of positive words chosen to describe the Belize tourism website is way more than the negative words chosen, and also the positive words are chosen with more total frequency. This goes on to show a positive overall impression of the website. However, this should just be taken as an indicator and whether it translates to overall better comprehension of the website information, and general learning is a different matter altogether.

5. Conclusion

From this study we see conclusive evidence that there is wide variability in the efficiency with which readers in this EFL context are able to analyze an English website. Future studies could systematically focus on testing accuracy related to different aspects of searching and reading information from English websites. These aspects could be directly related to demonstrating efficiency in understanding website design, website capabilities / features, learning aspect of the website, terminology and website information etc. Further, these accuracy scores would then be matched to self-reported scores on the CSUQ and QUIS questionnaire. This exploratory analysis as discussed in this article could help us obtain initial data on how EFL readers in a typical context as this would perform with English website information and the type of impression they have about the website. This will help us judge their levels of proficiency and the types of English websites they could be exposed to for various kinds of assignments, task-based language learning etc.

6. References

Carrell, P.L. (1987). Content and formal schemata in ESL reading. TESOL Quarterly, 21, 461-481. 1987 TESOL International Association Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press. Charney, D. (1994). The effect of hypertext on processes of reading and writing. New York: Modern Language Association. de Beaugrande, 1980 Text, Discourse, and Process: Toward a Multidisciplinary Science of Texts Nicholas C. Burbules. "Rhetorics of the Web: Hyperreading and critical literacy" in Page to Screen: Taking Literacy into the Electronic Era, I. Snyder (editor), Routledge, London, 1998. Boardman, M. (2004). The Language of Websites (Intertext). New York: Routledge Elling et al., (2012). Users’ Abilities to Review Web Site Pages. Journal of Business and Technical Communication Volume 26 Number 2 (April 2012) Lomicka, L. (1998). "To gloss or not to gloss": An investigation of reading comprehension online. Language Learning & Technology, 1(2), 41-50. Retrieved May 1, 2005, from Lomicka, L. (1998). "To gloss or not to gloss": An investigation of reading comprehension online. Language Learning & Technology, 1(2), 41-50. Retrieved May 1, 2005, from http://llt.msu.edu/vol1num2/article2/default.html. Hong-Nam, K., & Leavell, A. G. (2006). Language learning strategies of ESL students in an intensive English learning context.System, 34(3), 399–415. University of North Texas, Denton: Department of Teacher Education and Administration

Knight, J. (1994). Internationalization: Elements and checkpoints (Research Monograph, No. 7). Ottawa, Canada: Canadian Bureau for International Education.