![Page 1: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/1.jpg)
Helen Shen and Chen-hui TsaiHelen Shen and Chen-hui TsaiThe University of IowaThe University of Iowa
Evaluating Reading Proficiency Gain and Program Effectiveness: A Three-tiered Web-Based Reading Assessment Approach
1
![Page 2: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/2.jpg)
IntroductionIntroduction This presentation reports the evaluation
system for “The comprehensive web-based Chinese Reading Program” supported by a DOE grant.
• The program • The evaluation
2
![Page 3: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/3.jpg)
The Program: PurposeThe Program: Purpose• To remedy the acute dearth of appropriate
Chinese reading materials (beyond the class textbook)
• To furnish a superior Chinese reading environment in which reading can take place anytime, anywhere
• To improve users’ vocabulary knowledge, reading skill, and cultural knowledge through extensive reading
3
![Page 4: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/4.jpg)
The Program: The Targeted The Program: The Targeted UsersUsers• College students who are taking
Chinese courses at beginning, intermediate, and advanced levels
• High-school students who are taking Chinese AP courses
• Individuals who work in government agencies or private organizations and who wish to improve their reading skills in a systematic way
4
![Page 5: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/5.jpg)
The Program: Content The Program: Content
• Three levels of reading materials (900 Lessons)
• 90 achievement tests• 15 proficiency tests
5
![Page 6: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/6.jpg)
The Program: Format • Flash-embedded HTML files• Image files
• NJStar files
• Audio HTML files
6
![Page 7: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/7.jpg)
The Program: Academic The Program: Academic strengthstrength• Proficiency-guided definable reading levels• Cultural theme-based reading unit
organization• Fluency-oriented progressive reading skill
building• Emphasizing Intentional vocabulary gain• Supporting Constructive responses• Three-tiered web-based assessment
system
7
![Page 8: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/8.jpg)
The Program: The DesignThe Program: The Design
8
![Page 9: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/9.jpg)
The Evaluation The Evaluation A Three-tiered web-based assessment system
is developed to assess both individual reading achievement and the overall program effectiveness
• The flash-based built-in reading outcome assessment for individual users
• The FileMaker Pro reading database• The Quia web survey tool
9
![Page 10: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/10.jpg)
The Flash-based Reading Outcome The Flash-based Reading Outcome Assessment Assessment for Individual Reading Proficiency Gain for Individual Reading Proficiency Gain • Vocabulary pre- and post tests
• Multiple choice and open-ended Reading comprehension questions
• Reading speed and comprehension rate (REI) report
• Achievement tests
• Proficiency tests
10
![Page 11: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/11.jpg)
The FileMaker Pro The FileMaker Pro DatabaseDatabase This reading database allows us to record:• User’ contact information• User’s reading activities (reading time,
reading comprehension score, reading lessons completed, vocabulary tests score.)
• Built-in on-line survey
11
![Page 12: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/12.jpg)
The Evaluation: Quia Web The Evaluation: Quia Web SurveySurvey
• The Quia survey data allow us to collect users’ additional quantitative and qualitative feedback on certain arising issues during the use of the program.http://www.quia.com/sv/246903.html
12
![Page 13: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/13.jpg)
The Evaluation: Data The Evaluation: Data Analysis Analysis Data from FileMaker Pro • Distribution of Users in the world• Distribution of Reading levels and topics
among users
13
![Page 14: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/14.jpg)
The Evaluation: Data The Evaluation: Data Analysis Analysis
14
Africa America Asia Europe Oceania
South Africa 8
USA1171
China204
United Kingdom84
Australia78
Argentina4
Canada77
Thailand48
Germany46
New Zealand11
Algeria2
Brazil25
Singapore33
France42
Fiji1
Distribution of Users in the world (78 countries; 1771 valid entries)
![Page 15: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/15.jpg)
The Evaluation: Data The Evaluation: Data Analysis Analysis
15
Reading Level L1 Beginning L2 Intermediate L3 Advanced
Frequency of Read lessons
1712 840 189
Popular units U1 / 720打招呼和交往礼节 U11/ 493中国风俗 U 10/ 125中国各民族U5/ 266做饭和吃饭 U1/ 116中国家庭 U20/ 14中国法制和司法U13/ 114旅游 U3/ 42中国音乐和音乐家 U1/ 12中国现代城乡生活
Units not read U8/18/22 U6/8/9/16-30 U2-9/12-16/18/24/26/28/30
Distribution of Reading levels and topics among US users (2741)
![Page 16: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/16.jpg)
The Evaluation: Data The Evaluation: Data analysis analysis Data from The Quia web survey: One
month reading activity and survey (February 2009)
• Distribution of users and age groups in the world
• Qualitative assessment of program effectiveness
• Quantitative assessment of program effectiveness
16
![Page 17: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/17.jpg)
The Evaluation: Data The Evaluation: Data analysis analysis
17
Africa America Asia Europe OceaniaNationality 1 37 35 11 3
Age Range
Age group ranking
21-30 Under 20-above 50
21-30Under 2031-40
Under 20 –above 50
21-3031-40Under 20
Under 20-above 5021-30
Under 20-40
Distribution of users’ nationality and age groups (valid entries: 87), Feb. 09
![Page 18: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/18.jpg)
The Evaluation: The Evaluation: Qualitative Qualitative Qualitative data
18
Purpose Of helpfulness The part like most Problem SuggestionTeacherUse
Improve reading skills
Articles in various topics
The sound file does not work for me
Include pinyin in the reading text
Student use
Improve listening skills
Articles are interesting; very close to real life
more “beginning” level topics
Self -study
Expand vocabulary Audio files Make sound files downloadable
Pleasure reading
Improve reading speed
Njstar files
Improve cultural knowledge
The interface is appealing and easy to use
![Page 19: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/19.jpg)
The Evaluation: The Evaluation: Quantitative Quantitative
19
Questions Average 12 The reading topics presented are interesting and useful for me to learn Chinese
culture. 5.40
13 The structures of the reading materials are clear.
5.80
14 The difficulty level of the reading materials is arranged in a progressive order. 5.43 15 The amount of reading comprehension questions for each reading lesson are
suitable. 5.54
16 The pre- and post-reading vocabulary tests are helpful in vocabulary building. 5.66 17 I am making progress in reading Chinese by using this website. 5.37 18 The pictures presented in the reading text are clear and helpful. 5.34 19 The achievement tests are useful in helping me reviewing reading units. 5.29 20 The proficiency tests are useful in helping me find my reading strength and
weakness. 5.34
21 The sound quality of the audio file is good and I can hear the lesson without difficulty.
5.17
22 The 'help desk' is helpful for me to learn how to use this website. 5.09 23 The website is easy to navigate and imagines are clear.
5.77
Asia: from a 7-point scale ; 35 users
images are clear.
![Page 20: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/20.jpg)
The Evaluation: The Evaluation: Quantitative Quantitative
20
Questions Average 12 The reading topics presented are interesting and useful for me to learn Chinese
culture. 5.59
13 The structures of the reading materials are clear.
5.95
14 The difficulty level of the reading materials is arranged in a progressive order. 5.49
15 The amount of reading comprehension questions for each reading lesson are suitable.
5.70
16 The pre- and post-reading vocabulary tests are helpful in vocabulary building. 5.73 17 I am making progress in reading Chinese by using this website. 5.51 18 The pictures presented in the reading text are clear and helpful. 5.08 19 The achievement tests are useful in helping me reviewing reading units. 5.32 20 The proficiency tests are useful in helping me find my reading strength and
weakness. 5.41
21 The sound quality of the audio file is good and I can hear the lesson without difficulty.
5.30
22 The 'help desk' is helpful for me to learn how to use this website. 5.49 23 The website is easy to navigate and imagines are clear.
5.97
images are clear.
America: From a 7-point scale; 37 users
![Page 21: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/21.jpg)
The Evaluation: The Evaluation: Quantitative Quantitative
21
Population Item number
M SD t Sig.
AsiaUser
12 5.43 .22 -1.13 .272
USAUser
12 5.54 .26
T-test on the comparison of the ratings between Asia and US
![Page 22: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/22.jpg)
Conclusion:Conclusion: The three-tiered web-based assessment
system is an excellent approach conducting the formative assessment for the program, which helps us to know users needs, expectations, and problems they are encountering so that we can adjust the program based on users’ direct and indirect feedback. This system contributes to maintain a long-lasting high quality program.
22
![Page 23: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/23.jpg)
The reading program is free to all The reading program is free to all users!!!users!!!
http://chinesereadingworld.orghttp://collections.uiowa.edu/chinese
Contact information: Contact information: [email protected]@[email protected]@uiowa.edu
23
![Page 24: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/24.jpg)
The EndThe End
![Page 25: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/25.jpg)
HTML fileHTML fileBack
![Page 26: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/26.jpg)
NJStarNJStarBack
![Page 27: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/27.jpg)
Audio filesAudio filesBack
![Page 28: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/28.jpg)
FileMakerFileMakerBack
![Page 29: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/29.jpg)
PretestPretestGo to Post-test
![Page 30: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/30.jpg)
PosttestPosttestBack
![Page 31: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/31.jpg)
multiplemultipleGo to Open Q
![Page 32: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/32.jpg)
Open-ended QuestionsOpen-ended QuestionsBack
![Page 33: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/33.jpg)
Final ReportFinal ReportBack
![Page 34: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/34.jpg)
Achievement TestAchievement Test
Back
![Page 35: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/35.jpg)
Proficiency TestProficiency Test
Back
![Page 36: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/36.jpg)
FileMakerFileMakerBack
![Page 37: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/37.jpg)
Built-in SurveyBuilt-in SurveyBack
![Page 38: Helen Shen and Chen-hui Tsai The University of Iowa](https://reader038.vdocuments.us/reader038/viewer/2022103007/56815d04550346895dcb04e8/html5/thumbnails/38.jpg)
Quia SurveyQuia Survey