cresst onr/netc meetings, 17-18 july 2003, v1 18 july 2003 the workforce learning community bill...
TRANSCRIPT
CRESST ONR/NETC Meetings, 17-18 July 2003, v1
18 July 2003
The Workforce Learning CommunityThe Workforce Learning Community
Bill Bewley and Roxanne Sylvester
UCLA / CRESST
The ONR Workplace
Education Project
2003 Regents of the University of California
2CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Funded by an ONR grant to UCLA CRESST
– “Research-Based Distance Learning to Support Navy Workplace Education”
The Workplace Education The Workplace Education ProjectProject
3CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• The goal– Create a framework for application of
guidelines to courseware evaluation by real users
• The process– Expert review
– Select guidelines, recast as questions
– Test, revise, test—an iterative process
– Base technical quality judgment on instrument and rater reliability
– Report and deliver
Guidelines FrameworkGuidelines Framework
4CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Web-based instruction context
– Accuracy of course content, instructional design, operations, format, technical standards
• Other guidelines
– FSU “Evaluating the Course”: 27 categories, 119 questions
– ASTD “Courseware Certification”: 19 categories, 78 questions
– USMC “Interactive Multimedia Instruction Style Guide”: 12 categories, 50 pages
Guidelines FrameworkGuidelines Framework
5CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• The challenges
– Balancing technical quality with usability
– Creating sufficient user support to ensure rater reliability while not overburdening raters with information
Guidelines FrameworkGuidelines Framework
6CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• KMT Guidelines: Value Added
– Based on rigorous, up-to-date research
– Examination of technical quality of instrument; it may be possible to evaluate courseware well with fewer questions
– Examination of rater reliability
– Examination of scoring training issues related to calibration of rater scores
Guidelines FrameworkGuidelines Framework
7CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• How technical quality is established
– Representative sample population, various course types; variability of raters and courses
– Qualitative data on rater interactions with framework and course components
– Quantitative data on instrument reliability and rater reliability
– Development of rater scoring training materials
Guidelines FrameworkGuidelines Framework
8CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Initial work on KMT guidelines
– Reviewed guidelines in relation to formative evaluation of course components• Originally 58 guidelines, reduced to 52
– Selected guidelines supporting formative evaluation of course components• From 52 to 14 guidelines
– Turned guidelines into questions• From 14 guidelines to 45 questions
– Applied guidelines questions to three courses, reviewed results, revised guidelines selections and questions
Guidelines FrameworkGuidelines Framework
9CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Usability Study 1: Questions
• Courseware Evaluation Questions Usability Study– Videotaped think-aloud– Review of Systems Thinking lesson– Use of paper-based Guidelines questions– Graduate students and temporary workers– August-December 2003– Revision of questions; addition of definitions &
examples to enhance rater agreement
Guidelines FrameworkGuidelines Framework
10CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Usability Study 2: Prototype E-Tool
• Prototype E-Tool Usability Study– Think-aloud screen and audio capture– Examine e-tool functionality– Examine interplay between e-tool and
courseware– Begin to examine rater reliability– Graduate students, instructional designers,
SMEs– January to April 2004– Determine all features required– Use results to program final e-tool
Guidelines FrameworkGuidelines Framework
11CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Usability Study 3: Final E-Tool
• Final E-Tool Usability Study– Think-aloud screen and audio capture– Refine e-tool– Examine rater reliability– Develop rater training procedures– Graduate students, instructional designers,
SMEs– June to September 2004– Deliver October 2004
Guidelines FrameworkGuidelines Framework
12CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Continue formative evaluation
– Evaluate additional WLC courses
– Continue research on validity of framework and rater reliability
– Disseminate framework for review
– Develop worked examples
– Develop rater scoring training materials
– Conduct rater and trainer training
04/05 Tasks04/05 Tasks
13CRESST ONR/NETC Meetings, 17-18 July 2003, v1
• Bill Bewley
– 310-825-7995
• Roxanne Sylvester
– 310-206-5672
For More InformationFor More Information