kresge business administration library 1 evaluating video tutorials: measuring excellence and...
Post on 19-Dec-2015
216 views
TRANSCRIPT
Kresge Business Administration Library 1
EVALUATING VIDEO TUTORIALS:Measuring Excellence And Outcomes
Sally Ziph
Instruction Coordinator
Kresge Business Administration Library
Stephen M. Ross School of Business
University of Michigan
Elizabeth Beers
Web Developer, MPublishing
University of Michigan Library
University of Michigan
Kresge Business Administration Library 2
OUR VIDEOS
Kresge Business Administration Library 3
Rationale
• New MBA Consortium students needed company and industry information to prepare for interviews with recruiters.
• Videos were intended to help complete a specific task: build lists in OneSource, find career information in Vault, etc.
Kresge Business Administration Library 4
The Process
• Librarians created scripts based on frequently asked questions.
• Technologies included Audacity for audio and Camtasia for screen capture.
• Videos were integrated into the library website and advertised through email.
Kresge Business Administration Library 5
A Challenge
How do we evaluate the effectiveness of non-interactive task-based point-of-use video tutorials?
Kresge Business Administration Library 6
EVALUATING OUR VIDEOS
Kresge Business Administration Library 7
Best Practices
• Pre/Post tests• A/B tests• Focus groups and user interviews• Surveys• Statistics
Kresge Business Administration Library 8
Pre/Post Tests
From the literature:• Used to measure learning that takes place when
students view tutorial• Most suitable for assessing tutorials that seek to
teach a specific skill • Can be difficult to write questions that will give good
comparison data
Kresge Business Administration Library 9
Pre/Post Tests
For our purposes:• Tests intended to help with a task, rather than teach
a skill
Kresge Business Administration Library 10
A/B Tests
From the literature:• Used to compare outcomes from versions of the
same tutorial:– In person vs. online– Different instructional techniques
• Also useful for gathering subjective feedback• Difficult to tease out whether variance in outcomes
is due to content or presentation
Kresge Business Administration Library 11
A/B Tests
For our purposes:• Abbreviated project timeline made comparison
testing impossible.• Useful for refining tutorials
Kresge Business Administration Library 12
Focus Groups and User Interviews
From the literature:• Easy ways to get feedback from a variety of
viewpoints• Can be conducted iteratively and at any stage of the
development cycle• Require an experienced facilitator
Kresge Business Administration Library 13
Focus Groups and User Interviews
For our purposes:• Abbreviated project timeline made iterative
assessment impossible• Useful for future assessment
Kresge Business Administration Library 14
Surveys
From the literature:• Easy method for data collection• Easy to formulate questions – but can be difficult to
formulate good questions• Results often reflect subjective preferences
Kresge Business Administration Library 15
Surveys
For our purposes:• Quick method for getting feedback from the target
user group• Not the best data, but better than none.
Kresge Business Administration Library 16
Statistics
From the literature:• Raw numbers from Google Analytics or built in
utilities measure use.• Can be used to validate data from other
methodologies• Does not measure usefulness or quality of tutorial
content
Kresge Business Administration Library 17
Statistics
For our purposes:• Data we were collecting anyway may be useful for
future videos.• Most popular:
• Career resources• Company information• Industry reports
Kresge Business Administration Library 18
OUR EVALUATION
Kresge Business Administration Library 19
Our Survey
• Managed using Qualtrics• Distributed by targeted email to 45 students • Questions included:
– Did you watch one or more of the Kresge database videos?
– How useful were the instructional videos for the following situations?
– What changes would you make to the Kresge videos to make them more useful?
Kresge Business Administration Library 20
Results
• 13 out of 45 students responded (28%)• 72% of respondents did not use the videos:
– Videos were perceived as inconvenient or irrelevant– Students wanted other topics
Kresge Business Administration Library 21
Conclusion
• We need to clarify:– What the Kresge Library can do to help Consortium
students– Why these videos are worth their time
• We need to send reminders at strategic times throughout the program.
Kresge Business Administration Library 22
NEXT TIME?
Kresge Business Administration Library 23
In the Future
• Continue gathering statistics• Use focus groups to refine tutorials and identify new
topics• Consider testing methodologies to measure learning
and usefulness
Kresge Business Administration Library 24
QUESTIONS?
24