![Page 1: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/1.jpg)
Media & Learning Design (M&LD)
Research & Evaluation
Presentation to M&LD Steering Committee
By Christos Anagiotos ([email protected])
& Phil Tietjen ([email protected])
April 4 2012
![Page 2: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/2.jpg)
OUR CHARGE
Review prior approaches to evaluation within M&LD
Incorporate evaluation in new M&LD projects more systematically
Investigate state of the art in media & learning evaluation
Investigate what other Universities are doing in regards to the use of Media in online courses.
![Page 3: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/3.jpg)
Evaluation elements
Learning outcomes Learning experience Usability
![Page 4: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/4.jpg)
Research:Media enhances learning
Retention Transfer Cognitive Flexibility
![Page 5: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/5.jpg)
Course Surveys
![Page 6: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/6.jpg)
Course Surveys
Criminal Justice Public Administration
![Page 7: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/7.jpg)
Sample Questions
By watching the library learning tutorials, I learned things I previously did not know about the library
These tutorials will help make my research easier
Video cases allowed me to relate to the content
Video cases helped me in doing my assignments for the course
![Page 8: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/8.jpg)
Focus Groups
![Page 9: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/9.jpg)
Focus Groups
World Campus orientation videos: focused groups with students
M&LD participants: Focused groups with Instructional Designers
![Page 10: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/10.jpg)
![Page 11: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/11.jpg)
Identified Problems
![Page 12: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/12.jpg)
Problem = Low Response Rate
Surveys
![Page 13: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/13.jpg)
Response =
•Embedded Evaluation
•Learning Analytics
![Page 14: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/14.jpg)
![Page 15: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/15.jpg)
![Page 16: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/16.jpg)
![Page 17: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/17.jpg)
![Page 18: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/18.jpg)
Learning AnalyticsLearning analytics is the measurement, collection, analysis and Reportingof data and their contexts,
for purposes of understanding and optimizing learningand environments in which it occurs
![Page 19: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/19.jpg)
Universities that are using Learning Analytics University of Phoenix Cabelas University Baylor University Sinclair Community College University of Baltimore Purdue University Regis University (Library’s Distance
Learning Department) University of Rutgers-Newark (Law
Library) Khan Academy
![Page 20: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/20.jpg)
POTENTIAL OF LEARNING ANALYTICSA. Compare users (e.g. evaluation)B. Predict student performance
(Predictive Analytics)C. Understand student’s needsD. Identify media flaws E. Personalization of educational
material
![Page 21: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/21.jpg)
What data can we collect from current WC sources
1. ANGEL
1. Outside ANGEL- Google analytics- Flash Media Server
![Page 22: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/22.jpg)
What does ANGEL offer?
All data is connected to the student (PSU ID, IP address)
Individual analytics (very complicated to get group analytics)
Examples: Log in time, Log out time, Time
spend in each website, Items downloaded
![Page 23: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/23.jpg)
Google Analytics (Outside ANGEL)
Collect anonymous information about the user
Data is connected to IP Address Data is NOT to the PSU ID
Records much more data than ANGEL The data is presented in a more user
friendly way
![Page 24: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/24.jpg)
Media Flash server(M&LD Videos, Outside ANGEL)Collect anonymous information about
the user Data is connected to IP Address Data is NOT to the PSU ID
We can currently measure: Log in/ log out time Duration per visit, per visitor Streaming duration Play, pause hits
![Page 25: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/25.jpg)
How to make sense of data collected?
EXAMPLES
![Page 26: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/26.jpg)
Example 1: from Media Flash Server:
Course Ed. Leadership 802: Average Length of videos : 10
minutes Average watch time: 4 minutes
![Page 27: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/27.jpg)
Example 1: Possible explanations The content is not valuable or useful to
the viewers The user already got the info from
other sources (readings, discussions etc)
Users are tired or bored after watching the same person talking for more than 4 min.
The content may not be clear enough to the user
![Page 28: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/28.jpg)
Example 2
A video was watched 46 times by 12 users in 7 days.
Possible Explanations: High relevance to the user (e.g. used
for an assignment) Entertaining Confusing
![Page 29: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/29.jpg)
Comparison of data collectedANGEL: Pros: Data connected to the user PSU ID Cons: Very limited amount of data, tough to use
Google Analytics (Outside ANGEL):Pros: Large amount of data & Great detailCons: Data not connected to the individual users’
PSU ID
Flash Media Server (Outside ANGEL):Pros: Decent amount of dataCons: Data for the videos ONLYData not connected to the individual users’ PSU ID
![Page 30: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/30.jpg)
Combining the data we already collect
We can gather :
Data directly connected to each user (PSU ID) from the 3 sources
Group data Data for every activity in the course
website
![Page 31: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/31.jpg)
OTHER FORMS OF DATA THAT WC DO NOT COLLECT
1. Social Network Analysis (Student networks)
2. Record student screens
![Page 32: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/32.jpg)
1. Social Network Analysis (Student Networks)
Students’ social networks facilitate learning processes (Dawson, 2010).
These tools are making learner networking visible
Able to “see” (identify) students who are network-poor (apply interventions)
![Page 33: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/33.jpg)
Visualization of Social Networks Analysis
![Page 34: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/34.jpg)
2. Record student screens e.g. Team Viewer software
![Page 35: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/35.jpg)
What’s next in Learning Analytics? Personalization of educational material Knewton - Pearsons partnership
(video):
(Knewton Adaptive Learning Platform).
![Page 36: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/36.jpg)
Confidentiality issues
How much data we collect?
Students’ consent
Who has access to the data?
![Page 37: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/37.jpg)
Recommendations
Coordinate with IDs to implement regular evaluations
Establish regular meetings with IDs to discuss and analyze results
Develop internal visualization-reporting tools
Make the connection to student performance
Publicize our findings, let people, outside PSU, know what we are doing in M&LD.
![Page 38: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/38.jpg)
Some other ideas for evaluation
![Page 39: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/39.jpg)
![Page 40: Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu](https://reader035.vdocuments.us/reader035/viewer/2022062802/56649ea25503460f94ba57e3/html5/thumbnails/40.jpg)
Thank You
Questions?