media & learning design (m&ld) research & evaluation presentation to m&ld steering...

Media & Learning Design (M&LD)

Research & Evaluation

Presentation to M&LD Steering Committee

By Christos Anagiotos (cxa5065@psu.edu)

& Phil Tietjen (prt117@psu.edu)

April 4 2012

OUR CHARGE

Review prior approaches to evaluation within M&LD

Incorporate evaluation in new M&LD projects more systematically

Investigate state of the art in media & learning evaluation

Investigate what other Universities are doing in regards to the use of Media in online courses.

Evaluation elements

Learning outcomes Learning experience Usability

Research:Media enhances learning

Retention Transfer Cognitive Flexibility

Course Surveys

Course Surveys

Criminal Justice Public Administration

Sample Questions

By watching the library learning tutorials, I learned things I previously did not know about the library

These tutorials will help make my research easier

Video cases allowed me to relate to the content

Video cases helped me in doing my assignments for the course

Focus Groups

Focus Groups

World Campus orientation videos: focused groups with students

M&LD participants: Focused groups with Instructional Designers

Identified Problems

Problem = Low Response Rate

Surveys

Response =

•Embedded Evaluation

•Learning Analytics

Learning AnalyticsLearning analytics is the measurement, collection, analysis and Reportingof data and their contexts,

for purposes of understanding and optimizing learningand environments in which it occurs

Universities that are using Learning Analytics University of Phoenix Cabelas University Baylor University Sinclair Community College University of Baltimore Purdue University Regis University (Library’s Distance

Learning Department) University of Rutgers-Newark (Law

Library) Khan Academy

POTENTIAL OF LEARNING ANALYTICSA. Compare users (e.g. evaluation)B. Predict student performance

(Predictive Analytics)C. Understand student’s needsD. Identify media flaws E. Personalization of educational

material

What data can we collect from current WC sources

1. ANGEL

1. Outside ANGEL- Google analytics- Flash Media Server

What does ANGEL offer?

All data is connected to the student (PSU ID, IP address)

Individual analytics (very complicated to get group analytics)

Examples: Log in time, Log out time, Time

spend in each website, Items downloaded

Google Analytics (Outside ANGEL)

Collect anonymous information about the user

Data is connected to IP Address Data is NOT to the PSU ID

Records much more data than ANGEL The data is presented in a more user

friendly way

Media Flash server(M&LD Videos, Outside ANGEL)Collect anonymous information about

the user Data is connected to IP Address Data is NOT to the PSU ID

We can currently measure: Log in/ log out time Duration per visit, per visitor Streaming duration Play, pause hits

How to make sense of data collected?

EXAMPLES

Example 1: from Media Flash Server:

Course Ed. Leadership 802: Average Length of videos : 10

minutes Average watch time: 4 minutes

Example 1: Possible explanations The content is not valuable or useful to

the viewers The user already got the info from

other sources (readings, discussions etc)

Users are tired or bored after watching the same person talking for more than 4 min.

The content may not be clear enough to the user

Example 2

A video was watched 46 times by 12 users in 7 days.

Possible Explanations: High relevance to the user (e.g. used

for an assignment) Entertaining Confusing

Comparison of data collectedANGEL: Pros: Data connected to the user PSU ID Cons: Very limited amount of data, tough to use

Google Analytics (Outside ANGEL):Pros: Large amount of data & Great detailCons: Data not connected to the individual users’

PSU ID

Flash Media Server (Outside ANGEL):Pros: Decent amount of dataCons: Data for the videos ONLYData not connected to the individual users’ PSU ID

Combining the data we already collect

We can gather :

Data directly connected to each user (PSU ID) from the 3 sources

Group data Data for every activity in the course

website

OTHER FORMS OF DATA THAT WC DO NOT COLLECT

1. Social Network Analysis (Student networks)

2. Record student screens

1. Social Network Analysis (Student Networks)

Students’ social networks facilitate learning processes (Dawson, 2010).

These tools are making learner networking visible

Able to “see” (identify) students who are network-poor (apply interventions)

Visualization of Social Networks Analysis

2. Record student screens e.g. Team Viewer software

What’s next in Learning Analytics? Personalization of educational material Knewton - Pearsons partnership

(video):

(Knewton Adaptive Learning Platform).

Confidentiality issues

How much data we collect?

Students’ consent

Who has access to the data?

Recommendations

Coordinate with IDs to implement regular evaluations

Establish regular meetings with IDs to discuss and analyze results

Develop internal visualization-reporting tools

Make the connection to student performance

Publicize our findings, let people, outside PSU, know what we are doing in M&LD.

Some other ideas for evaluation

Thank You

Questions?