managing data-driven learning projects

Managing Data-Driven Learning Projects

Aaron E. Silvers & Megan Torrance,

Learning Solutions 2015 Orlando, FLUSE THE TWITTERS!!!!!

@aaronesilvers @MMTorrance #LScon

“ - Robert DeNiro, RONIN (1998)

“I never walk into a place I don't know how to walk out of.”

#KeepingTheEndInMind

Data-Driven Learning Projects

What doThere are a couple of ways data, like xAPI data, opens new doors for Learning & Development projects.

look like?

Adaptive Learning

There’s a lot of promise for

adaptive learning in the

enterprise. What exactly

does “Adaptive Learning”

mean? Well…

Adaptive Learning works like the story of Goldilocks and the Three Bears

Photo by SlyOwl - http://flic.kr/p/j9fFYy

http://flic.kr/p/j9fFYy

First, Goldilocks (“online

self-paced eLearning”)

can choose her porridge…



Online, Self-Paced Learning• Pre-testing preselects content

based on a pre-tests or learner’s history to “test out” of content. This porridge is too hot (arbitrary).

• Linear eLearning delivers the same content to every user. This porridge is too cold (not all that adaptive).

• Micro-adaptive adjusts the sequence of content based on a learner’s reactions in real-time.This porridge is just right.



\

Next, Goldilocks has to

choose among the Micro-

Adaptive systems…



Micro-Adaptive Systems• Preference-based systems adapt to

the style or medium preferred by each learnerThis chair is too big (it’s imaginary).

• Rule-based systems are pre-programmed with closed loops of “if… then…” logic. This chair is too small (“rules”).

• Algorithmically sequenced systems calculate the sequence of content dynamically.This chair is just right.

Photo by Viracocha - http://bit.ly/1A6fcdM

http://bit.ly/1A6fcdM

\

Next, Goldilocks has to

choose among different

types of Micro-adaptive,

algorithmically

sequenced systems…



Algorithmically-Sequenced Systems

• Test-driven systems are based on automated test sequencing. This bed is too hard (so testy).

• Memorization systems prioritizes repetitions based on data about the individual, the learning activity and the interaction between the two. This bed is too soft (narrow applications).

• Proficiency-driven systems look broadly at what someone is trying to learn, pulling from a wide range of content. This chair is just right.

Photo by BlueStarMedia - http://bit.ly/1y0IEVZ

http://bit.ly/1y0IEVZ

Adaptive Learning

Indeed, lots of promise for

adaptive learning but the

way forward has some

bumps.

Challenges

You probably don’t have enough content.

Photo by Tim Ebbs - http://flic.kr/p/aiBFtv

Photo by Brian Corredor - http://flic.kr/p/dLW2aT

The LMS is a fortress for content.

http://flic.kr/p/dLW2aT

IA

Nav

LLR&A Au T Re

Inv Feed Look G

Sta Weed Find M

Use Need DFor more on Content Strategy:

http://bit.ly/contentWrangling

You need a content strategy.















Photo by woodleywonderworks - http://flic.kr/p/4Lx3mh

Defining competencies

while the nature of work

changes is hard.


Adaptive Learning

relies on tracking lots of

learning activity data.

This means way more

interaction design and

development when

creating content.

Learning Analytics

The Big Data approach

…i.e. looking for a needle in

haystacks without knowing

what the needle looks like.


Remember all that

tracking we’re doing?

Where adaptive systems

keep the data mostly out

of view, Analytics are

meant to show you what

the data means.

WHO TAKES WHAT LEARNING PATH? WHEN? WHAT IS THEIR CAREER, AGE, ETC.? WHAT PSYCHOMETRIC TESTING INFORMATION IS AVAILABLE? ITEM ANALYSIS & PRE/POST TESTING? HOW MANY TIMES DID IT TAKE TO “PASS”? WHAT PERFORMANCE DATA DO WE HAVE? WHAT COURSES AND LEARNING PATHS DO THE BEST PERFORMERS TAKE? WHAT KINDS OF COURSES ARE PEOPLE ASKING FOR? WHAT FEEDBACK ARE WE GETTING FROM COURSE EVALUATIONS ACROSS MULTIPLE CURRICULA, TOPICS, INDUSTRY BENCHMARKING, ETC.?

There are many

questions we can ask

right now. Things we

“should” be doing

already, even if we don’t

see immediate benefits.

There’s the stuff we’re already doing…

• We’re collecting data with our LMS databases

• Our CRMs and ERPs and ESNs are capturing a lot of information about employee activity

• HRIS systems and Talent Management have uniform (kinda standardized) ways of describing people.

Making Sense of It All…

• Sentiment Analysis - What do the words people use tell us about their disposition to learn?

• Engagement Analysis - What’s the activity level with learning content?

• Cohort Analysis - Who forms what groups for what reasons?

• Keyword Analysis - How do people seek info & what do they find?

• Conversion Rate - How many people respond (i.e. comment)?

• Amplification Rate - How many times is something shared?

• Applause Rate - How many likes/favorites/bookmarks?

• Economic Value - Short/Long Term Revenue/Cost Savings?

Getting Started While Already in Motion

Matching structured and

unstructured data so you

can actually do something

with it in the real world.

How Our Data Is Structured

STRUCTURED tabular, orderly

UNSTRUCTURED not tabular not tagged

wild wild west




wild wild west

Data-type

metatagging #LSCon

#omg1Distotsawesome




wild wild west

LMS

It Would Be So Simple, Except…

Learning Informatics

The Little Data approach

…i.e. known needles in

small haystacks you make.

Speaking of small haystacks

Speaking of small haystacks

…getting bigger

…and less structured

Activity streams are

interesting … for a while.

And data visualizations are

pretty … for a while.

Phase 2 will involve far

more complex interactions

and data requirements to

define.

The RISC - VTA PDF Annotator

app developed by Float

Mobile Learning allows an

employee to use PDF job aids

like they read/highlight/

annotate on Amazon Kindle.

In this case, there are only

four activities that matter:

highlighting, underlining or

noting a word or passage and

leaving a note for the page.

The data collected has

a known pattern to it,

so specific reports can

be created that add

distinct value back to

the organization about

how they’re used.

…and that data can represented

in different ways for different

uses. In this data visualization,

content owners and designers

can see exactly which pages in a

PDF have the most annotations,

and what kind — helpful if they

need to know where to make

edits.

Data-Driven Learning Projects

So…How do we manage that? This is where y’all ask the tough questions and we share what works :)

managing data-driven learning projects

Data & Analytics