Organizing to Ret Analytics RightVince Kellen, Ph.D.Senior Vice Provost, Analytics and Technologies Vince.Kellen@uky.eduThis is a living document subject to substantial revision! September, 2014

SilosAre recursive

Get reproduced across time and space reliably, without effort

Arise naturally due to human sociological/biological tendencies

It takes constant effort to mitigate their adverse effects

Sharing data and analysis widely requires a reconceptualization of silo structures

2

Organization dysfunction

Information as power

Defensiveness

Data hoarding

Process separation

Empire building

Excessive control

Fear of scrutiny

Loss of power

3

We are competitive animalsInformation becomes a [tool, weapon]

We instinctually manage information to enhance our competitiveness

Competition relies on information hiding

IT tools become part of our body

How we personally utilize information is part of our biological heritage. This is hard to change, if at all

4

Shift from production concerns to consumption ones Production

• Collecting, integrating, cataloging, categorizing, transforming, abstracting, analyzing, model-building, visualization, dashboarding, distributing, publishing. If you build it they will come (hopefully)

Consumption• Motivating, collaborating, expressing, integrating, improving action, increasing ambition, desire, recognition.

If they build it everyone will come

5

A Proposed Analytic Maturity Scale

6

Our process

7

A. Merging of mobile and BI strategy

B. Merging of IR and BI units

C. Super high-speed infrastructure

D. Single analytic value chain

E. Analytics community of practice

F. Data transparency

G. Community sourcing and norming

H. Community rules of etiquette8

Our Community of Practice Rules of EtiquetteBe safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please have good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and university information appropriately.

Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what absolutely needs to be private, the way the university runs it's business often involves multiple colleges and units at the same time. Don't use your access to take unfair advantage of another unit.

Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data management processes that might need to be changed to improve data quality.

Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking great care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In time, we can accommodate more ways of looking at the same data while not interfering with other views or taxonomies.

Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a few select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the universitycan improve accordingly.

9

Organizing IT

Our organizational model makes a big difference. Other universities fail to take advantage of a tool like this for purely political reasons

Making key data transparent to all does not help those who made their living being the data ‘go to’ person

We had to merge two units (Institutional Research and Business Intelligence), losing 1/3 of the staff. This let us hire three data scientists with different analytic backgrounds

The tool let the staff transition their skills

10

What we have done and what we would like to do First steps over the past year

• Mobile micro-surveys: Learning from the learner. In one year, 134,458 surveys harvested. Survey response rates are holding at about 40%. We can instantly analyze all responses for retention and progression issues

• Student enrollment, retention, demographics, performance, K-Score, facilities utilization, instructor workload, student revenue and financial aid, student progression and more

• High speed, in-memory analytics architecture. Lowest level of detail, maximum semantic expressiveness, one-second per click for analyst are key design philosophies

• Open data and organizational considerations

Coming down the road?• Micro-segmentation tool to enhance user and IT productivity, develop personalized mobile student

interaction/intervention• Models for learner technographics, psychographics, in addition to behaviors, performance, background• Advanced way-finding for streaming content like lecture capture• Content metadata extraction and learner knowledge discovery• Real-time measures of concept engagement and mastery• Real-time learner recommendations and support engine • Use graphing algorithms to perform more sophisticated degree audit what ifs

11

Our analytics technology infrastructure roadmap 2014-15

12

List builder Iteratively query any/all fields of your choosing, linking in an AND or OR fashion

Combine different lists using SET manipulations

Refresh lists regularly (nightly or otherwise)

Apply the set name as a filter on ALL models

This provides advanced filtering and combining that works regardless of the user interface

Our AA team can build and maintain Lists easily. So can some users

Since lists are refreshed nightly, we can keep track of each time a student (or other entity) as it added or removed from a list

We can develop workflow apps using this. Backend, front-end agnostic

13

List builder example

14

List builder example

15

2009 2010 2011 2012 2013 2014 2015Academic year

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

Avg

19

109

107

8

107

160

161

118

36

154

55

155

165

47

162

14

19

51

109

162

165

56

Fast/Slow ProgressionStudent headcount

8

50

100

165

Cohort YearFall 2008

Fall 2009

Fall 2010

Fall 2011

Fall 2012

List builder visualization exampleFound all students who take a lot of classes at one point in their career and then took less classes at another point in their career.

Interpretation: These students start with a bang but fade at the finish

How long did this analysis take? Start to finish with this visualization:

25 minutes

16

K-Feed:

Intelligent, personalized alerts, news, reminders

17

Identifying smaller segments of students

In addition to our work on difficult student cases, we needed to find a way to reach a ‘murky middle’ group of students

Identify students who are just as likely to come back as they are not

The predicted reenrollment was about 50%

After interventions, the actual enrollment was about 65%

The whole enchiladaPersonalize learning, learning analytics and IPAS analytics

into one real-time architecture• Real-time personalized interactions

• Target on-demand peer tutoring based on student’s profile• Deliver micro-surveys and assessments to capture additional information needed to improve personalization• Give students academic health indicators that tell students where they can improve in study, engagement,

support, etc.• Let students opt their parents in to this information so the family can support the student • Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user

temporal proclivities, mix and match messages across learning, support and progression areas

• Allow for open personalized learning• How content gets matched to students is psychologically complex • Several theories of how humans learn give many insights• Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive

reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective memory, etc.

• We need an open architecture to promote rapid experimentation, testing and sharing of what works and what doesn’t

University of Kentucky

Herding cats We shared with everyone that we are

building the bridge as we walked on it

We established a community of practice and rules of analysis etiquette

We built tailored objects for colleges, let users choose their own front end tool

We relied on word-of-mouth adoption and some teasing-revealing

Guess what happened?

20

Top-down versus bottom-up Doing this top down is like pushing water

uphill. Its harder than pushing a rock uphill

The great leader is one who the people say “We did this ourselves”

Consider analytics to be a process of self discovery. Each person has to go through the stages of maturity

Paradoxically, this also requires strong top-down commitment and action! Organizational maneuvers like reorganizations are [normally] required

21

Questions?

22