moving into analytics: practical steps (166252572)

7/29/2019 Moving into Analytics: Practical Steps (166252572)

http://slidepdf.com/reader/full/moving-into-analytics-practical-steps-166252572 1/43

MOVING INTOANALYTICS:PRACTICAL STEPS

Educause Southeast Conference5/30/2013

Linda Gilbert

Kris Nagel

Laura Ledford

Georgia Gwinnett College

5/30/2013



INTRODUCTIONS

Who we are

Linda Gilbert – Special Assistant to VP Ed TechKris Nagel – Assistant VP, Ed Tech

Laura Ledford – Enrollment Management

Georgia Gwinnett College



ABOUT OUR INSTITUTION

Georgia Gwinnett CollegeBegan in 2006

Massive growth since inception

Enrollment in 2006 = 118

Enrollment in 2012 = 9300+

“Campus of Tomorrow”



SESSION OVERVIEW

Analytics: The Hope and The Hype – Linda Gilbert

Building the Foundation – Kris Nagel

Activity 1: Identifying “where we are”

The “Big Data” Project at GGC – Laura Ledford

Activity 2: Surveying the field

Discussion



ANALYTICS:

THE HOPE AND THE HYPE

Linda Gilbert



THE TECHNOLOGY HYPE CYCLE

Technology Trigger

Peak of Inflated Expectations

Trough of Disillusionment

Slope of Enlightenment

Plateau of Productivity

Visibility/Expectations

TimeSource: Gartner Research

http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp



VISIBILITY/EXPECTATIONS

Horizon Report 2013 (Two to three years out)


“Positioned in the same two-to-three year adoption

timeframe as it was last year, learning analytics

continues to be an emerging field, one that is growing

quickly, but is still just out of reach for most

educators.”

Horizon Report 2013



VISIBILITY/EXPECTATIONS



MOOC – Learning Analytics and Knowledge (LAK) 2013

SoLAR – Society for Learning Analytics Research

Educause – Featured Topic in Library

http://www.educause.edu/library/analytics

Educause Review Special Issue July/Aug 2012

Educause Analytic Sprint, July 2012

ECAR Study of Analytics in HE, August 2012



(Learning) Analytics

BusinessIntelligence

Big Data

EDM

StatisticalmethodsIntelligent

Tutors

Personalization

Adaptivelearning

INFLUENCES

Source: http://www.slideshare.net/gsiemens/iadis-shanghai



ANALYTICS IN HIGHER ED

Academic Analytics

Learning Analytics

Predictive Analyticshttp://net.educause.edu/ir/library/pdf/ELI3026.pdf



ACADEMIC ANALYTICS

“Academic analytics, although a relatively new concept,refers to the analysis of data to help educational institutions monitor

progress on key institutional goals, such as student retention,faculty productivity, and the impact of outreach and engagement.”

http://www.educause.edu/library/academic-analytics

“A process for providing higher education institutions with the data

necessary to support operational and financial decisionmaking.”

http://net.educause.edu/ir/library/pdf/ELI3026.pdf

Focus: Institutional, regional/national/international







LEARNING ANALYTICS

Learning analytics is the measurement, collection, analysis, and reporting of data

about learners and their contexts, for purposes of understanding

and optimizing learning and the environments in which it occurs.

Society of Learning Analytics Research, cited by

“Field associated with deciphering trends and patterns from educational big data, or

huge sets of student-related data, to further the advancement of apersonalized, supportive system of higher education

Horizon Report 2013

“The use of analytic techniques to help target instructional, curricular, and support

resources to support the achievement of specific learning goals.” http://net.educause.edu/ir/library/pdf/ELI3026.pdf

Focus: Learners, Faculty, (Departments)

http://www.nmc.org/publications/2013-horizon-report-higher-ed



http://www.nmc.org/publications/2013-horizon-report-higher-ed



PREDICTIVE ANALYTICS

“An area of statistical analysis that deals with extractinginformation using various technologies to uncover relationships and

patterns within large volumes of data that can be used to predictbehavior and events.


“Educational Data Mining is an emerging discipline, concerned withdeveloping methods for exploring the unique types of data that comefrom educational settings…, it often has multiple levels of meaningfulhierarchy, which often need to be determined by properties in the dataitself , rather than in advance.”

LAK2013 Course Materials Focus: All levels





MODELS

1. Extracting and analyzing data from learning management systems

2. Building an analytics matrix that incorporates data from multiple sources (socialmedia, LMS, student information systems, etc).

3. Profile or model development of individual learners (across the analytics matrix)

4. Predictive analytics: determining at-risk learner

5. Automated intervention and adaptive analytics: i.e. the learner model should beupdated rapidly to reflect near real-time learner success and activity so thatdecisions are not made on outdated model

6. Development of "intelligent curriculum" where learning content is semanticallydefined

7. Personalization and adaptation of learning based on intelligent curriculum wherecontent, activities, and social connections can be presented to each learner based

on her profile or existing knowledge8. Advanced assessment: comparing learner profile with architecture of knowledge in

a domain for grading or assessment Source: LAK2013 (MOOC)



MODELS

Focus Type of analytics

1 What happened? Standard reports in real time

2 How many, how often, where? Ad hoc reports (real time)

3 Where exactly is the problem? Query/drill down (real time)

4 What actions are needed? Alerts (real time)

5 Why is this happening? Statistical analysis

6 What if these trends continue? Forecasting/extrapolation

7 What will happen next? Predictive modeling

8 What’s the best thing that could happen? Optimization

Adapted from the Davenport/Harris Framework Source: http://www.educause.edu/library/resources/building-organizational-capacity-analytics



MODELSFocus

(Unit of Analysis)

Learner (Learning Analytics)

Institution (Academic Analytics)

Forecasting (Predictive Analytics)

Type of Analysis(Level of Inference)

Reporting (Descriptive Analytics)

Gilbert 2013



INTRIGUING DIRECTIONS

Trend analysis and prediction Early warning/intervention

Enrollment/Admission decisions

Personalization and adaptive learning

Online class content Mentoring support

Modeling: Learners, knowledge domains, relationships

Development of policies, norms, and technology OLA: Open Learning Analytics Architecture (SoLAR)



CONSIDERATIONS AND CONCERNS

Messy Data

Availability

Data choices

Human Biases

WYSIATI

Limits to Prediction

“Black Swans”

Autonomous Agents

Feedback Loops – Self-fulfilling prophecies

“The problem starts when smart people innice suits and lab jackets proclaim that “the

data says…” In truth, the data never saysanything.

We interpret it in one way or another and

there are lots of ways to interpret itincorrectly.” - Greg Satell

http://www.digitaltonto.com/2011/the-pitfalls-of-prediction/



CONSIDERATIONS AND CONCERNS

Data Tools

Many current systems are stand alone, single functionality

Visualization Tools Needed

Data Use/Culture

Good tools =/= Good models =/= Good decisions

Privacy and Ethics:

Just because we can… should we?



MOVING INTO ANALYTICS

10/26/2012

Source: http://www.educause.edu/library/resources/building-organizational-capacity-analytics



BUILDING THE FOUNDATION

Kris Nagel



OVERVIEW OF CHANGE

Then No procedures for requesting data

Multiple places where one could expectto get data

IR / IE

Enrollment

OET

HR and Financial area

Often, multiple units working on samedata request

Lack of clarity about “who can approvesharing this data?”

Complex requests would “stall out” inapproval process from multiple units

Now

Centralized data request form

Work distributed to appropriate

unit

Data “stewards” and “managers”

identified.

Approval process documented.

Regular meetings in place to

discuss complex requests

10/26/2012



OVERVIEW OF CHANGE

Then Lack of communication

Between units with data

With data requestors

Responsibilities not clear, notdocumented

Data managers/stewards

End users

Regular training on relevant lawsminimal and “CYA” in tone

FERPA – student records

HIPPA – Student Affairs/counseling

GLB – Accounting/Banking

Now

Discussion part of the process

Among data stewards

With requestors

Shared understanding and

vocabulary about dataresponsibilities

Policies in place, includingconfidentiality agreement

Training for data managers in place;

refresher courses for all under

development

Still work to do… but tremendous progress!

10/26/2012



THE ROAD WE TRAVELED: THE JOURNEY

Identifying the need… and “whose problem is it?”

Getting permission from TPTB

Getting the right people in the room

Figuring out where to start!

Locating resources USG

EDUCAUSEData

requestprocess Job

roles

Policy

Cross-systemvocab

UserEd

10/26/2012

Policy Dimensions of Analytics inHigher Education. Educause

Review July 2012

Policy Drivers

Data Governance,

Data Classification,

Roles and

Responsibilities,Data Policies,

Legal Requirements



THE ROAD WE TRAVELED:LESSONS LEARNED

Leadership

Collaborate

“Manage up!”

Technology Infrastructure

Expect it to be more complicatedthan you think

Processes & Practices

Document and communicate!

Skills & Values

Look for pre-existing wheels andlevers

Culture & Behaviors

Take time to develop

Prioritize – and work iteratively

Communication / Education

Policy and ConfidentialityAgreements

Data request process

Formal training

Documentation

10/26/2012

Rejoice in incremental/iterative improvements!

Building Organizational Capacity for Analytics

Educause Paper February 2013



MAKING IT REAL …TANGIBLE RESULTS

Data Policies (beyond basic student record policies) addressing User responsibilities

Data management structure

Data classification

Confidentiality Agreement (Culture and Values)

Included in onboarding, along with basic FERPA training Checkbox on data request form

Processes Centralized Data Request Form (Technology Support of Process)

Regular meetings for Data Request Review

Online tools to support work

Documentation of work done! (Technology support)

10/26/2012



MAKING IT REAL …DATA REQUEST PROCESS OUTCOMES

Building culture of data awareness

Need to ask

Need to secure/protect

Building repository of information

Tracking reports created across campus

Reports often answer questions from several units

Creates data dictionary for campus from items requested

Identifies needs for recurring and real-time data access

Building expectation of accessibility

Easy to make a request, spawns automatic email notifications

Manage approval and fulfillment process within on-line form Data provider is easier to identify

10/26/2012



ACTIVITY 1:IDENTIFYING “WHERE WE ARE”

Linda Gilbert

10/26/2012



THE BIG DATA PROJECT AT GGC

Laura Ledford

10/26/2012



WHAT IS BIG DATA?

A collection of data sets so large that itbecomes difficult using on-hand databasemanagement tools or traditional data

processing.Gartner defines it as 3-dimensional (three V’s)

Volume

Velocity

Variety

10/26/2012



IMPLICATIONS OF BIG DATA

The “3 V’s” make data difficult to work with

Privacy and security issues must be considered

Need to rethink data management options

Must complement with “big judgment” to sufficientlymanage the “big data”

Need to figure out how to leverage the benefits of “bigdata” while “avoiding paralysis by analysis”.

10/26/2012



BENEFITS OF BIG DATA

More accurately predict behaviors andoutcomes

Allows testing of multiple outcomes beforegoing live

Allows personalization

10/26/2012



10/26/2012

EXAMPLE

How Target Figured Out A

Teen Girl Was PregnantBefore Her Father Knew

Forbes Magazine



10/26/2012

REQUIRED - must be unique, within

this sheet as well as within your

cookbook account, indexed forsearch

REQUIRED - any text, indexed for search any text - optional, but if you fill

it in, must also fill in Data System

and Time Context, indexed forsearch

REQUIRED - coded,

can be a list

separated bycommas

coded - required if you have a

technical definition

HS Overall GPA GPA provided on HS transcript and used foradmission decision Not used for admissions. The 16-17

required HS courses are all that's

considered. Note - entire transcript

scanned in for viewing as of F2011.Decision: NOT NOW. Revisit later.

Not in Banner. Paper versionof HS transcript. Probably not

a converted GPA.

HS GPA - BOR version GPA on 16-17 courses required by BOR andused from Admissions See HS Overall GPA. Can break out

Math/Eng/Sci for required courses. BANNER TABLE: SORSHCHBANNER FIELD:

SORHSCH_GPA Banner/ADM field

HS Math GPA GPA for all Math classes taken or credited

during HS. May include 8th grade math if

Algebra, should include AP course. QUESTION:weighted or unweighted for AP? QUESTION:

Include joint enrollment? If so, weighted orunweighted?

Math MIGHT be extracted from HS

GPA - BOR version. Will only include

those on transcript. BANNER TABLE: ZORHSWK

BANNER FIELDS:

ZORHSWK_EFF_YEARZORHSWK_SBJC_CODE =

MATRZORHSWK_UNITS

ZORHSWK_GRADE_CODE

ZORHSWK_HSGR_CODE*ZORHSWK_POINTS

*translated grade code

ALL are not in Banner. Paper

version of HS transcript.

Probably not a converted GPA.May be able to get classes that

are part of HS-GPA-BOR inBanner. Will probably need to

be calculated.



A BETTER EXAMPLE

GGC wants to determine predictive behaviors of incoming students based on potential indicators,such as:

Test results

High school GPA

Personal factors

number of hours worked

Family responsibilities

whether the student was a first generation student.

10/26/2012



PROCESS AND DECISION-POINTS

Identify existing data

Desired data stored in multiple places on campus and inmultiple formats

Need to choose “best” version

Determine a common identifier Collect additional data

Create incoming student survey

Embed logic for processing

Some data must be created from a combination of othervariables.



LESSONS LEARNED

Data can be difficult to aggregate across systems, and to verify.

Data choices involve multiple decision points

Example: ACT scores vs. SAT scores.

Do we take the “best score” from each area or the best test overall as thepredictor

Often requires multiple individuals with different skill sets. Data managers to indicate which data field is correct (grades vs. grade changes,

academic standing)

Assessment data

Analytic skills

Collection strategies

Surveys, Student Information System, etc. Programming that can pull data together from multiple sources



ACTIVITY 2SURVEYING THE FIELD

Linda Gilbert

10/26/2012



FOR YOUR CASE STUDY

Review your case study

Identify the analytic focus (ex: learner, course, faculty, department, institution).

Identify the type of analysis (ex: reports, real-time reports, trend analysis, prediction,

etc.)

Identify the decisions to be made from the data

Identify key indicators used

Other comments

We’ll collect your reports via PollEveryWhere (open-ended)

Remember to use CASE NUMBERS when reporting



SUMMARY

The original poll has been removed from this updated version of thepresentation, and replaced with the following summary of data from theworkshop participants.

Use the reference list as a key.

The articles are numbered, and the numbers correspond to the case studies.

Not all articles were addressed in the workshop session.

Responses have been slightly edited for space and clarity.



SUMMARY

Article Focus Analytics Decisions/Goal Key indicators Comments 1 Learner Dashboard Course selection GPA, degree requirements Next step -recommending

majors 2 Learner Dashboard Contact students Alerts (student actions)

3 Learner, LMS Reports, Trend analysis LMS log data and studentsuccess

LMS activities, studentperformance, demographics Good study

4 5 Learner to Institutional All options possible Example: Course offerings,

delivery, schedulingCourse selection by major;scheduling conflicts Meaningful data can lead to

culture change

6 Learner Predictive ID at-risk students clicks

7 Learner Predictive When to intervene responses to nudgesCapitalized on students' use of mobile technology

8 9 Student retention Conceptual models Prioritize resources Usage indicators - surveys,

attendance, etc. Emphasizes the need to usedata

10 Learner, Institution Trend analysis Prioritize resources Modeling based ondemographic shifts in state Using data to drive business

plan

11 12 Learner All options possible Predicting student success degree requirements Future developments have

potential

13 14 Learner Real-time reports ID student engagement in

online courseparticipation in courseactivities Realtime feedback more useful

than end-of-course



QUESTIONS, COMMENTS, AND

DISCUSSION

10/26/2012



RESOURCES AND CREDITS

Handout and slides will be posted under the session topic at Educause

Presenters:

Linda Gilbert ([email protected])

Kris Nagel ([email protected])

Laura Ledford ([email protected])

Credits: We would like to acknowledge the other charter members of the DataManagers and Stewards Working Group at GGC:

Juliana Lancaster

Nancy Grattan

Lily Hwang

mailto:[email protected]






moving into analytics: practical steps (166252572)

Documents