institutional reporting leone nurbasari, anu pamela sarly, acu hiedi wilkinson, usc

INSTITUTIONAL REPORTING L E O N E N U R B A S A R I , A N U

PA M E L A S A R LY , AC U

H I E D I W I L K I N S O N , U S C

Institutional Reporting from the Australian Graduate Survey

Leone NurbasariPlanning and Performance MeasurementJuly 2014

Presentation Overview• Key Performance Indicators• Program (Course) Review reports• Discipline reporting on CEQ/PREQ• Graduate destination reporting• Comments analyses• Other reporting

3

Key Performance Indicators• ANU Strategic Plan• College Operational Plans and KPIs

4

KPI generation• Raw unit record data at 6 digit FOE, assigned to College scores based on

taught disciplines (rolling 5 year average, minimum 10% of load taught by that College)

• Benchmarking against Go8 data that is weighted according to ANU discipline weightings

5

6

Example of some underlying 6 digit FOE data and calculations for KPI generation

Program (Course) Review Reports• Program reviews • Data warehouse

reports – CEQ, Employment & Further study

• Supplemented by additional detailed analyses (including item level results)

• Supplemented by anonymised open-ended comments

7

Discipline reporting – CEQ/PREQ• Pseudo colleges (custom groupings of 2, 4 and 6 digit FOEs)• Pseudo departments (also custom groupings of FOE)• Benchmarking – Go8 and national, based on ANU taught FOE (excludes non

ANU disciplines)• Percentage of individual student-item response in

disagreement/neutral/agreement categories, aggregated to scales

8

9

Graduate destination reporting• Data warehouse

reports – employment rates, sector, industry, salary, employer, job search

• Cohort-specific analysis

• Ad-hoc data requests

10

Comment analyses• CEQuery with charting via Excel• Various student cohorts and org. units• Adopting Geoff Scott’s method of proxies for

importance and quality

11

Other reporting• Topic Papers – comprising data

from various surveys and student feedback sources, eg. HDR Satisfaction Report

• Ad-hoc reporting from internal pivot tables – GDS, CEQ, PREQ

12

Questions?

• Email Leone.Nurbasari@anu.edu.au • Web http://unistats.anu.edu.au/surveys

13

AGS Data and Reporting at ACU

Pamela Sarly | Acting Manager, Statistical Analysis & Surveys

2014 Survey Manager Information Forum17 July 2014

A bit about ACU…

1900

1963

1991Good

SamaritanSisters

Teacher Training in NSW, VIC, QLD and

ACT

Merger of education colleges, forming

ACU 2014

One of the fastest

growing universities in

Australia

1857

TeacherTraining

in NSW and VIC

ONE OF THE FIRST

EDUCATION PROVIDERS IN

AUSTRALIA

Data as at 11 July 2014

29,513 Enrolments22,245 EFTSL2,839 international students

12 Schools across 4 Faculties

12,952 commencing students16,561 continuing students

Staff FTE• 1,051 Academic• 982 Professional

490 Higher Degree Research Students6,263 Postgraduate Students21,934 Undergraduate Students826 Non-Award Students

Australia's leading Catholic university which is supported by

more than 2,000 years of Catholic intellectual tradition.

AUSTRALIAN GRADUATE SURVEYData Governance at ACU

• A central contact for surveys: o Office of Planning and Strategic Management

• Publish aggregated data through internal sites:o Staff Site (www.acu.edu.au/opsm)o SharePoint site

• Data updates communicated to relevant staff via email

AUSTRALIAN GRADUATE SURVEYInstitutional Reporting

• Strategic Plan - Traffic Light Report

• Quarterly Report

• TEQSA Risk Indicators

• New Course Application

• Course Review and Renewal

• Published in July and December

• Progress against the University’s Strategic Goals and Key Result Areas

TRAFFIC LIGHT REPORT

STRATEGIC SCORECARD

• Published in January, April, July and October

• High level quantitative data that is linked to Strategic Goal 1 (Student Experience)

QUARTERLY REPORTStrategic Plan Goal 1: Student Experience

• Use of SPSS TAS for qualitative data analysis and reporting

• Better linkage to Student Evaluation on Unit and Teaching surveys

• Future of AGS – to inform our new Strategic Plan 2015+

AUSTRALIAN GRADUATE SURVEYFor the future…

Thank youQuestions?

COMMUNICATING RESULTS FOR EFFECTIVE DECISION

MAKING

Australian Graduate Survey

Note that the figures contained within this presentation are fictitious.

Why do we need to communicate results?

AGS results are used at USC to:Inform strategic decision makingAssess program and institution performanceInfluence improvements to programsReview learning and teachingDevelop internal policies

Why use visualisations?

“Numbers have an important story to tell. They rely on you to give them a clear and convincing voice.”

Stephen Few

Choosing the right medium

Textual

Visual

Oral

Tapping into human nature

Over 50% of the cerebral cortex is involved with the processing of visual inputs.

-Lu & Dosher

There are over 130 million sight receiving cells in the retina.

- Vries 2011Source: Atranik 2011

Tableau

Knowing your audience

USC’s Organisational Chart

Know your goal

Enable insight

"The purpose of visualization is insight, not pictures"

Shneiderman 1999

GDS Dashboards Flowchart

CEQ Dashboards Flowchart

Levels of Analysis

Graduate Destination Survey Benchmark Group Dashboard

Course Experience Questionnaire Longitudinal Dashboard

Graduate Destination Survey Faculty Dashboard

Graduate Destination Survey Program (Course) Dashboard

Course Experience Questionnaire Program (Course) Dashboard

How dashboards are used

Are they being used?

Who is using them?

What are they using them for?

Director - Quick reference point

Program leaders - Program reviews

Executive - Marketing and recruitment - Identifying strengths and

weaknesses - Information sharing

What next?

Reports by Field of Education

Beyond Graduation SurveyUniversity Experience

SurveyInternal SurveysInteractive DashboardsQualitative Data

Visualisations

Bibliography

Atranik.org 2011, http://antranik.org/functional-areas-of-the-cerebral-cortex.

Azzam, T & Evergreen, S 2013, Data Visualization, Part 2: New Directions for Evaluation, Number 140, Google eBook.

Card, S, Mackinlay, J & Shneiderman, B 1999, “Readings in Information Visualization - Using Vision to Think”, Morgan Kaufmann, Massachusetts.

De Vries, J 2011, The Five Senses, Random House, Victoria.

Few, S 2006, Information Dashboard Design: The effective visual communication of data, O’Reilly, California.

Lu, Z and Dosher, B 2013, Visual Psychophysics: From Laboratory to Theory, MIT Press, ISBN: 9780262019453.

University of Leicester (Learning Development) 2012, Presenting numerical data, <http://www2.le.ac.uk/offices/ld/resources>

If you have any questions after today:

hwilkins@usc.edu.au

Questions?