Learning Analytics: Assisting Universities with Student Retention

2015 Final Report - Part 2 Appendices

Charles Darwin UniversityBatchelor Institute of Indigenous Tertiary Education

Griffith UniversityMurdoch University

The University of Newcastle

Deborah West

Charles Darwin University

http://www.letstalklearninganalytics.edu.au/

Support for the production of this report has been provided by the Australian Government Office for Learning and Teaching. The views expressed in this report do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.

With the exception of the Commonwealth Coat of Arms, and where otherwise noted, all material presented in this document is provided under Creative Commons Attribution-ShareAlike 4.0 International License http://creativecommons.org/licenses/by-sa/4.0/.

The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided) as is the full legal code for the Creative Commons Attribution-ShareAlike 4.0 International License http://creativecommons.org/licenses/by-sa/4.0/legalcode.

Requests and inquiries concerning these rights should be addressed to:Office for Learning and TeachingDepartment of Education and Training

GPO Box 9880, Location code N255EL10Sydney NSW 2001

<learningandteaching@education.gov.au>

2015

ISBN 978-1-76028-462-6 PRINT

ISBN 978-1-76028-463-3 PDF

ISBN 978-1-76028-464-0 DOCX

Table of contents

Appendix A - DVC Certification 4

Appendix B – References 5

Appendix C – Demographic Frequency Data of Academic Level Survey Participants 12

Appendix D - Relevant OLT Projects 13

Appendix E – Impact Statement 15

Appendix F – Completed Dissemination Activities 18

Appendix G – Scheduled Dissemination Activities 19

Appendix H – High Level Framework Summary 20

Appendix I – Discussion Questions 22

Appendix J – Evaluation Report 26

Learning Analytics: Assisting Universities with Student Retention 3

Appendix A - DVC Certification

Learning Analytics: Assisting Universities with Student Retention 4

Appendix B – References

ACODE (2014). Benchmarks for Technology Enhanced Learning. Retrieved 12 May, 2015 from http://www.acode.edu.au/pluginfile.php/579/mod_resource/content/3/TEL_Benchmarks.pdf

Australian Government Department of Education and Training (2014) 2013 Student Data: Appendix 4 - Attrition, Success and Retention. Retrieved 26 May, 2015 from https://www.education.gov.au/selected-higher-education-statistics-2013-student-data

Arnold, K., Lonn, S., & Pistilli, M. (2014). An exercise in institutional reflection: The Learning Analytics Readiness Instrument (LARI). Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ‘14), Indianapolis, IN, USA. doi:10.1145/2567574.2567621

Arnold, K. & Pistilli, M. (2012). Course signals at Purdue: Using learning analytics to increase student success, Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), Vancouver, British Columbia, Canada. doi:10.1145/2330601.2330666

Baer, L., Norris, D., Duin, A., & Brodnick, R. (2013). Crafting transformative strategies for personalized learning/analytics. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460354

Baker, R., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1 (1), 3-17.

Balacheff, N., & Lund, K. (2013). Multidisciplinarity vs. multivocality: The case of “Learning Analytics”. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460299

Beer, C., Jones, D. & Clark, D. (2012). Analytics and complexity: Learning and leading for thefuture. Proceedings Future Challenges: Sustainable Futures, ascilite, Wellington, NZ. Retrieved April 13, 2015 from http://eprints.usq.edu.au/23092/2/Beer_Jones_Clark_ascilite_2012_PV.pdf

Behrendt, L., Larkin, S., Griew, R., & Kelly, P. (2012). Review of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander People: Final Report. Canberra: Australian Government. Retrieved May, 12, 2012 from http://docs.education.gov.au/documents/review-higher-education-access-and-outcomes-aboriginal-and-torres-strait-islander-people-0

Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008). Review of Australian Higher Education: Final Report. Canberra: Department of Education, Employment and Workplace Relations. Retrieved May, 12, 2015 from www.mq.edu.au/pubstatic/public/download.jsp?id=111997

Learning Analytics: Assisting Universities with Student Retention 5

Buckingham Shum, S., & Ferguson, R. (2012). Social learning analytics. Educational Technology & Society, 15(3), 3-26.

Camilleri, V., de Freitas, S., Montebello, M., & McDonagh Smith, P. (2013). A case study ‐inside virtual worlds: Use of analytics for immersive spaces. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460341

Campbell, J., DeBlois, P., & Oblinger, D. (2007). Academic analytics: A new tool for a new era, Educause Review, July/August 2007. Retrieved May 12, 2015 from https://net.educause.edu/ir/library/pdf/ELIWEB0710.pdf

Campbell, J., and Oblinger, D. (2007) Academic analytics: White paper. Educause. Retrieved 12 May, 2015 from http://net.educause.edu/ir/library/pdf/pub6101.pdf

Chaloux, B., & Miller, G. (2013). E-learning and the transformation of higher education. Chapter 1 (pp. 3-23) in G. Miller, M. Benke, B. Chaloux, L. Ragan, R. Schroeder, W. Smutz, Wayne, K. Swan, (Eds.) Leading the e-Learning transformation in higher education: meeting the challenges of technology and distance education. Stirling, VA: Stylus Publishing.

Chatti, M., Dyckhoff, A., Schroeder, U., & Thüs, H. (2012). A reference model for learning analytics. International Journal of Technology Enhanced Learning, 4(5/6), 318–331.

Chowdry, H., Crawford, C., Dearden, L., Goodman, A., & Vignoles, A. (2013), Widening participation in higher education: analysis using linked administrative data. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176: 431–457. doi: 10.1111/j.1467-985X.2012.01043.x

Clarke, J., Nelson, K., & Stoodley, I. (2013). The place of higher education institutions in assessing student engagement, success and retention: Proceedings Higher Education Research and Development Society of Australasia, Auckland, New Zealand. Retrieved May 12, 2015 from http://eprints.qut.edu.au/60024/2/60024.pdf

Clow, D. (2013). MOOCs and the funnel of participation. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460332

Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current state and future trends: A citation network analysis of the learning analytics field. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ‘14), Indianapolis, IN, USA. doi:10.1145/2567574.2567585

Dawson, S., Tan, J., & McWilliam, E. (2011). Measuring creative potential: Using social network analysis to monitor a learners’ creative capacity. Australian Journal of Educational Technology, 27(6), 924-942.

Learning Analytics: Assisting Universities with Student Retention 6

Dekker, G., Pechenizkiy, M. and Vleeshouwers, J. (2009) Predicting students’ drop out: a case study. Proceedings of the 2nd International Conference on Educational Data Mining (EDM '09), Cordoba, Spain. http://www.educationaldatamining.org/EDM2009/uploads/proceedings/dekker.pdf

Diaz, D. (2000). Comparison of student characteristics, and evaluation of student success, in an online health education course. Unpublished doctoral dissertation, Nova Southeastern University. Retrieved 12 May 2015 from http://home.earthlink.net/~davidpdiaz/LTS/pdf_docs/dissertn.pdf

Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) (2012a). 2011 full year higher education student statistics Appendix 2 – Equity Groups. Canberra: Australian Government. Retrieved 18 June, 2013 fromhttp://www.innovation.gov.au/HigherEducation/HigherEducationStatistics/StatisticsPublications/Pages/2011StudentFullYear.aspx

Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE) (2012b). 2011 full year higher education student statistics Appendix 4 - attrition, progress and retention. Canberra: Australian Government. Retrieved 14 June 2013 fromhttp://www.innovation.gov.au/HigherEducation/HigherEducationStatistics/StatisticsPublications/Pages/2011StudentFullYear.aspx

Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE). (2013a). Students: selected higher education statistics 2012 (Appendix 2: equity data). Canberra: DIICCSRTE.

Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education (DIICCSRTE). (2013b). Moving to an enhanced indicator of higher education students’ socio-economic status. Canberra: DIICCSRTE.

EDUCAUSE (2010). 7 Things you should know about analytics. Retrieved May 12, 2015 fromhttp://www.educause.edu/ir/library/pdf/ELI7059.pdf

EduTech Wiki (2013). Learning analytics. Retrieved 14 June, 2013 from http://edutechwiki.unige.ch/en/Learning_analytics#Introduction

Ferguson, R., (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317.

Fiaidhi, J. (2014). The next step for learning analytics. IT PRO/Institute of Electrical and Electronics Engineers, September/October 2014. Retrieved May 12, 2015 from https://www.computer.org/csdl/mags/it/2014/05/mit2014050004.pdf

Frankola, K. (2001). Why online learners drop out. Retrieved May 12, 2015 from http://www.workforce.com/articles/why-online-learners-drop-out

Learning Analytics: Assisting Universities with Student Retention 7

Gallagher, M. (2014). Micro-economic reform of the Australian higher education industry: Implications of the Abbott Government’s Budget of 13 May 2014. Address to the EduTECH Higher Education Leaders Congress, Brisbane, Australia. Retrieved May 10, 2015 from https://go8.edu.au/sites/default/files/docs/article/edutech_presentation_-_4_june_2014-pdf_version.pdf

Gašević, D., Mirriahi, N., Long, P., & Dawson, S. (2014). Editorial – inaugural issue of the Journal of Learning Analytics. Journal of Learning Analytics, 1(1), 1–2.

Goldstein, P., & Katz, R. (2005). Academic analytics: The uses of management information and technology in higher education. ECAR Research Study, Vol. 8. Retrieved May 10, 2015 fromhttp://www.educause.edu/library/resources/academic-analytics-uses-management-information-and-technology-higher-education

Hanna, D. (2000). Higher education in an era of digital competition: choices and challenges. Madison, WI: Atwood Publishing.

Heathcote, E., & Dawson, S. (2005) Data mining for evaluation, benchmarking and reflective practice in a LMS. Proceedings e-Learn 2005: World conference on e-Learning in corporate, government, healthcare & higher education, Vancouver, Canada. Retrieved April 13, 2015 from http://eprints.qut.edu.au/2368/1/2368.pdf

Holman, C., Aguilar, S., & Fishman, B. (2013). GradeCraft: what can we learn from a game-inspired learning management system. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460350

James, R., Krause, K-L., & Jennings, C. (2010). The first year experience in Australian universities: Findings from 1994 to 2009. Centre for the Study of Higher Education, Melbourne: University of Melbourne. Retrieved May 10, 2015 from http://www.cshe.unimelb.edu.au/research/experience/docs/FYE_Report_1994_to_2009.pdf

Kennedy, G., Corrin, L., Lockyer, L., Dawson, S., Williams, D., Mulder, R., Khamis, S. & Copeland, S. (2014). Completing the loop: returning learning analytics to teachers. Proceedings Rhetoric and Reality: Critical perspectives on educational technology, 31st ascilite Conference, Dunedin, New Zealand. Retrieved May 12, 2015 from http://www.academia.edu/12074024/Completing_the_loop_returning_learning_analytics_to_teachers

Kift, S. (2009). Articulating a transition pedagogy to scaffold and to enhance the first year student learning experience in Australian higher education: Final report. ALTC Senior Fellowship. Retrieved April 14, 2015 from http://www.altc.edu.au/resource-transition-pedagogy-report-qut-2009

Learning Analytics: Assisting Universities with Student Retention 8

Kift, S., Nelson, K., & Clarke, J. (2010). Transition pedagogy: A third generation approach to FYE - A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), 1-20.

Koshy, P. (2014). Student equity performance in Australian higher education: 2007 to 2012. National Centre for Student Equity in Higher Education (NCSEHE), Perth: Curtin University.

Macfadyen, L., & Dawson, S. (2010). Mining LMS data to develop an ‘early warning system’ for educators: A proof of concept. Computers & Education, 54(2), 588-599.

Mirriahi, N., & Dawson, S. (2013). The pairing of lecture recording data with assessment scores: A method of discovering pedagogical impact. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460331

Mirriahi, N., Gašević, D., Long, P., & Dawson, S. (2014). Scientometrics as an important tool for the growth of the field of learning analytics. Journal of Learning Analytics, 1(2), 1 4.‐

Nelson, K., Clarke, J., Stoodley, I., & Creagh, T. (2014). Establishing a framework for transforming student engagement, success and retention in higher education institutions. Final Report 2014, Canberra, Australia: Australian Government Office for Learning & Teaching.

Nelson, K., Kift, S., & Clarke, J. (2012). A transition pedagogy for student engagement and first-year learning, success and retention. In I. Solomonides, A. Reid & P. Petocz (Eds.) Engaging with learning in higher education. Faringdon, UK: Libri Publishing.

Norris, D., Baer, L. & Offerman, M. (2009). A national agenda for action analytics. National Symposium on Action Analytics, St. Paul, MN. Retrieved May 12, 2015 from http://lindabaer.efoliomn.com/uploads/settinganationalagendaforactionanalytics101509.pdf

Norris, D., & Baer, L. (2012). A toolkit for building organizational capacity for analytics. Washington, DC: Strategic Initiatives, Inc. Retrieved 12 May, 2015 from https://docs.google.com/a/student.rmit.edu.au/file/d/0B0CqeHHcp-YzMEZPY2hCRHpyLVU/edit

Norris, D., & Baer, L. (2013). Building organizational capacity for analytics. Boulder, CO: Educause. Retrieved 10 May, 2015 from https://net.educause.edu/ir/library/pdf/PUB9012.pdf

Ochoa, X., Suthers, D., Verbert, K., & Duval, E. (2014). Analysis and reflections on the third Learning Analytics and Knowledge Conference (LAK 2013). Journal of Learning Analytics, 1(2), 5 22.‐Pechenkina, E & Anderson, I (2011) Background paper on Indigenous Australian higher education: trends, initiatives and policy implications. Retrieved May 12, 2015 from

Learning Analytics: Assisting Universities with Student Retention 9

www.innovation.gov.au/HigherEducation/IndigenousHigherEducation/ReviewOfIndigenousHigherEducation/Pages/Research.aspx.

Raca, M., Tormey, R., & Dillenbourg, P. (2014). Sleeper’s lag: A study on motion and attention. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ‘14), Indianapolis, IN, USA. doi:10.1145/2567574.2567581

Rivera, J., and Rice, M. (2002). A comparison of student outcomes & satisfaction between traditional & web based course offerings. Online Journal of Distance Learning Administration, 5(3).

Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state-of-the-art. IEEE Transaction on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 40(6), 601-618.

Siemens G. (2012). Learning analytics: Envisioning a research discipline and a domain of practice, Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), Vancouver, British Columbia, Canada. http://learninganalytics.net/LAK_12_keynote_Siemens.pdf

Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the productivity of the higher education sector: policy and strategy for systems-level deployment of learning analytics. Society for Learning Analytics Research/Australian Government Office for Learning and Teaching. Retrieved 14 April, 2014 from http://www.itl.usyd.edu.au/projects/SoLAR_Report_2014.pdf

Siemens, G. & Long, P. (2011) Penetrating the Fog: Analytics in learning and education. EDUCAUSE Review, 46(4) July/August. Retrieved 20 May, 2013 from http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education

Suthers, D., & Verbert, K. (2013). Learning analytics as a “middle space.” Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460298

Thomas, L. (2002).Widening participation in post compulsory education.‐ Education & Training, 44(4/5), 241 – 242. doi:10.1108/et.2002.44.4_5.241.2

Tinto, V. (2009) Taking student retention seriously: rethinking the first year of university. ALTC FYE Curriculum Design Symposium, Queensland University of Technology, Brisbane, Australia. Retrieved May 8, 2015 from http://www.fyecd2009.qut.edu.au/resources/SPE_VincentTinto_5Feb09.pdf

Tsui, J., Lau, J., & Shieh, L. (2014). Septris and SICKO: Implementing and using learning analytics and gamification in medical education. Educause Learning Initiative. Retrieved May 10, 2015 from https://net.educause.edu/ir/library/pdf/ELIB1401.pdf

Learning Analytics: Assisting Universities with Student Retention 10

Universities Australia (2013). A smarter Australia: An agenda for Australian higher education 2013-2016. Canberra, ACT: Universities Australia. Retrieved April 21, 2015 from https://www.universitiesaustralia.edu.au/news/policy-papers/Universities-Australia-Policy-Statement-2013---2016#.VVFLwPmqpBc

van Barneveld, A., Arnold, K., & Campbell, J. (2012). Analytics in higher education: Establishing a common language. Educause Learning Initiative. Retrieved May 10, 2015 from http://net.educause.edu/ir/library/pdf/eli3026.pdf

van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. JISC CETIS Analytics Series, Analytics for Learning and Teaching, 1(3). Retrieved May 12, 2015 from http://publications.cetis.ac.uk/wp-content/uploads/2012/11/Analytics-for-Learning-and-Teaching-Vol1-No3.pdf

Verbert, K., Govaerts, S., Duval, E., Santos, J., Assche, F., Parra, G., & Klerkx, J. (2013). Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499 – 1514. West, D., Lizzio, A. Miles, Toohey, D., C. Huijser, H., Bronniman, J. & Searle, B. (2013) Learning Analytics: How it can assist Universities with Student Retention. OLT Strategic Commissioned Project Application. Willcoxson, L., Manning, M., Hibbins, R., Joy, S., Thomas, J., Girardi, A., Leask, B., Sidoryn, T., Cotter, J., Kavanagh, M., Troedson, D., & Lynch, B. (2011). The whole of university experience: Retention, attrition, learning and personal support interventions during undergraduate business studies. Surrey Hills, NSW: Australian Learning and Teaching Council. Retrieved May 10, 2015 from http://eprints.usq.edu.au/20138/

Wolff, A., Zdrahal, Z., Nikolov, A., & Pantucek, M. (2013). Improving retention: Predicting at-risk students by analysing clicking behaviour in a virtual learning environment. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), Leuven, Belgium. doi:10.1145/2460296.2460324

Zaïane, O. (2001). Web usage mining for a better web-based learning environment. Proceedings 4th IASTED International Conference on Advanced Technology for Education (CATE ‘01), Banff, Canada. Retrieved 12 May, 2015 from http://webdocs.cs.ualberta.ca/~zaiane/postscript/CATE2001.pdf

Learning Analytics: Assisting Universities with Student Retention 11

Appendix C – Demographic Frequency Data of Academic Level Survey Participants

Variable Category Absolute Frequency

Relative Frequency

Location (n = 351)

Australia 341 97%New Zealand 10 3%

Primary Work Role (n = 353)

Teaching Students 188 53%Learning Support 47 13%Management/Administration 37 11%Other 32 9%Research 24 7%Academic Development 18 5%Student Support 7 2%

LMS at Institution (n = 353)

Blackboard 203 58%Moodle 124 35%Brightspace (D2L) 13 4%Sakai 2 1%Other 11 3%

Employment Basis (n = 351)

Full Time 284 81%Part Time 44 13%Casual 18 5%Other 5 1%

Academic Level (n = 351)

Lecturer 124 35%Senior Lecturer 88 25%Other 59 17%Associate Professor 30 9%Associate Lecturer/Tutor 30 9%Professor 20 6%

Length of employment in current institution(n = 324)

Less than 1.5 years 40 12%1.5 – 5 years 68 21%5 – 10 years 96 30%10- 20 years 86 27%More than 20 years 34 11%

Length of employment in Higher Education Sector(n = 345)

Less than 1.5 years 11 3%1.5 – 5 years 42 12%5 – 10 years 85 25%10- 20 years 130 38%More than 20 years 77 22%

Involvement in teaching students (n = 353)

Teaches students 276 78%Does not teach students 77 22%

Learning Analytics: Assisting Universities with Student Retention 12

Appendix D - Relevant OLT Projects

The following is a list of relevant OLT projects which have been considered during the project and, which are linked to in the broader project literature review.

Project title Lead institution

Summary

Enabling connected learning via open source analytics in the wild: learning analytics beyond the LMS (2014)

Queensland University of Technology

This project extends the field of Learning Analytics (LA) by developing an open source toolkit for performing sophisticated analysis of learners' engagement in connected learning environments.

Innovative perspectives and approaches for enhancing the student experience (2014)

University of Melbourne

This project builds capacity of higher education institutions by developing new perspectives and approaches for enhancing the student experience.

Engaging postgraduate students and supporting higher education to enhance the 21st century student experience (2014)

Bond University

The project engages postgraduate students and higher education personnel to contribute their experiences, insights and strategies to enhance the 21st century student experience, with an enquiring lens on the growing postgraduate coursework experience.

Shaping the 21st century student experience at regional universities (2014)

University of the Sunshine Coast

This project seeks to understand the 21st century experiences of the diverse undergraduate student populations of regional universities to identify how equitable outcomes for these students can be achieved.

Open for learning: using open data tools and techniques to support student learning (2013)

Monash University

This project explored the extent to which providing open access to data through digital technology allows students to re-configure and re-use data in order to address real-world problems relating to their learning behaviours and learning decisions.

Developing a culturally appropriate data quality framework for Aboriginal and Torres Strait Islander higher education statistics (2013)

University of Notre Dame

This project addressed the persistent and problematic nature of data and statistics pertaining to Aboriginal and Torres Strait Islander peoples, in particular in relation to their participation in higher education.

Learning Analytics: Assisting Universities with Student Retention 13

Project title Lead institution

Summary

Evidenced-based approach to the design and redevelopment of inclusive technology enhanced learning environments (2013)

University of South Australia

This project aimed to demonstrate the benefits of an evidence-based approach to the design of inclusive technology enhanced learning (TEL) environments.

Data mining of learning behaviours and interactions for improved sentiment and performance (2013)

University of Technology Sydney

This project aimed to develop innovative and workable data mining technology for an in-depth understanding of what makes good academic sentiment and performance and how poor-performing students can receive early intervention in a customized way in order to advance to high-performing status.

Student retention and learning analytics: a snapshot of current Australian practices and a framework for advancement (2013)

University of South Australia

This project aimed to undertake a review of learning analytics across Australian universities within an international context. The project focuses on how learning analytics are informing teaching practice, personalised learning, and applications for improving retention and identification of students at-risk.

Completing the loop: returning meaningful learning analytic data to teachers (2013)

University of Melbourne

The project has developed a web-based analytics tool that will support teachers to more easily interpret learning analytics to help them improve teaching and learning practices.

Improving educational opportunities for Aboriginal education workers through technology-based pedagogy (2013)

Murdoch University

Design-based research (DBR) informed by Indigenous research theories were used to analyse the potential educational roles for AEWs enabled by E-Learning and new technologies.

Exploring the role of technology in fostering sense of belonging in students studying by distance (2013)

Charles Sturt University

This project examined not only the impact of specific programs designed to enhance student belonging, but moreover how institution-wide change in learning technologies and policies affects practice.

Learning Analytics: Assisting Universities with Student Retention 14

Appendix E – Impact Statement

Anticipated Changes at:

Impact on: Project Completion

6 months post-completion

12 months post-completion

24 months post-completion1

1 Team Members

Project members all have influence within their institution around the development of learning analytics infrastructure, strategies and/or implementation

Some team members have gained additional input to the development of learning analytics in their institution

Team members have influenced vendors roadmap and development of proprietary analytics packages

University of Newcastle to scope a student-focused analytics solution targeting retention outside of the larger MIS and from within Blackboard.

University of Newcastle implementation of student-focused analytics solution (for students)

2 Project team members immediate Students2

3 Spreading the Word

Presentations have been requested and completed in Malaysia and Singapore

Article for special edition of journal requested by editors

One external

Follow up ‘use case’ workshop run at one conference to provide examples of reports and gather further feedback Identified

applicability of

1 Learning analytics is at a very early stage and a fast moving field so it is difficult to project how this work would be useful in this time frame beyond being something that others would build on.2 only one of the project team members has a direct teaching role; most of us are located at a central level so

are likely to influence students in our institution more broadly

Learning Analytics: Assisting Universities with Student Retention 15

Anticipated Changes at:

Impact on: Project Completion

6 months post-completion

12 months post-completion

24 months post-completion

institution has engaged in a workshop to consider use cases to inform the development of learning analytics

2 additional institutions have requested workshops

Request to work with 2 Malaysian universities to extend this study in Malaysia

Several papers are scheduled at various conferences in the coming months

2 use case workshops are run in outside institutions

framework in Malaysia

4 Narrow opportunistic adoption in partner institutions (impacting on some students)

Learning analytics reports on retention have been trialled by some academics in 1 partner institution although they are available to all academics

Framework is applied robustly in 3 institutions to progress learning analytics initiatives

5 Narrow systemic adoption (in partner institutions impacting on all students)

Use case workshops have been run in two partner institutions

Use cases have been used to inform the development of reports at 1 partner institution

Framework has been used to inform development (or move forward on learning analytics

Advancement of the use of learning analytics for retention in partner institutions

The use of learning analytics reports is included in policy procedure in at least one partner institution.

Learning Analytics: Assisting Universities with Student Retention 16

Anticipated Changes at:

Impact on: Project Completion

6 months post-completion

12 months post-completion

24 months post-completion

in 3 partner institutions

6 Broad opportunistic adoption (in outside institutions impacting on some students)

Increase in the number of academics using learning analytics reports to assist with student retention

7 Broad systemic adoption (in outside institutions impacting on all students)

One LMS vendor has done initial mapping of academic use cases to available reports in their product

LMS vendor has adopted technical implementation approach by one institution as a standard

Use cases have been used as model for implementation in 3 institutions

Framework utilised by a number of outside institutions to inform the ongoing development of learning analytics for retention

Potential addition of reports to 1 vendors product

Additional projects to extend the work of the project in the sector (e.g. next iteration of another version of a framework)

Additional projects to extend the work on the project

Learning analytics becomes a core element of retention plan reporting and/or informing of retention strategies

Application of learning analytics work to various disciplines

Learning analytics emerges as a ubiquitous

Learning Analytics: Assisting Universities with Student Retention 17

Appendix F – Completed Dissemination Activities

Event Date Event Title, Location

Brief Description of Event Number of participants

10 June 2014 OLT Conference Sydney

Workshop on Learning Analytics with Prof Phil Long and Prof Gregor Kennedy

>80

8 July 2014 First Year Experience Conference, Darwin

Joint paper with Ms Cassandra Colvin (UniSA team)

>100

21 August Learning Analytics presentation, CDU

Presentation to introduce learning analytics to the academic community

>50

26-28 August 2014

Blackboard Learning and Teaching Conference, Gold Coast

Workshop on Learning Analytics Use Cases

>80

1 October Nanyang Technological University, Singapore (webinar)

Presentation on learning analytics for staff PD in learning and teaching

>200

8 October 2014 Flinders University Use case workshop with Flinders staff >20

11 November IRU, MRUN workshop, Universiti of Malaya, KL

Presentation on project to IRU/MRUN members

>80

18 November 2014

Charles Darwin University

Use case workshop with CDU staff >20

20 November 2014

Australian Learning Analytics Summer Institute

Panel session >80

23-26 November 2014

ASCILITE Conference, Dunedin NZ

Joint Poster Presentation on Project with UniSA team

13 February 2015 Newcastle University Use case workshop with Newcastle staff >60

16-20 March 2015

Learning Analytics Knowledge Conference, USA

Joint poster presentation on Project with UniSA team

9 April 2015 Let’s Talk Learning Analytics National Forum

Project Forum >150

Learning Analytics: Assisting Universities with Student Retention 18

Appendix G – Scheduled Dissemination Activities

Event Date Event Title, Location Brief Description of Event

16-19 June 2015 35th Annual Conference of the Society for Teaching and Learning in Higher Education (STLHE), Vancouver, Canada

Conference presentation on project findings

17-19 June 2015 10th eLearning Forum, Singapore Conference paper presentation on project findings

22-24 June, 2015 AACE EdMedia 2015: World Conference on Educational Media and Technology, Montreal Canada

Best Practice session on project findings

6-9 July 2015 HERDSA Conference, Melbourne Conference paper presentation on project findings

25-27 August 2015

Blackboard Teaching and Learning Conference Follow up workshop on use cases

27 -30 October 2015

ISSOTL conference Melbourne Paper presentation (if accepted)

Learning Analytics: Assisting Universities with Student Retention 19

Appendix H – High Level Framework Summary

See Next Page

The framework can also be downloaded as a stand-alone pdf from here.

Learning Analytics: Assisting Universities with Student Retention 20

Learning Analytics: Assisting Universities with Student Retention 21

Appendix I – Discussion Questions

See Next Three Pages

The discussion questions can be downloaded as a stand-alone pdf from here.

Learning Analytics: Assisting Universities with Student Retention 22

Learning Analytics: Assisting Universities with Student Retention 23

Learning Analytics: Assisting Universities with Student Retention 24

Learning Analytics: Assisting Universities with Student Retention 25

Appendix J – Evaluation Report

Evaluation Reflections

Charles Darwin University – SP13-3268

Learning Analytics: assisting universities with student retention

Background

The overall aim of this project was to develop a framework which cross references the tools available through learning management systems and learning analytics with variables and actions to improve retention of at risk students. The project addressed this aim through an extended literature review, two national surveys, in-depth interviews and case studies.

The focus of this project is on the use of learning analytics by higher education institutions in Australia, for student retention. The data gathered was related to academic and executive views and uses of learning analytics.

The guiding focus of the evaluation was to determine if the project’s aims were achieved, and outcomes delivered, within budget and on time.

The intended outcomes were to:

- Gain a better understanding of the maturity of the learning analytics implementation and application within Australia

- Explore and identify the range of ways analytics are being used in the sector

- Increase awareness of the role, function and potential uses of analytics for student retention

- Increase awareness of the elements that need to be considered for learning analytics implementation

These outcomes were intended to be accomplished via the following demonstrable outputs:

Overview of use/potential use and limitations of learning analytics

Framework for evaluating use of analytics for student retention

Case studies on the use of the framework to evaluate analytics

National forum

Learning Analytics: Assisting Universities with Student Retention 26

Evidence

The first interactions between the Project and Evaluation Teams were at the OLT workshop for the 2013 Strategic Commissioned Projects, conducted in April 2014. Within the Learning Analytics Cluster, there were two projects, with this project being led by Associate Professor Deborah West from Charles Darwin University, with team members Henk Huijser, David Heath, Alf Lizzio, Carol Miles, Danny Toohey, Bill Searle and Jürg Bronnimann.

In order to identify that the project’s aims were achieved and outcomes delivered both formative and summative evaluation strategies were utilised. The Evaluation team was provided with access to the key documentation from the project team and were included in significant project team communications. In addition, a member of the evaluation team was a participant in virtual and face to face project team and reference group meetings. Specifically, an evaluation team member participated in the 3 day project team meeting in September 2014.

Throughout the lifecycle of the project the evaluation team provided input and advice.

The Evaluator found several key factors that contributed to the successful achievement of the project aim and goals. These factors include:

● Regular meetings of the project team with the Evaluator and Reference Group from the beginning of the project, which were well supported by project plan updates and reports on activities. This ensured that the team were provided formative feedback to further enhance the proposed project outcomes.

● Active and sustained communications between project team members from the partner institutions.

● Strong project management, as demonstrated by extensive and appropriate documentation and insightful input to the project from the project manager.

● Appropriate knowledge of institutional structures and priorities, ensuring that the activities undertaken related to institutional strategies and requirements in this emerging field.

● Establishment and maintenance of a genuine team approach to the project. This was exemplified in the breadth of contribution made by project team members at national forum.

● Diversity of skill set in the project team, which ensured a range of perspectives, breadth of analytical skills, and variation of insight into the project communication requirements.

Learning Analytics: Assisting Universities with Student Retention 27

Project Management

It has been documented that effective project management has the following elements:

● Identifying requirements,● Establishing clear and achievable outcomes,● Balancing the competing demands for quality, scope, time and cost,● Managing the expectations of various stakeholders, and● Adapting plans to overcome challenges.

From a Project Management perspective, the project was well managed and all stakeholder groups were involved. There was effective and significant communication with all members of the project team and involvement of the reference group assisted with project execution and promulgation of project outputs.

From the outset it was evident that this was an active, enthusiastic and well-led project with clear project goals and strategies.

The leadership from Associate Professor Deborah West along with the capable and conscientious oversight of David Heath as Project Manager, were key factors in the success of this project. This facilitated a collaborative team environment with a clear and ongoing focus on the project deliverables. Inclusion of a member of the Evaluation Team in key project discussion provided formative evaluation and input throughout the project and was facilitated by the project leader in a positive and generative manner.

Achievement of Outcomes

The key summative evaluation questions centred on whether the project was able to identify the usage of learning analytics tools, generate a meaningful framework, establish documented case studies and undertake the national forum.

There were over 353 valid survey responses from academic staff, and the data within these surveys were able to be analysed and mapped to identify the context and tensions that are currently present in the use of learning analytics to address student retention.

Institutional data was derived from 23 interviews across 15 institutions, and the completion of 24 institutional level surveys (22 from Australia and 2 from New Zealand). It should be noted that this extent of response at sector level is a positive reflection on the execution of the institutional level survey by the project team.

This was supplemented with five case studies. Survey inform also provided insight into the current state of LMS use in the sector.

Learning Analytics: Assisting Universities with Student Retention 28

The final phase of the project featured the national forum, which included input from the University of South Australia Learning Analytics OLT Strategic Grant project team. The national forum was a well attended (148 attendees) event, that generated informed discussion on the Learning Analytics field in general, and the project outputs specifically.

The national forum in particular was used to elicit the final input to the institutional Learning Analytics framework.

The forum and overall project is supported by the website at: http://www.letstalklearninganalytics.edu.au/

To date (June 2015) there have been 13 workshops/presentations undertaken, addressing and disseminating the project and its outcomes. Over 900 attendees have participated at these events. The project team have reported that a further 8 presentations are scheduled by the end of 2015.

Summary

The project activities, and in particular the national forum, ensured that a large number of stakeholders (institutions, academic managers and educators) were not only party to the collection of the sector data, but also engaged with the rapidly changing agenda of Learning Analytics, and its specific application to student retention.

The form of the surveys and the associated data provide important insights into the tension between institutional level managerial use of learning analytics data and the explicitly educational use of the data. The variation in perspectives identified by the project, and the means by which a whole-of-institution capability is advanced, is a key insight to the effective maturation of learning analytics in the sector. The institutional framework is now a tested tool for extending and informing the necessary dialogue, at multiple levels, within institutions, and it can be expected that the framework will be actively used across the sector.

Overall the evaluators appreciated the opportunity to work with the capable and enthusiastic project team. The extent of interest in the forum, the active nature of the forum, and the ongoing positive inter-institutional dialogue in the weeks following its conclusion is reflective of the value of this project.

Learning Analytics: Assisting Universities with Student Retention 29