development of a framework for evaluating the impact and … · development of a framework for...
TRANSCRIPT
Development of a framework for evaluating the impact and sustainability of mobile learning initiatives in higher education
Angela Murphy Australian Digital Futures Institute
University of Southern Queensland
Helen Farley Australian Digital Futures Institute
University of Southern Queensland
The field of mobile learning is becoming more capable of supporting high quality learning
experiences and students are increasingly demanding greater mobility and flexibility. As a result,
Higher Education Institutions are increasingly considering the implementation of institutional m-
learning strategies. We present the aims and approach of a three year project to be conducted by
the Australian Digital Futures Institute to develop an m-learning evaluation framework (MLEF)
that will aid the selection and justification of m-learning initiatives. The framework will be
encapsulated in an easy to use online evaluation toolkit which will consist of: a standardised
evaluation framework, resources and guidelines; an m-learning maturity model; a database of m-
learning exemplars; and an interactive mobile user model. The project is a collaborative initiative
between USQ, ANU and UniSA and is supported through the Australian Government's
Collaborative Research Networks (CRN) program. Participatory monitoring and evaluation
(PM&E) methods will be used to develop outputs and deliverables.
Keywords: Mobile learning, m-learning, mobile learning, evaluation frameworks, sustainability
Background: Evaluating Mobile Learning Initiatives in Higher Education
As mobile technologies have evolved and become more capable of supporting learning experiences in both
blended and stand-alone contexts, the field of m-learning has emerged as a new learning paradigm and become a
focus of research and development activities (Kukulska-Hulme et al., 2011; Engel et al., 2011). Over the past
ten years, a number of pilot or experimental research studies have been conducted across sectors to investigate
the impact of mobile technologies on learning and teaching (e.g. Elias, 2011; Biggs & Justice, 2011; Wong,
2012). One of the most consistent conclusions of these studies is that there are still a number of barriers that
influence the adoption of m-learning initiatives in education, both at an institutional and at a user level. HE
institutions are cautious about investing extensively in mobile technologies because of the rate of emergence of
new models and the speed with which devices become obsolete. Few HE institutions have therefore
implemented well-financed and highly visible m-learning initiatives that are operationalized within policy and
practice.
A report conducted for the JISC e-Learning programme in late 2010 indicated that the most prominent issue in
the field of m-learning is the lack of full scale evaluations of mobile technology in Higher Education (Wishart &
Green, 2010) and the absence of a stable platform from which to effectively research the role, drivers and
impact of mobility on learning (Park, 2011). A significant challenge facing most HE institutions is identifying
strategic and operational priorities for investment in m-learning capabilities within a rapidly changing field,
while maximising the educational outcomes for students and minimising institutional costs. This poster
describes a three year project to be undertaken by the Australian Digital Futures Institute that will address this
gap by developing an effective assessment mechanism that can be used to evaluate whether m-learning
initiatives are successful, scalable and replicable.
Project Aims and Approach The aim of this project is to develop a framework for mobile learning or m-learning that will enable Higher
Education (HE) institutions, learning designers and educators to evaluate the impact and sustainability of m-
learning initiatives within a range of learning contexts. The m-learning evaluation framework (MLEF) will be
developed to facilitate and support HE institutions in the assessment, development and embedding of m-learning
policies and/or practices to enhance the learning experiences of students and support long-term planning for
improved learner and institutional outcomes. The framework will be independent of specific technologies and
therefore will remain relevant despite the emergence of new devices.
The focus of the project on building an evaluation model that is sufficiently flexible to accommodate the current
and future needs of students and educators for m-learning initiatives. For this reason an iterative approach will
be used to ensure that each commencing stage is built from the insights obtained in the previous stage and allow
the inclusion of new insights and innovations in the field as the research project matures. Participatory
monitoring and evaluation (PM&E) methods will be used as the project involves the development of artefacts,
such as the toolkit, which are aimed at being responsive and relevant to the needs of the education community.
The project will be undertaken as a three-year collaborative initiative between three participant universities;
USQ, ANU and UniSA. Three levels within higher education will be examined as recommended by Quinton
and colleagues (2010): pedagogical, technical and organisational. The following groups that represent each layer
will be consulted during this stage to identify the needs, expectations and challenges of each level when
considering the implementation of m-learning initiatives:
Figure 1: Groups to be consulted in the development of the MLEF
Outputs and Deliverables The primary deliverable of the study will be an online m-learning evaluation toolkit that will provide a set of
procedures, guiding principles, methods and examples to be used by HE institutions and educators when
evaluating m-learning initiatives for inclusion in policy and practice. The m-learning evaluation toolkit will
consist of the following components:
A conceptual evaluation framework and evaluation resources consisting of a set of evaluation criteria and
standardised assessment instruments as well as checklists, guidelines and step-by-step tutorials for the
evaluation of m-learning initiatives within various contexts;
A m-learning maturity model that will enable HE institutions to assess the maturity of their m-learning
capabilities and provide best practice recommendations for policy development and institution-wide co-
ordination and communication;
A database of m-learning exemplars in the form of case studies and resources that have been demonstrated
to contribute to high quality learning experiences; and
A mobile user model consisting of normative data compiled from the research component of the study that
provides insight into the context, background, needs and learning styles of students and enables
benchmarking of the role, drivers and impact of m-learning within various learning contexts and
environments.
The project will result in a rigorous and transferable conceptual framework for m-learning evaluation that will
enable higher education institutions to consider the impact of new m-learning strategies within the context of
current capacities and the future impact on the quality of the student learning experience.
Acknowledgement This project is supported through the Australian Government's Collaborative Research Networks (CRN)
program.
References
Biggs, B. & Justice, R. (2011). M-learning: The Next Evolution. Chief Learning Officer, 10(4), 38-41.
Elias, T. (2011). Universal Instructional Design Principles for M-learning. International Review of Research in
Open and Distance Learning, 12(2), 14.
Engel, G., Palloff, R. & Pratt, K. (2011). Using Mobile Technology to Empower Student Learning. Paper
presented at the 27th
Annual Conference on Distance Teaching and Learning, University of Wisconsin.
Kukulska-Hulme, A., Pettit, J., Bradley, L., Carvalho, A. A., Herrington, A., Kennedy, D. M. & Walker, A.
(2011). Mature students using mobile devices in life and learning. International Journal of Mobile and
Blended Learning, 3(1), 18–52.
Park, Y. (2011). A Pedagogical Framework for M-learning: Categorizing Educational Applications of Mobile
Technologies into Four Types. International Review of Research in Open and Distance Learning, 2(2), 78-
102.
Quinton, S., Pachman, M., & He, R. (2010). Evaluation of the TELT platform. Sydney, NSW: University of
New South Wales.
Wong, W. (2012). Tools of the Trade: How Mobile Learning Devices Are Changing the Face of Higher
Education. Community College Journal, 82(5), 54-61.
Wishart, J. & Green, D. (2010). Identifying Emerging Issues in Mobile Learning in Higher and Further
Education: A report to JISC. University of Bristol.
Author contact details:
Angela Murphy, [email protected]
Helen Farley, [email protected];
Please cite as: Murphy, A. & Farley, H. (2012). Development of a framework for evaluating the impact and
sustainability of mobile learning initiatives in higher education.
Copyright © 2012 Angela Murphy and Helen Farley.
The author(s) assign to the ascilite and educational non-profit institutions, a non-exclusive licence to use this
document for personal use and in courses of instruction, provided that the article is used in full and this
copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ascilite to publish this
document on the ascilite website and in other formats for the Proceedings ascilite 2012. Any other use is
prohibited without the express permission of the author(s).