Macquarie UniversityiLearn Implementation 2012

Evaluation Report

www.mq.edu.au/iLearn/

Macquarie UniversityiLearn Implementation 2012

Evaluation Report

www.mq.edu.au/iLearn/

March 2013

Report Commissioned by Professor Judyth Sachs

Authors

Associate Professor Maree Gosper

Dr Jayde Cahir

iLearn Evaluation Working group

Dr Trudy Ambler - Learning and Teaching Director, Faculty of Arts

Andrew Burrell - Director Centre for Open Education

Helen Carter - Manager Educational Development and Design Group, Learning and Teaching Center

Dr Margot McNeill - Senior Lecturer and Manager iLearn Change Stream, Learning and Teaching Center

Dr Mitch Parsell - Associate Dean Learning and Teaching, Faculty of Human Sciences

LTC-A413-016 • 10 May 2013

CONTENTS

1. Executive Summary 1Key findings 1Recommendations 2The Report 3

2. Introduction 42.1 The Evaluation Framework 42.2 Evaluation Methodology 5

3. Enablement to Support Good Practice and Innovation 83.1 iLearn Governance 93.2 iLearn Management 10

3.3 Policy and Strategy 203.4 Evaluation and Reporting 213.5 Issues and Future Directions 21

4. Enhancement of Teaching 234.1 Satisfaction and Effectiveness 234.2 Transformation in Teaching 254.3 Issues and Future Directions 27

5. Enhancement of Learning 285.1 Satisfaction and Effectiveness 285.2 Transformation in Learning 325.3 Issues and Future Directions 34

6. Conclusion 35

References 36

Appendices 37Students and staff Surveys for Session 2 37iLearn Platform in 2012 47Minimum Standards for Faculties in 2012 48Staff comments and responses to the scaled questions 49A list of iLearn Evaluation reports 53

1

1. EXECUTIVE SUMMARY

In July 2010, MACALT recommended the University adopt a new Learning Management System, Moodle 2.0 as a replacement for Blackboard CE6. The recommendation was subsequently adopted by the University’s Executive and thus began an ambitious university-wide project to develop and implement Moodle and a suite of 13 associated technical integrations collectively referred to as iLearn. In parallel with the development of technical infrastructure has been a comprehensive academic change management program aimed at transforming teaching and learning. Throughout the second half of 2011 and 2012, technical and academic change aspects of the project were developed and trialled with full implementation taking place throughout 2012.

This report outlines the evaluation of the implementation of iLearn in 2012. It covers the effectiveness of the outcomes achieved, the processes employed and the achievement of educational and organisational change; hence, the focus of this evaluation is:

1. Enablement to support good practice: governance and management

2. Enablement to support innovation: building technical capacity

3. Enablement to support innovation: building capacity of staff

4. Quality of learning and teaching: staff experiences

5. Quality of learning and teaching: student experiences

These foci are based on the intention that a change in a LMS needs be more than a migration process between ‘old’ and ‘new’, rather, it should prioritise the enhancement and enablement of learning and teaching;resulting in wider transformation of organisational and technical structures as well as the building of capacity in academics to manage and develop their units.

Key findingsUse of iLearn: In 2012 there was a significant growth in online presence, as there are now 144% more online units. A total of 2426 units were harvested from Blackboard and at the end of the year there were 3506 active units in iLearn. Minimum standards for online units were also implemented and this shows in the adoption of iLearn functions such as the discussion forums, which were used in 100% of units.

iLearn functions: iLearn functions need to be seen to have a practical use otherwise they will be ignored by staff members. Even if there is an identified use staff members have indicated that they need to have the time to first learn how to use it and then implement it into their unit. Lack of time was indicated as the number one issue in preventing staff from exploring all the iLearn functions.

Staff and student satisfaction levels: A majority of staff surveyed indicated they were very satisfied with the levels of support from the iLearn support team. The open-ended responses gave high praise to the iLearn support team. The student satisfaction levels with the way iLearn enables access to unit information and content has increased across the two sessions. The satisfaction levels in general are all slightly higher in comparison to Session 1. Again, the satisfaction levels for all forms of iLearn support and services had increased across the two sessions but so too are the expectations of that level of support.

Transformation in learning and teaching practices: Transformation is a slow process, and while the confidence levels and staff capacity is increasing, the online learning transformation will take years and adequate resourcing will be required to support this ongoing change and development. Across the two sessions there were increases in: staff members’ confidence levels (S1=71.9% and S2=78.1%); staff members’ agreement that the implementation of iLearn made them reconsider how they teach (S1=36.8% and S2=47.4%); and the staff

2

members that would like to use iLearn functions to enhance their teaching (S1=68.4% and S2=87.9%). There were also increases in the number of students that would recommend MQ’s use of iLearn as an example of good practice to other universities (S1=61.4% and S2=69%) as well as increases in the number of students that agree iLearn enhances their learning experience (S1 = 60.6% and S2=71%).

Issues and enhancements: The most common ongoing iLearn technical issue for staff was connectivity and how this impacts on the everyday teaching experience. In comparison to Session 1, the staff satisfaction in relation to the support for iTeach, Echo360, UNITS and OneHelp has increased. The iLearn function that most staff and students would like to see enhanced is the dialogue module. According to the student surveys the predominant issues have changed since Session 1 from problems with the system to problems with how the system is used and a lack of standardisation across units. This suggests that post-implementation will require ongoing professional development.

Post-implementation: The project will be in operational mode at the beginning of 2013, however, continuing to innovate and develop tighter integration with existing and future Macquarie University systems requires ongoing funding. This is not only restricted to the technology, but the development of University policy, the perpetual change management required at all levels, structures for evaluation and reporting; and continuity of the Quality Assurance and Enhancement processes.

RecommendationsA number of issues and implications for future directions have been identified throughout the report and these have been consolidated into seven recommendations.

Continued support for transforming learning and teaching. The iLearn faculty-based teams were very successful and these structures and relationships need to be maintained in the future. There needs to be ongoing commitment to the professional development of staff including sessional staff, to continue both technical and pedagogical capability and confidence.

Future planning for the University: online learning strategy.In the absence of a university-wide IT and web strategy, iLearn has been seen as the default solution for communicating and facilitating various student and staff services which has raised a number of governance, management and resource implications. In view of this, it is timely for the university to develop a more comprehensive and transparent approach which aligns online learning with a University web strategy and a broader IT development plan.

Further development of learning technologies.There needs to be a continued focus on the University’s learning and teaching infrastructure and further alignment work is needed on many of the technologies that form the iLearn platform such as the integrating UNITs and iTeach, resolution for video hosting and web conferencing, the new content management system iShare. For the future clear and transparent strategies and pathways are needed for:

• capturing the expectations and needs of students and staff;

• assessing requests for potential ‘add-ons’, such as web conferencing and a solution for videos;

• adequately resourcing the development and implementation of enhancements.

Development of the University’s leadership role within the Moodle and Open Source Communities.

Further development of the University’s leadership role within the Moodle and Open Source communities of practice (COP) is required. A planned approach is needed which could include

3

strategies for creating new networks, developing and maintaining current networks, hosting events like iShare Symposium in December 2012 and disseminating research findings.

Sustainable funding models for core learning and teaching systems. It is essential that the funding models for core learning and teaching systems are clarified, timely and planned with resources available within the regular funding cycle for routine maintenance and development as well as support for innovation within the university and the broader open-source community.

Ongoing evaluation review and reporting of iLearn.Systematic evaluation and reporting on the performance and effectiveness of iLearn is essential for the continued development of systems and processes that enhance the student learning experience and outcomes. The various feedback and reporting strategies that have been developed through the iLearn development and implementation phases need to be consolidated into a comprehensive evaluation plan, which has a clear focus, systematic data collection, dissemination pathways and strategies for action and reporting on outcomes.

Ongoing development of University policy related to online learning.The iLearn Implementation highlighted a number of gaps in the current University policies, for example there are no policies for social media and there is ongoing work needed for policies relating to copyright and privacy. The policy landscape in relation to online learning requires constant monitoring, and this needs to be recognised in the form of defined roles, responsibilities and lines of communication.

The Report The report is structured in five sections:

1. Introduction –provides an overview of the evaluation methodology and data collection;

2. Enablement to support good practice and innovation – outlines the iLearn governance and management arrangements in relation to the University’s Quality Enhancement Framework and identifies issues and implications for ongoing transformational change;

3. Enhancement of teaching –the experiences of staff;

4. Enhancement of learning – the experiences of students;

5. Conclusions – will provide concluding remarks on the iLearn implementation and the recommendations that have arisen.

4

2. INTRODUCTION

The guiding principles underpinning the development and implementation of iLearn are to prioritise the enhancement and enablement of learning and teaching through:

• Agilityandflexibility to enable the university to stay abreast of change

• Enablement to support innovation in learning and teaching

• Enhancement of the student learning experience

• Sustainability of infrastructure (reliability, security, interoperability) and academic programs

• Quality of teaching and learning through supporting staff in their work; and enhancing the learning experience for students

• Alignment with the University’s strategic directions and priorities

• Consolidation through building on existing expertise and successful practice.

In addition with the transition from Blackboard CE6 to iLearn the intention was for the change to be more than a migration process between ‘old’ and ‘new’. Rather, it should prioritise the enhancement and enablement of learning and teaching, resulting in wider transformation of organisational and technical structures as well as the building of capacity in academics to manage and develop their units, and ultimately the confidence and skills to explore new approaches teaching and learning.

Within the framing of these guiding principles, the evaluation of the development and implementation of iLearn will cover:

• the overall effectiveness of the project measured against the guiding principles

• the critical success factors and lessons learnt

• recommendations for ensuring the sustainability of infrastructure, academic programs and innovation.

2.1 The Evaluation Framework The evaluation of iLearn is based on the evaluation Framework recommended by the ALTC which encompasses four dimensions (Figure 1).

Figure 1. Evaluation Framework adapted from the Centre for the Advancement of Learning and Teaching (CALT) University of Tasmania Project Evaluation Toolkit: http://www.utas.edu.au/pet/sections/frameworks.html

Project Implementation and Completion

Project Processes

Project Commencement

Post-Project

Outputs

Outcomes

Impact

Project Management• Planning• Communication• Resource• Risk assessment• QA• Budgeting, etc.

Project goals/aimsobjectives/proposed outcomes

Learning or teaching product;faculty documentation

Sta�:change in work practice;attitudes or values;organisational change

Students:learning effectiveness/efficiency;changes in attitudes/approaches;improved access to programs;greater flexibility

Possible foci for

evaluation

1. Project commencement

2. Project Processes

3. Project Implementation and Completion

4. Post-Project

5

The focus of this report is Dimension 3 – project implementation and completion which covers the outcomes and outputs achieved at the end of the Implementation Phase in 2012.

Dimension1and2,Projectcommencementproceses were covered in separate evaluation during 2011 led by Dr Beverly Pasian, from University of Applied Sciences, in the Netherlands. Dr Pasian used the iLearn Project as case study to evaluate a framework she developed for identifying non-process factors that contribute to project management maturity.

Dimension4,Post-projectimpactcoveringlongertermeducationalandorganisationalchange requires longer term strategies extending over two to three years. A Macquarie University PhD Scholarship has been awarded to Adriana Voerster to explore the nature and impact of transformational change taking place as a result of the transition to iLearn. The Scholarship resides in the School of Education, under the supervision of Associate Professor Maree Gosper and Dr Laurie Field. There is scope in this PhD candidature for exploring more specific questions of transformational change relating to, for example: the nature of that change and the impact it is having on the design and delivery of the curriculum; the engagement of students and the achievement of quality outcomes; innovation and sustainability in teaching practice; and the role of distributed models of leadership in the change process. Ms Voerster commenced her studies in December 2012 and is currently developing her research plan. 2.2 The Evaluation Team

To conduct an evaluation of outputs and outcomes of the iLearn implementation during 2012, the iLearn Steering Committee established an Evaluation Working Group, chaired by Associate Professor Maree Gosper to conduct the evaluation. The Working Group has a membership representing the interests of Faculties, the Learn team, the Learning and Teaching Centre and the Centre for Open Education:

• Associate Professor Maree Gosper (Chair)

• Dr Trudy Ambler, Faculty of Arts

• Andrew Burrell, Centre of Open Education

• Helen Carter, Learning and Teaching Centre.

• Dr Margot McNeill, iLearn Team, Leraning and Teaching Centre

• Dr Mitch Parsell, Faculty of Human Science

Dr Jayde Cahir was appointed on a part-time basis to manage the evaluation, under the direction of Associate Professor Gosper and in consultation with the Working Group.

Michael Marston and the TEDS Team at the Learning and Teaching Centre have provided services in the development and delivery of online surveys of staff and students. Cathy Rytmeister (LTC) has undertaken additional statistical analysis of survey data.

The focus of the evaluation was decided in consultation with the Working Group and this was subsequently endorsed by the iLearn Steering Committee, MACALT and the Provost.

The Working Group has discussed the iLearn summative evaluation on a regular basis during 2012, either in meetings or via email. Evaluation updates have been reported on a regular basis to the iLearn Steering Committee and MACALT.

2.2 Evaluation MethodologyThe focus of the evaluation of the implementation of iLearn is to:

• review the overall effectiveness of the project measured against the guiding principles;

• review the effectiveness of the management and communication strategies in place to achieve the outcomes;

• identify the critical success factors and lessons learnt;

• make recommendations for ensuring the sustainability of infrastructure, academic programs and innovation.

To achieve this, the evaluation is structured around three key themes:

6

1. Enablement to support good practice and innovation in learning and teaching through:

• The development of organisational supports e.g. procedures, business rules, quality processes, analytics for systematic reporting;

• Building staff capability e.g. through training, support, professional development, Faculty liaison teams;

• Building technical capacity e.g. through systems integrations, iTeach.

2. Enhancement of teaching and learning explored through staff experiences, which included levels of engagement with all iLearn functions and satisfaction with services and support facilities.

3. Enhancement of the student learning experience explored through students’ experiences, which included levels of engagement with all iLearn functions and satisfaction with services and support facilities.

Several data-gathering techniques have been employed: surveys, interviews, staff reflections and a review of the 2012 iLearn Portfolio, Netspot/LTC group, Informatics/LTC Integration group and iLearn Steering Committee meeting minutes.

2.2.1 SurveysSurveys of staff and students were conducted in Session 1 and repeated in Session 2, 2012 to identify engagement with iLearn, satisfaction with services and support and map changes to learning and teaching across the University.

The Surveys were modelled on the SEET Survey exploring students’ experiences and expectations of technologies at University (Gosper, Malfroy & McKenzie, In press). This provided a reference point for gauging changes in the satisfaction levels of students in relation to their learning experience and the services and support provided.

The survey consisted of a combination of multiple choice, scaled and open-ended response questions. The student survey sought information:

1. Demographics relating to the learning context e.g., mode of study, Faculty and Department;

2. Technologies used to access iLearn e.g., desktop computers, mobile devices, networks;

3. Experiences of learning and teaching using iLearn functions organised in three categories:

• Functions for organising study e.g., announcements, calendar, RSS feeds, assignment submission, gradebook;

• Functions for engagement with unit content e.g., links to web sites and eReserve, Echo360, videos, lessons module, book module, glossary;

• Functions for engaging with learning activities e.g., Dialogue tool, discussion forums, blog, chat, quizzes, wiki, database, workshops, twitter feeds;

4. Tools participants would like to make more use of in the future (listing all iLearn functions);

5. Overall satisfaction with iLearn and associated support services;

6. Most valuable features and suggested improvements.

For comparison purposes, the staff survey mirrored the student survey, with questions being rephrased to reflect a teaching rather than learning focus. Additional questions were added to capture skills development, confidence in using iLearn, pedagogical change and administrative tools such as iTeach and UNITS.

The Surveys were delivered through University evaluation system, Teaching Evaluation for Development Service (TEDS), using EvaSys.

7

AnalysisThe results of the surveys were generated through EvaSys and further analysis was undertaken with SPSS and Excel. Open-ended questions were analysed through NVivo.

ParticipantsFor the student surveys a sample of 25% of the student population was invited to participate in the student surveys. Selected sampling was undertaken in an attempt to mitigate the effects of over surveying. Different samples were taken for each Session, which were representative of undergraduate and postgraduate profiles; internal and external profiles; and Faculties and OUA. In Session 1, the Survey was administered between 1st – 20th May and in Session 2 between 1st – 21st October.

For the staff surveys invitations were sent to all staff members who were registered in iTeach as convenors or tutors. In Session 1, the survey was administered between 19th June - 10th July and in Session 2 between 7th and 28th October.

DisseminationofsurveyfindingsMACALT, the iLearn Steering Committee, iLearn Portfolio Group received regular updates and reports on the results from the surveys. A summary of the findings was published on the iLearn website and publicised across campus through the MQ Marketing channels. There are also several publications in academic journals that are currently under review.

2.2.2 Focus Groups and InterviewsFocus groups and interviews were planned for teaching academics and iLearn project team members in Session 2 to explore the themes in the staff survey in more depth and identify issues and implications for moving forward post implementation. (See Appendix 1 for the list of questions). Invitations to participate were issued through the Session 2 Staff Survey. Only one academic staff member volunteered thus the focus groups could not be conducted as originally planned and an interview was undertaken.

The interview was transcribed and presented to the academic staff member for validation before it was analysed. Following confirmation that the transcript was a true representation of the one-on-one interview the data was analysed and sent to the appropriate teams as a source of feedback.

The findings from this interview were used along with the responses to the open-ended questions in the survey to provide a more nuanced interpretation of staff experiences of iLearn.

iLearn project team members were invited to participate in a Focus Group and due to a low response rate, feedback was gained informally through questions posed after team meetings. Notes were taken and later checked by team members for accuracy. In addition Faculty implementation team members produced regular written reflections of the implementation process and these were also sent to the evaluation officer for analysis. Interviews were also conducted with team leaders and key project personnel.

2.2.3 Document ReviewThe evaluation officer was responsible for taking the minutes for the following meetings: iLearn Steering Committee, iLearn Portfolio, Netspot and LTC/Informatics Integration Team. These records were reviewed to identify issues related to the theme of enablementtosupportgoodpracticeandinnovation.

8

3. ENABLEMENT TO SUPPORT GOOD PRACTICE AND INNOVATION

It was recognised from the outset that the move to iLearn was ambitious due to the underlying principles of transformational change in learning and teaching. Transformational change by its nature a whole of environment approach encompassing: technical, academic and organisational infrastructure; executives, senior managers through to individuals; and both formal and informal organisational arrangements and enablers. The scope of this evaluation cannot capture the complexities of what transpired. Instead we aim to identify the planned mechanisms that were put in place to enable and support good practice and innovation, the key issues that have emerged and the implications these have for the iLearn as it moves beyond the intensity of the initial development and implementation phases into the post implementation ‘business as usual’ phase.

Macquarie University’s Quality Enhancement Framework (Figure 2) identifies four organisational enablers that are fundamental to enhancing good practice and innovation in learning and teaching - governance, management, strategyandpolicy, and evaluationandreview.

In 2008-2010 the Blueprint for the Future Project used this Framework as the basis for the Quality Enhancement Framework for Learning Technologies. The development and implementation of iLearn therefore had the benefit of being situated within an environment with well-established institutional level governance and management structures, policy frameworks and reporting strategies for learning technologies. Each of these elements is reviewed below, to first of all to provide an overview of how iLearn is integrated into the University’s structures and practices and second to situate good practice and innovation within the quality enhancement framework.

Figure 2. Macquarie University’s Quality Enhancement Frameworks

9

3.1 iLearn Governance The governance and management arrangements for iLearn are depicted in Figure 3. Governance is enacted through three cascading levels of responsibility:

MACALT - PROVOST - university wide stakeholders;

iLearn Steering Committee - faculty management and iLearn project team representatives;

iLearn portfolio team – Learning and Teaching Centre representatives.

When issues arise, for example at Faculty level, they are flagged within the relevant project stream (Technical or Change) and presented at iLearn Portfolio Meeting. If necessary they are then escalated to the iLearn Steering Committee, and should further strategic input be necessary they are then presented to MACALT.

Figure 3. iLearn integrated governance and management structure

At the MACALT Meeting 12 December 2011 a transparent process around decision making for enhancements to the University learning and teaching platform was agreed to. MACALT will be the ultimate decision-maker; but there needs to be a means of getting legitimate projects forwarded to MACALT. It was agreed:

• Operationalenhancementsthatdon’thavebudgetimplicationsordon’tvisiblyimpactthewholeUniversitycancontinuetobedealtwithbytheLTC.

• Otherrequestsfromindividuals,FacultiesandfromtheEmergingTechnologiesGrantsScheme,aretobefilteredandprioritisedthroughtheiLearnSteeringCommittee-whichwillcontinueinsomeformpost2012implementation.

• RecommendationsaretobeputtoMACALTforendorsementtoensurealignmentwithUniversityprioritiesandalsoalignmentwithbudgetprocesses.

10

Issues and Future Directions Roles and responsibilities of the iLearning Steering Committee and MACALT: The three tiered governance structure has remained in place throughout the implementation period. One of the issues to arise is the scope of authority in the post-implementation phase, namely, iLearn Steering Committee and its relationship to MACALT, for example:

To what extent is the iLearn Steering committee empowered to make decisions and what is the nature of these decisions?

Is MACALT an advisory or a decision making body?

Where does the final decision rest for issues that are referred to MACALT?

Relationship with the Senate Learning and Teaching Committee (SLTC): One of the strengths of the iLearn project has been the close integration of technical and academic perspectives. Both the SLTC and MACALT have responsibility for the learning and teaching environment however technical infrastructure is largely referred to MACALT and the carriage of academic policy (e.g. the Learning Technologies Policy) is through the SLTC. The dual interests have been addressed through cross-representation on both Committees. However, it may be time to review this relationship and decide whether a more formal transparent arrangement is required.

3.2 iLearn ManagementThe iLearn Project has been managed through three streams – a Technical Stream to manage the integration of Moodle and accompanying technologies into the University environment, a Change Management Stream to manage the integration of new technologies into academic practice and a Project Management Stream to monitor the overall project. Project management is beyond the scope of this evaluation however we will comment on communications strategies as these were vital to all aspects of the project.

3.2.1 Technical Stream The guiding principles underpinning iLearn platform were: to enable and enhance learning and teaching through sustainable infrastructure (reliability, security, interoperability) and agility and flexibility to enable the university to stay abreast of change. During the development and implementation phase of iLearn (2011-2012) activity has been focussed on core systems and services managed through the Technical Stream. As outlined in the iLearn Implementation Report 2011-2012 (p. 27), the deployment of a hosted Moodle service across multiple environments encompassed:

• The configuration, customisation, enhancement and further development of that service to meet the identified functional requirements;

• The integration with related Macquarie systems;

• The integration with third party systems providing additional functionality;

• The migration of existing content from Blackboard CE6;

• The development of a number of internal systems that provide help, reporting or other administrative functions.

An additional un-scoped objective to deliver the Equella Learning Content Management Systems was incorporated during the course of the project. In all, the full implementation involved 13 integrations between the Moodle environment and other systems – Echo 360, iShare (Equella), Turnitin, iTeach, UNITS, OneID LDAP, Australian Access Federation (AAF), Dataswitch, Learning Activity Management System (LAMS), Wimba Voice Tools, Help.iLearn (FAQ engine), Evasys and Crocodoc API. Further details are available in Appendix 2.

11

Building technical capability Building the technical capability within the new LMS involved collaborations between the three key stakeholders - the Learning and Teaching Centre, Informatics and Netspot.

Capability building involved the accommodation of two levels of activity:

1. operational imperatives to ensure sustainable infrastructure of current operations; and

2. the development of capability for ongoing enhancements.

1.OperationalimperativestoensuresustainableinfrastructureofcurrentoperationsNetspot is the host organisation for the iLearn platform. Informatics developed two systems for iLearn: iTeach (formally known as MORTICE) and UNITS. Specific management structures have been put in place (See Figure 1) to manage and monitor the collaboration between the Learning and Teaching Centre and both Netspot and Informatics. In addition policies and procedures for the daily management of the iLearn platform that cover all facets of iLearn day-to-day management have been developed (see: https://wiki.mq.edu.au/display/iLearnOps/iLearn+-+Policies).

These arrangements were not reviewed as part of the evaluation. Instead the experiences of staff and students in accessing iLearn was captured to identify issues for future development and planning.

Staff access to iLearnStaff members were asked, in the two surveys, to indicate the frequency that they use desktop computers, laptops, SMART phones and tablets to access iLearn, which included the networks they most frequently use and from which location. This data was collected using a Likert 5 scale divided into the following categories: Never or rarely, A few times a Session, A few times a month, A few times a week, At least once a day. Figure 4 shows the two ends of the spectrum for Session 1 and Session 2 -the technologies that are used once a day and the ones that are never or rarely use. It also reveals that there was no change in their use over the year as mobile phones, tablets and the OneNet network were consistently ‘never or rarely’ used by staff.

Figure 4. Ways of accessing iLearn by staff

12

According to findings the high expectations surrounding mobile devices and online learning is currently overestimated. Online learning via Smart phones is its infancy, this not only relates to the wireless network issues but the cost of these devices, the ongoing maintenance and the practicalities such as the iLearn layout is not suited to mobile screens. This is a similar situation for Tablets, however, the most common complaint related to the unreliable network access.

The OneNet network, despite the technology, received a high representation of being ‘Never or rarely’ used. The staff responses to the ongoing technical issues experienced over the Session offers some indication for why this is the case. Staff comments included:“LackofaccesstoOneNetinsomeareasofthecampus”;“MQwirelessnetwork issoslow it’sunworkable”; “Variousproblemswithnetworkover thesession”; “OneNetneverworks!”; “TheOneNetnetworkisunfortunatelyslow,andpronetodropouts”and“Themainthingisprettyappallingspeedsonthewirelessnetworkoncampus,andevenphonecoverage,whichmadeitnotworththeefforttouse”.

During this evaluation study it became apparent that a University network upgrade is needed to cope with the demands of users as well as the dramatic increase in the number of online units. Staff members’ use of technology to access iLearn is essential data for the future planning of the Macquarie University network. The future planning of the Macquarie University network needs to consider the present and future reliance on wireless technologies as well as the fact that the University now has twice as many online units and is striving to increase student numbers.

Students access to iLearn The results from the student surveys shown in Figure 5 were similar to that of staff. Students were asked to indicate the frequency that they use desktop computers, laptops, SMART phones and tablets to access iLearn, which included the networks they most frequently use and from which location. This data was collected using a Likert 5 scale divided into the following categories: Never or rarely, A few times a Session, A few times a month, A few times a week, At least once a day. In the table below we have listed ‘Once a day’ and counter referenced it against the ‘Never or Rarely’ category to show the distinct differences. In both surveys the results were almost identical.

Figure 5. Ways of accessing iLearn by students

13

The most common technology used to access iLearn were laptops at other locations or desktop computers at home whereas the technologies that were ‘never or rarely used’ were Tablets on campus or at other locations. In both surveys the results were almost identical and furthermore correlate with the findings from 2010 Student IT Experience Survey. The results are also very similar to the staff survey, aside from staff predominantly using of desktop computers at the University and students usually using laptops at other locations.

OneHelp -iLearn service desk All enquiries from staff and students were channelled through OneHelp. Over the year, staff satisfaction in relation to the OneHelp support for iLearn, iTeach, Echo360, increased as can be seen by the survey results from Session 1 and 2 shown in Figure 6. The percentages shown are for staff who were satisfied to very satisfied with the different aspects of support shown.

Figure 6. Staff satisfaction with OneHelp

There was a significant increase in staff satisfaction levels with Onehelp over 2012 because of this the neutral/mixed feelings category decreased but the dissatisfaction levels remained almost static with a range of 15.6% to 18.8% in Session 1 and 15.5 to 17% in Session 2. Overall, staff satisfaction levels did increase with regards to learning systems support but so too are the expectations of the level of support required which has ongoing resource implications.

Student satisfaction with the support provided is shown in Table 7 and shows the percentage of students who agreed/strongly agreed with statements about the support provided. Satisfaction levels were relatively low but these are balanced by a high representation of neutral or mixed feelings and the very low representation of dissatisfaction amongst students. In both Sessions the ‘neutral or mixed feelings’ category dominated the responses and this could be for any number of reasons like: students have not used the service, students have had mixed experiences with the service or students did not receive the information they needed through the automated help service. Unfortunately, students reasons for selecting the neutral or mixed feelings category were not elaborated upon in the open-ended responses.

Overall, the satisfaction levels, in Session 1, were slightly lower than the SEET Survey in 2010 where 55% of students indicated that they were satisfied with the availability of support services like just in time IT help, e-learning and service desk help. However, on a positive note, as the above graph illustrates, the satisfaction levels with online support for iLearn have increased between Sessions 1 and 2.

14

Figure 7. Student satisfaction with iLearn Support

2.ThedevelopmentofcapabilityforongoingenhancementsThe development of capability to support enhancements requires the development of processes and procedures to cater for initiatives forthcoming from stakeholders on campus and those from the wider open source community.

On-campus enhancements Throughout the duration of the project requests for enhancements to iLearn have emanated from:

1. Staff with particular needs;

2. Minor operational enhancements identified through the LTC;

3. Major enhancements identified by the LTC arising from operational imperatives and visibility of future directions;

4. Requests arising from Faculties and other organisational units e.g the extension of Turn-It-In to include the Grademark option to manage assignment submission and marking.

To manage enhancements and customisation requests to iLearn emanating from points 1 and 2 the following structures were put in place:

• iLearn Enhancement Teams and the iLearn Exchange were established to field requests;

• The iLearn Reference group was established to filter and manage the requests.

During implementation Level 3 and 4 requests were dealt with through a recommendations moving through iLearn Portfolio, the iLearn Steering Committee with reference through to MACALT where necessary.

The iLearn Enhancement Teams: are made up of virtual special interest groups that focus on current and future system improvements. These teams communicate via a wiki to pool their information and plans. The iLearn Exchange is a monthly gathering for staff to share what has been working well in iLearn, as well as help suggest and investigate possible enhancements. Once the feature request is received it is then investigated by one of the special interest groups established within the LTC and that group offers recommendations. The recommendations are then sent for development to the iLearn Reference Group for review.

15

The iLearn Reference Group (formally known as the iLearn Operational Reference Group): was established mid-2012 to advise the Learning Technologies Group in the LTC on issues that affect the operation of iLearn. This group’s role is:

1. To advise on issues relating to the operations of iLearn and other Learning Technologies;

2. To gather, assess, prioritise, and recommend improvements to iLearn and other Learning Technologies;

3. To oversee project groups that are established to develop and implement improvements to iLearn and other Learning Technologies;

4. To approve minor releases to iLearn;

5. To refer issues to iLearn Steering Committee.

Open source community In 2010, MACALT identified the opportunity for Macquarie University to take a leadership role in the Moodle Community of practice. Macquarie University is one of sixteen Australian universities, hosted by Netspot that is currently using Moodle at an enterprise level. One aspect of the Moodle Community is Netspot’s Online Client Community which provides regular updates in ‘Conversations’ if there is a larger issue that is effecting multiple institutions. Netspot has also endorsed Moodle plug-ins and supported the development of these through the Netspot Innovation Fund 2011-2012. One example of the University’s leadership role was the iShare symposium in December 2012.

With the move to iLearn and specifically Moodle, Macquarie is now working within an open-source development environment. While open source opens a number of opportunities for development, it is not an open environment where ‘everything goes’ and the ongoing development of iLearn requires careful management. Any decisions regarding upgrades, inclusions and modifications ought to be made with a view to maximising applicability across the university, stability of systems and compatibility with the future directions of iLearn.

IssuesandFutureDirectionsUniversity Network: The need for a University network upgrade is an ongoing issue. This issue was connectivity and how a slow system impacts on the everyday teaching experience. The comments in the staff survey referred to a various range of technical issues including: system slowness; the unreliable network connects (including wireless) across campus; incompatibilities with browsers, iPads and iPhones; and remote access issues.

Resourcing issues: There were resourcing issues within Informatics, which transpired into the slow delivery of iTeach and UNITS and delays in subsequent work. Requests made by the LTC were repeatedly delayed by Informatics and this was due to developers in informatics being dispersed across several projects rather than having dedicated resources. There was also pressure on resources within the LTC with the late addition of Equella into the iLearn platform as well as the launch of UNITS in the lead up to Session 2.

Funding enhancements: A key issue and potentially the biggest barrier to maximising return on investment with regards to the technical capacity of the iLearn platform is the lack of clarity around post-implementation funding models for future development and enhancement. It is essential that the funding models for core learning and teaching systems are clarified for sustainability and consolidation to be fully realised.

3.2.2 Change Stream The integration of iLearn into academic practice was the responsibility of the Change Stream with a specific charter to:

• Foster and support;

• Transform online learning using ilearn as a catalyst for enhancing their learning and teaching environments;

• Build capability by supporting academics to learn how to design and build their units in a dynamic, open source environment, rather than relying on centralised development.

16

To achieve this a hub and spoke model was adopted with core support and services managed within the LTC (educational development and design support, training and support) and these were aligned with Faculty Support Teams duplicated in each of the four Faculties. An important facet of change management is communication and these networks were built and used by the iLearn Liaison Officers, iLearn Training and Support Officers.

FacultySupportTeamsEach Faculty had their own support team consisting of:

• Liaison Officer to work closely with and understand the specific needs of the Faculty/Office in relation to the transition to iLearn;

• iLearn Training and Support Officers - working with the Faculty and supported by a Central LTC Help Desk;

• Educational Developer;

• Educational Designer (Formally Online Educational Designer);

• Faculty personnel.

The approach by each faculty team was to guarantee that staff members were able to access the support or information they required, not only so that they could become confident users, but also to feel confident when using iLearn and be prepared to explore new approaches to their teaching.

Faculty Implementation Plans: Each faculty developed their own Faculty Implementation Plans which identified the tasks, timelines, internal processes associated with training and support needs for their academic staff members, transition of learning materials from Blackboard to iLearn through 2011/2012 and projects for LTC EDDgroup to work on (Faculty Partnership Program). All units were required to have an online presence and a key feature of the implementation plans was the development of minimum standards. The standards set for the beginning of 2012 are listed in Appendix 3.

Faculty Support Plans: provided details of the support available. All support requests were managed through ilearn.help@mq.edu.au with any issues sent to iLearn Training and Support Officers as required. The central help desk briefed the iLearn Liaison Officers on any major issues or priorities that would need to be managed within their assigned Faculty/Office. The feedback and requests received through ilearn.help@mq.edu.au were also used to develop FAQs and training workshops as well as the planning of information sessions and any other assistance available to staff. Based on staff members’ feedback this support strategy was successful in streamlining support procedures, which again is highlighted in the results of the staff survey.

Several strategies were employed to encourage networks and communities within and across Departments to share ideas and experiences:

• iLearn Champions and Example Units: early adopters were identified as Champions and invited to work with the Educational Developers and Online Educational Designers to create Example units highlighting various iLearn functions, for use in training, showcasing and as freely available resources.

• iLearn showcases: example of practice conducted within Faculties and in university wide forums enabled convenors to share experiences and ideas.

• iFest: a special program of Unit Showcases and ‘special’ workshops based on staff and Faculty needs was held prior to Session 2.

• iLearn Exchange: the iLearn Lunch was remodelled as the iLearn Exchange in the transition to business as usual. It is held monthly around a particular theme, and has the mission of encouraging staff to share what has been working well for them as well as to investigate possible enhancements to iLearn.

Non-FacultySupportCustomised support was also provided to organisational units operating outside Faculty structures: Centre for Open Education (COE); Australian School of Advanced Medicine (ASAM); Applied Finance; Macquarie City Campus; Macquarie Graduate School of Management (MGSM) and Sydney Institute of Business and Technology

17

(SIBT). Unsolicited positive feedback at the end of Session 2 from these offices such as: “TheiLearntransitionwasthesmoothesttransitionI’vewitnessed”.

CentralisedTrainingandSupportThe iLearn training program was designed to scaffold learning and to support proactive and retroactive facilitation. In order to build online competence and enable academics to choose options to best suit their needs, training was designed to be available face-to-face and online. The training sessions (detailed at www.mq.edu.au/iLearn/training.htm) included: iLearn Tours; Moodle Basics 1 and 2; Moving to Moodle workshops; Specific Moodle Tools workshops; New to iLearn Convenors; Sessional staff training; Faculty based training. The iLearn training program included a total of 459 iLearn workshops with 3113 attendances during 2011 & 2012. The iLearn online training unit was developed for staff members that could not attend face-to-face workshops and up until Session 2, 2012, a total of 121 participants had completed this course.

The iLearn Drop-in Clinic: was implemented to support staff at the beginning of each session. The iLearn Drop-in Clinic provided face to face training and support for staff during this critical time. It was designed to address academic staff members’ last minute questions or concerns. This was an additional form of support so that workshops, online resources and the help desk were not the only way of staff members obtain answers to their iLearn queries. In 2012, the Drop-in Clinic was very popular amongst staff with a total of 377 visits at the beginning of Session 1 and 463 visits in Session 2 and because of this success the initiative will continue in 2013.

A range of iLearn self-help resources for use within and outside training sessions was developed. Overall, 87 Quick Guides; 26 Fact and Information Sheets; 34 Example units; 8 Student information pages were developed as well as an iLearn Quick Tour. The resources have been used extensively by staff. Table 1 gives an indication of the ‘unique page views’ thus it does not count multiple views from the same user. When comparing the figures for 2011 and 2012 there has been an 800% increase in the number of ‘unique page views’. The entire web site had 13,947 unique page views in 2011, compared with 182,606 from January 2012 to end of November in 2012.

Table 1. Unique page views for iLearn self-help resources for staff.

Resource entry page

Training entry page

Quick guides

Info sheets

iLearn Tour

Example units

2011 Total 592 926 834 174 261 823

2012 Total 5,440 5,703 7,041 1,407 1,212 2,038

2012 Jan-June 2,724 4,350 3,758 901 850 1,303

2012 July-Dec 2,716 1,353 3,283 506 362 735

In addition to this range of resources there is also an iLearn Style Guide Working Group that is focusing on the visual representation of iLearn units in an effort to increase the standardisation of units visual presentation, which also includes accessibility and copyright requirement. The iLearn Style Guide was made available for use in the Drop-in Clinic in 2013 and will be presented to Faculties shortly for their feedback and customisation.

iLearn online resources for students: Prior to the roll-out of iLearn in 2012, student awareness campaigns began in 2011. At the beginning of Session 1, 2012, student postcards were printed and distributed across campus, iLearn advertisements ran on the big screens around campus starting for a fortnight starting in O-week. The Enrolment pack and Student Diary for 2012 also contained information about iLearn. The campaign focused on introducing iLearn as the new LMS, system login details, direction for online resources and OneHelp. A similar campaign was run in Session 2, 2012. In addition to, and in response to concerns by convenors that they would be expected to “train” students to use iLearn, a resource was produced for convenors to run for their students at the start of lectures or orientation covering the same information above.

18

Table 2 gives an indication of the use of resources by student in 2012 Once again, the numbers are for ‘unique page views’ only for the first half and second half of the year. This was the first time these resources were made available to students thus there are no 2011 figures available to make a comparison.

Table 2. Unique page views for iLearn self-help resources for students.

2012

Stud

ent

Info

Pag

e

Get

ting

St

arte

d

Acc

ess

to

iLea

rn

Tech

nica

l In

fo

Stud

ent

Gui

des

Onl

ine

st

udy

tips

Confi

dent

-ia

lity

Feed

back

Ass

ignm

ent

Jan-June 10,078 3,340 4,347 959 1,637 575 248 219 3,693

Jul - Dec 3,065 1,187 6,433 467 2,228 420 1,261 102 10,241

CommunicationA central element to change is communication. Multiple forms of communication were relied upon during the implementation of iLearn in 2011 and 2012, as outlined in Table 3.

Table 3. Communication strategies for iLearn

Strategy Details

iLearn homepage This homepage where staff and students units were displayed was used to communicate various announcements by creating a ‘block’, for example the summary survey results were published here.

iLearn website This website was used for any new or notices and there are designated sections for Staff , Students and online resources for all.

MQ Announcements This announcement system managed by Marketing was for campus wide implementation information only thus following the prerequisite conditions of use

iLearn Project gmail group

This email address was used for internal project team communications and each email is entitled ‘Heads up’. This address will be changed to ilearn.team@mq.edu.au in 2013 to suit the new configuration of the group

iLearn News gmail group

This email address was used for internal University communications, particularly system downtimes, outages and other important announcements. In 2013, there will also be an opt-in facility for anyone to join this email group

iLearn updates newsletter

This newsletter was instigated by the iLOs during the implementation of iLearn and included information like: iLearn updates, workarounds, issues, training, resources and any other area of iLearn that required further promotion.

iLearn project team newsletter

This newsletter was sent during the implementation of iLearn, the audience was the project team and the content related to communications from the Technical or Change streams.

LTC newsletter The LTC newsletter was sent to 200 subscribers and to the HODs for distribution to department staff, any appropriate information regarding iLearn is included.

Internal Learning Systems & Services (LSS) documentation and recording

Systems like: One Forge Radar, OneHelp, Wiki were used to keep track of known issues, bugs, fixes within the LSS team.

Meetings Internal meetings provided a chance to share information internally and the minutes a record of everything said

Training workshops iLearn implementation information was also disseminated during face-to-face workshops

19

All these communication strategies will either be continued, developed upon or amalgamated post implementation. During the implementation phase the iLearn Liaison Officers largely took on the role of communicating technical issues and outages to staff members when necessary but moving forward this will be done by Learning Systems Team within the LTC. This centralised plan is for operational communication and includes the transition from implementation to business as usual. The technical stream will be introducing in 2013 a new system status page, showing all systems supported by the LTC, which will offer increased transparency for all iLearn users. The introduction of this new system is also included in the draft operational communication plan.

StaffSatisfactionwithSupportProvidedA majority of Macquarie staff surveyed (Figure 8) are very satisfied with the iLearn support they have received from the iLearn support team during the implementation of iLearn, and the satisfaction levels increased in Session 2.

Figure 8. Staff satisfaction with support from the iLearn Support Team

There was nothing but praise for the iLearn support team, specifically, the iLearn Liaison Officers, Faculty iLearn Support Officers and the LTC Trainers. Some indicative comments included: “ExcellentworkfromalloftheiLearnsupportteam”;“IcannotfaultourfacultyiLearnteam,oncetheygottheproblemtheywereextremelyhelpfulandefficient - they foundwork-aroundsand listened toourcomplaints”; “iLearn supportpeopleare superb”; “ThebestthingaboutiLearnisthesupportteam.Wereitnotfortheirhappy,helpful,andcalmpresence,Ithinkthatmostofthelecturerswouldhavejustgivenup!”;“TheiLearnteamhavebeenoutstanding.Fabulousongoing,individualisedsupport”.As these comments suggest one of the strength of the Faculty-based support teams was their ability to deliver localised service, which was essential to assist the transformation of online learning and teaching. There was a very low percentage of staff surveyed that indicated they were dissatisfied. In the open-ended questions some staff members referred to selecting the netural/mixed feelings category because they did not use this form of support.

During the iLearn implementation the Faculty teams built close linkages with the Learning and Teaching Centre’s general Professional development program in the lead-up to Session 2. Faculty teams were well positioned to maximise opportunities for building networks and encouraging transformation within Departments,

20

for instance; they were able to facilitate program-wide conversations around curriculum design and online learning. This is a substantial change from the University’s long-standing culture of individualised approaches to unit design. The faculty teams were also able to facilitate compliance with transformation initiatives such as the implementation of online minimum standards across the University.

While the Faculty-based support teams were in a position during the iLearn implementation to build and maintain close relationships with each faculty and encourage transformation through these faculty networks, it is unclear how this will change in the post-implementation phase. This is a concern as the transformation is only beginning and staff members will require ongoing support.

IssuesandFutureDirectionsFunding: The main issue and potentially the biggest barrier to maximising return on investment with regards to the building staff capacity is the lack of clarity around post-implementation funding models to guide the transformation and change management process. Further work is needed in integrating iLearn into academic process and with the reduction in iLearn support after the project phase consideration will need to be given as to what type of support can be provided by the LTC staff and the boundaries of that support.

Resourcing: The level of resourcing will may have an impact on the rate at which transformational change takes place. Resources are needed for the team responsible for working with academics and Faculties to support delivery of Threshold standards, Quality Enhancement processes, for example, the inclusion of publisher sites on iLearn, which requires a scalable solution and boundaries around LTC support; as well as a consistent solution around boundaries of support within the Faculties and ongoing analysis of usage within iLearn in order to make informed decisions about training and resources needed.

Communication with stakeholders: During the implementation phase the iLearn Liaison Officers largely took on the role of communicating all technical and change management issues to Faculties and the project team. A major issue in the Faculty-based teams is in the planning for post-implementation, as it was no longer envisaged that the iLearn Liaison Officer role would be required when the project moved into operational mode. This is not the case, as the transformation has only begun and the change management processes will be ongoing and the strong relationships that have been built during the iLearn implementation need to be maintained.

3.3 Policy and Strategy Development and implementation drew on existing University policy to develop business processes and procedures around the deployment of iLearn in particular learning Technologies Policy and Acceptable Use Policy (AUP); however during the implementation gaps were revealed within the Universities current policies on Copyright, Privacy and Confidentiality and Acceptable Use Policy (AUP). In addition it has highlighted the need for policy around Public Comment and Social Media.

CopyrightIt is becoming increasingly difficult to apply the current Copyright law to the ever evolving development of digital media and technologies in general. A study of the implications for iLearn in relation to Copyright Law was undertaken as part of the project in order to identify gaps and potential issues before the next University Copyright Audit by Copyright Agency Limited (CAL) in 2014. There is evidence of very limited understanding about copyright basics amongst University staff members, e.g. use of eReserve, attribution of sources of knowledge in online units, copyright and Echo360 as well as copyright ownership and the distinction between economic and moral rights. The Copyright Working Group has developed an online unit in iLearn called ‘Copyright for Educators’ that addresses these concerns.

There are many areas whereby the University maybe exposed with regards to Copyright law. These are:

• The University’s ownership of teaching materials versus Australian Government grants requiring all teaching resources to have Creative Commons licences;

• The relationship between disability/accessibility legislation and copyright legislation;

21

• The relationship between University’s copyright licences and Partner providers (SIBT);

• Learning and teaching materials not in eReserve thus will not automatically be included in iShare upgrade;

• Possible timing conflict with iShare rollout and the impending CAL audit in 2014.

PrivacyandconfidentialityThe University must comply with the Privacy and Personal Information and Protection Act 1998 (NSW) (PPIPA). Broadly it has obligations with respect to how it collects holds, uses, stores and discloses personal information. The project team worked with Julia Hall, University Solicitor, to ensure compliance within the new LMS with the introduction of the following:

• iLearn a pop-up window for the student to accept terms and conditions before they use the iLearn site;

• A refined Confidentiality and Privacy statement live at the beginning of Session 2, 2012;

• Privacy settings in iLearn were reformatted to comply with PPIPA;

• Contractors involved with iLearn were required to sign a privacy and confidentiality deed poll.

This work will be ongoing with the introduction of Learning Analytics and the continual expansion of ‘boundaries’ within online learning technologies.

PublicCommentandSocialMediaThere is a need for a policy that provides a framework to address appropriate student and staff behaviour when using social media (such as facebook, twitter, blogs) and other web enabled applications where that use has a connection with the University. This is currently not covered by any one policy at Macquarie University. There is a draft policy document; however, it is on hold at present awaiting a University strategy on how to progress the ‘social media’ issues within the University.

IssuesandFutureDirectionsiShare: The main issue and potentially the biggest barrier to maximising return on investment with regards to the technical capacity of the content management system (iShare) is the lack of clarity around post-implementation funding models for future development and enhancement. iShare is a system that will ensure the University is compliant with Copyright law. It is essential that the funding models for core learning and teaching systems are clarified for sustainability and consolidation to be fully realised.

Communication with Stakeholders: The main issue here is very limited understanding of University policy related to online learning amongst University staff. This includes how University policy can be implemented and the procedures for doing so. There are also some cases where there is no University policy to protect staff and students such as the use of social media. The CopyrightforEducators online unit may address the limited understanding of copyright basics but the University is exposed in other policy areas as outlined above Once the policy gaps are filled a communication campaign can begin. The appropriate teams will need to communicate changes at the faculty level to ensure that staff members understand the requirements and know the procedures to follow if University policy is breeched. At the same time a student communication campaign will run through appropriate channels.

3.4 Evaluation and Reporting Throughout the project a range of strategies have been in place to monitor progress towards the desired outcomes of buildingcapabilityandtransforminglearningandteaching.

• Prior to the transition to iLearn regular reporting of LMS analytics to MACALT meeting had been established. These have continued and have provided a consistent measure of the increasing use being made of iLearn (Moodle and Echo360) and associated support services. These are available on the MACALT wiki.

• At the faculty-level, data-bases of unit development were generated to monitor the transition to iLearn and

22

adherence to minimum standards.

• A range of formal and informal strategies have also been employed to provide feedback from staff and students using the system and services. Included were Surveys of user experiences and expectations, feedback from workshops and training sessions, reviews of helpdesk enquiries, and informal feedback through user networks.

• Regular reports were delivered to project teams on a regular basis e.g. the iLearn Portfolio on a weekly basis; iLearn Implementation Team on a fortnightly basis; and iLearn Steering Committee on a monthly basis.

3.5 Issues and Future Directions Evaluation: As we move beyond implementation phase it will be important to maintain some or all of the data-bases, evaluation and reporting strategies that were put in place in order to continue to monitor the effectiveness and relevance of iLearn and associated services and to inform future directions. The LTC, Informatics and other service providers, as well as Departments and Faculties all have different information needs and requirements therefore it will be necessary to make decisions about what information is needed, by whom and for what purposes.

Analytics: A vast amount of systems-generated information is automatically collected from iLearn e.g. (access, time on task, tools usage) which can be profiled by users, units, programs, Departments, Faculties. iLearn data can also be combined with datasets generated through informatics/analytics to provide a rich data set that could be used for a range of purposes including institutional reporting and planning, enhancing the student experiences and outcomes and informing curriculum redesign. Questions remain about which information should be extracted and for what purposes? Given that the extraction, meaningful interpretation and ethical use of data requires specialist skills and expertise, then a planned and strategically aligned approach to analytics is warranted.

23

4. ENHANCEMENT OF TEACHING

The transition to iLearn provided the opportunity for staff to review their current approaches to learning and teaching and modernise their teaching by making more effective use of the online tools and functions available. A clear focus of the change management stream was to, not only introduce staff to the new system and train them in its use but, support them in reconceptualising their approaches to teaching.

The staff survey was the primary means of gauging satisfaction with the tools and functions in iLearn and the changes in teaching that have taken place throughout the first year.

In Session 1, a total of 242 responses were received which represented 13% of staff using iLearn. In Session 2, 127 responses were received representing 10.5% of the sample. The breakdown by Faculty is given in Table 4.

In Session 1, 74.4% of participants had no prior experience with Moodle and in Session 2 only 18% had no prior experience, which indicates a transformation at a basic level.

Table 4: Staff Survey – faculty participation

Session 1% of 242 respondents

Session 2% of 127 respondents

Arts 37.7% 28.3%

Science 21.3% 26.0%

Human Science 21.3% 24.4%

Business & Economics 19.7% 21.3%

4.1 Satisfaction and Effectiveness iLearn offers a range of functions for facilitating learning and teaching. In order to ascertain levels of satisfaction and effectiveness, the functions were grouped into three categories:

• Functions for helping students to organise their study - announcements, calendar, RSS feeds, assignment submission, gradebook;

• Functions for engagement with unit content which include links to web sites and eReserve, Echo360, videos, lessons module, book module, glossary;

• Functions for engaging with learning activities which include the dialogue tool, discussion forums, blog, chat, quizzes, wiki, database, workshops, twitter feeds.

Participants were asked to rate their satisfaction with ilearn for supporting teaching and learning in each of these categories as well as for supporting interaction and communication with students.

A five point likert scale ranging from very satisfied to very dissatisfied was used. As shown in Figure 9 there was a distinct increase in satisfaction levels across the two sessions: iLearn as an online support for unit content > 11.5%; iLearn as a support for organising their teaching > 12.3%; iLearn supports their students’ engagement with learning activities > 10.6%; iLearn support interaction and communication with their students > 4.8%.

24

Figure 9. Staff satisfaction with iLearn for supporting learning and teaching

At the individual function level, Figure 10 shows the percentage of staff who agreed or strongly agreed that the functions were effective. The top five iLearn functions were: Announcements, Links to readings and/or external sites, Discussion Forums, Online assignment submission and Echo360. Although the tools were consistent over both Sessions the ranking was slightly different.

Figure 10. The effectiveness of iLearn tools - staff perspectives

25

Overall there has been a positive shift in perceptions of effectiveness, although there are two notable exceptions with popular functions namely online assignment submission and Echo 360. An analysis of comments from the open-ended questions revealed that technical issues were one of the key factors influencing effectiveness. In general a range of technical issues were highlighted by staff members including: slowness of the systems; the unreliable network connects (including wireless) across campus; incompatibilities with browsers, iPads and iPhones; remote access issues; ‘randomess’ such as disappearing documents and editing functions not working.

There were also technical issues that related to a particular function in iLearn, most were connected to Echo360 and Quizzes, however the Dialogue module and Chat were also mentioned as being unworkable. Staff reported that Echo360 ‘did not record properly’, it was ‘continuously failing’, lecture recordings were ‘cut short’ and issues with the labelling of lecture recordings in the system, which was the cause of many student complaints. The Quiz function technical issues were: data loss, formatting problems, the quiz not updating during the editing process, further problems with data entry and set. The most common response was that iLearn was ‘painfully’ slow; and for time-poor staff members this was the cause of many frustrations. These issues were addressed before Session 2, thus while there were some issues with Echo360 and Quizzes these were minimal by comparison, which is reflected in the second survey as there were less than half the amount of complaints. Also many of the grievances were with the communication functions, in particular, the dialogue module, this was not because of system failure but the layout.

4.2 Transformation in Teaching There has been a growth in the use of iLearn over the two Sessions of implementation, reflecting the minimum standard for all Faculties for of all units having an online presence. Overall, 2426 units were initially harvested from Blackboard and there are now 3506 active units in iLearn.

An analysis, conducted by the iLearn team, of the use being made of the functions and tools in iLearn begins to provide a picture of the actual transformation in teaching that is taking place. Table 5 shows the percentage of individual units that are actively using the in-built tools and functions. It is apparent that there has been a relatively low uptake of functions other than, discussion forums, assignment submission, dialogue, quizzes and turn-it-in (iLearn Implementation Report, 2011-2012). The only tool with significant increased use over the two Sessions is turn-it-in which could be due in part to its endorsement and promotion in several Faculties

Table 5 Percentage of units making use of iLearn tools/functions

iLearn Functions Session 1 % of units Session 2 % of units

Forum 99 100

Dialogue 64 42

Turn-it-in 27 33

Assignment 26 23Quizz 21 20

Glossary 8 9

Chat 5 5

Group Selection 5 5

Database 4 2

Wiki 4 3

OUBlog 3 3

Voice tools 3 3

26

iLearn Functions Session 1 % of units Session 2 % of units

Choice (Poll) 2 3

Lesson 1 1

Workshop 1 1

LAMS 0 0

Survey 0 0

Transformation in learning and teaching takes time, with staff firstly having to gain the requisite skills and confidence in using the software before they can explore pedagogical advantage. Findings from the surveys reveal that the foundations for change have been laid but there is still work to be done if transformational change is to be achieved.

Figure 11 shows the percentage of participants who agreed or strongly agreed with statements regarding skills and confidence.

Table 5 (...continued) Percentage of units making use of iLearn tools/functions

Figure 11. Skills and confidence levels of staff

• There has been an increase in confidence in using iLearn for teaching over the two Sessions with 78% of participants being confident at the end of Session 2.

• Although confidence has risen throughout the year in the use of Echo360 and iTeach is was still relatively low at the end of Session 2 for iTeach. This could, in part, be a reflection of the intermittent technical problems experienced during Session 1. Details of the technical issues experienced by staff members are outline in the ‘Outlined issues and enhancements’ section.

There has been a slight decrease in participants’ assessment of their skills in using iLearn from Session 1 to Session 2. On the surface this is at odds with the increase in confidence levels. However it could be due to a growing awareness of the potential of the new system and that further training is needed to move beyond the basics of iLearn

27

In regard to changing pedagogical practice, although staff members are not making use of the more specialised functions in iLearn (see Table 5), findings from the survey indicate that the move to iLearn is encouraging reflection on teaching practice. By the end of Session 2, 47% of participants felt that the move to iLearn had helped them to reconsider their teaching. Importantly 88% of participant have indicated that they would like to make use of the functions in iLearn to enhance their teaching in the future. Arising from an analysis of open-ended comments, the functions identified for further use were Quizzes (34 participants), Wikis (18 participants) and Online Assignment Submission (10 participants), Gradebook (6 participants).

4.3 Issues and Future DirectionsRealising the pedagogical benefits of iLearn is closely tied to the capability of the software to support learning and teaching processes. This was evident in the analysis of open-ended comments where the identification of tools for future use was accompanied with requests for technical enhancements for example:

• Quizzes - enhancements to make them more intuitive as well as an overhaul to speed up the marking;

• Wikis – further enhancements for stability and to the ‘look and feel’;

• Online Assignment Submission - group assignment submission with group-level rubric evaluation;

• Gradebook - Turnitin results to be linked to Gradebook or ability to use common user IDs for importing; and

• Dialogue tool – enabling messages to be sent to more than one student; being able to forward messages and copy students or staff; enabling students to dialogue each other; increasing efficiency of the system in relation to closing dialogue, ability to order dialogues in different ways such as by subject, date or time.

In Appendix 4, are the corresponding tables with more detailed information including staff comments and responses to the scaled questions (survey questions 8-10) which show the broad spectrum of responses and calls for future enhancements.

Transformational change is a slow process, and while the confidence levels and staff capability is increasing, the progress can largely be seen as laying the foundations for the future. Akerlind (2007) identified five different approaches to developing as a university teacher, each stage building on the achievement of the previous. In the iLearn environment, these translate into:

1. improving content knowledge i.e. knowledge and skills of the system

2. building experience in how to teach

3. building a repertoire of teaching strategies

4. determining which strategies work best

5. understanding what works and why to effectively facilitate learning

Akerlind (2007) maintains that it is only when academics have reached the third stage that will they be interested in further development. The findings indicate that with 88% of participants seeking new ways of using iLearn to enhance their teaching, there is progression towards the third stage. However, staff members have indicated that they need to have the time to first learn how to use it and then implement it into their unit. Lack of time was highlighted as the number one issue in preventing staff from exploring all the iLearn functions. Considering that 74.4% of staff surveyed in Session 1, had no experience of iLearn these preliminary results suggest that transformation is progressing at a steady pace.

Integrating new processes into academic practice is dependent on maintaining the positive relationships between the Faculties, offices and LTC and reinforcing communication and feedback between the various stakeholder groups. In preparing for the end of the iLearn implementation, the change stream designed good practice statements, quality assurance processes and highlighted the quality enhancement support available, using the principles of Technologies in Learning and Teaching Policy. Many of these are still to be embedded into academic practice in Faculties.

28

5. ENHANCEMENT OF LEARNING

In Session 1, a total of 712 responses were received which represented 9.2% of the selected sample. In Session 2, 482 responses were received representing 6.2% of the sample. The breakdown by Faculty is given in Table 6.

Table 6. Student survey- participation by faculty and level

Session 1 % of 712 respondents Session 2 % of 482 respondents

Arts 32.8% 31.4%

Business & Ec 28.8% 23.4%

Human Science 28.8% 26.6%

Science 14.6% 18.4%

Undergraduate 79.3% 71%

Postgraduate 20.7% 29%

Internal 75.8% 68%

External 22.1% 26.8%

1st Year 36.7% 28.5%

2nd & beyond 63.3% 71.5%

5.1 Satisfaction and EffectivenessThe SEET Survey in 2010 clearly indicated students’ expectations for technologies to be part of their learning experience. They were strategic about their use of technologies, valuing those that had a clear purpose in supporting their learning, provided flexibility and efficiency in access and supported communication and interaction. The Learning Management System (Blackboard-WebCT at the time) was popular with students strongly indicating they would like more use made of the LMS in the future.

With the move to iLearn, findings from the student survey indicate that satisfaction with the LMS has risen. A direct comparison between the 2010 SEET Survey and the iLearn surveys revealed that global student satisfaction levels have increased (Table 7). In the SEET Survey 55% of the 1104 students surveyed indicated they were satisfied with the technology used by Macquarie University and 60% agreed that the way the technology was used enhanced their learning. After the first year of implementation, satisfaction levels have climbed to 71% and 69% respectively.

Table 7 Global satisfaction levels between 2012 and 2012.

% agreement with: 2010 SEET

SurveyN= 1104

Session 1 SurveyN = 712

Session 2 SurveyN = 482

The way / iLearn is used enhances my learning experience

60% 60% 71%

I would recommend MQ’s use of technology /iLearn as an example of good practice

46% 61% 69%

29

In order to gain a better understanding of the global satisfaction levels, participants were asked to indicate their satisfaction with iLearn in regard to the way it has been used to:

• enable access to unit information and content;

• provide support for organising study;

• support engagement in learning activities;

• support interaction and communication with staff.

The percentage of participants who were satisfied/very satisfied with each of these categories is shown in Figure 12. Satisfaction has increased in all categories ranging from 63% to 77% in Session 1 to between 69% to 84% in Session 2. Corresponding dissatisfaction levels were running at between 11% and 16% in Session 1, falling to between 6% and 12% in Session 2.

Figure 12. The percentage of participants satisfied with the way iLearn has been used

Similar to the Staff survey, students were asked to rank the usefulness of specific iLearn functions in each of the three categories of organising study, accessing unit content and engaging in learning activities. Figure 13 shows a comparison between Session 1 and Session 2 of the percentage of students that agreed /strongly agreed that the tools were useful.

30

Figure 13: The percentage of students in Sessions 1 and 2 who found iLearn tools to be useful

The ‘top five’ iLearn functions in terms of usefulness in Session 1 mirrored those identified by staff - Echo360, Links to readings and external sites, Online assignment submission, Discussion forums, and Announcements. These same technologies were popular in Session 2 however Unit Guides were introduced for the first time and assumed top ranking. The low levels of perceived usefulness of some of the functions could simply reflect lack of awareness rising from the low level of use being made of them by staff. Indeed, in the open ended responses some students indicated that they did not know iLearn had so many functions until they completed the Student iLearn Experience Survey.

31

Further analysis of the usefulness of the tools and general satisfaction levels was undertaken in relation to enrolment mode, program level and year of study.

EnrolmentmodeThere were few significant differences according to whether students were enrolled internally or externally. The differences that exist are accounted for by external-only students rating the usefulness of links to readings, e-Reserve, lecture recordings and the Dialogue tool more highly than the internal-only and mixed groups, possibly due to external students’ greater reliance on these tools. A similar pattern of results was found in relation to iLearn support for communication with teaching staff, probably reflecting externals students’ use of iLearn as their main means of contact with staff.

ProgramlevelIn contrast to the Session 1 results, there are several significant differences between undergraduate and postgraduate students in relation to the usefulness of iLearn tools with announcements, gradebook, lecture recordings, videos and quizzes rated significantly more useful by undergraduate students than by postgraduate students. On the other hand, postgraduate students rated links to readings and external websites higher than did undergraduate students.

While undergraduate students rated their general satisfaction consistently higher on average than postgraduate students, the only significant differences were on iLearn support for interaction and communication with fellow students and both the “global” satisfaction items (iLearn enhances learning experience and recommendation of MQ’s use of iLearn).

YearofstudyA comparison of first year (students who had not experienced Blackboard) versus later years (students who almost certainly have used Blackboard) was undertaken. Since we can assume that most students in their first year commenced in Session 1, the main analysis of interest is to compare first year students in Session 2 with those in Session 1, to gauge the effect of familiarity with iLearn untainted by previous use of Blackboard.

Similarly, a comparison of Session 1 and Session 2 results for second and later year students may help in understanding the extent to which the relative dissatisfaction of second and later year students in the Session 1 results was due to the transition from a familiar LMS (Blackboard) to a new LMS (iLearn), although of course the familiarity/transition effect is likely to last into a second Session of iLearn use.

Comparisons of Session 1 and Session 2 results revealed:

• There is much less difference between first and later year students in the Session 2 survey (only the usefulness of Announcements and the Calendar are rated significantly differently, with first years giving the higher score in each case).

• There are many significant differences between Session 1 and Session 2 ratings by second and later year students, both in terms of the usefulness and of the various iLearn tools and their satisfaction with various aspects of iLearn, with the Session 2 ratings being higher in all cases.

• For the second and later year group, the differences between Session 1 and Session 2 on all of the “satisfaction” items including the global items (iLearn enhances learning experience and would recommend MQ’s use of iLearn) are highlysignificant (p < 0.005); furthermore, there are no significant differences between first year and later year students in Session 2 on any of these items.

Together, this tells us that it is highly likely that resentment at having to change from Blackboard to iLearn was a major source of dissatisfaction for the second and later year students in the Session 1 survey. The small but statistically insignificant general increase in first year ratings between Session 1 and Session 2 may indicate that bothstaffandstudents’increasing familiarity and comfort with iLearn have a positive impact on scores overall.

32

5.2 Transformation in LearningOverall, the aspects of iLearn that students found the most valuable for supporting their learning reflect the research findings from the 2010 SEET Survey and are those that:

• provide ease of access to content such as Echo360, links to readings and external sites as well as eReserve and the organisation of content such as the weekly set up of iLearn. Echo360 was also ranked the highest in the iLearn functions and there were many comments with at least 126 references outlining the value of online access to lecture content and the ability to revise lecture content at will;

• offer efficiency in administrative processes - online assignment submission;

• facilitate communication and interaction e.g. discussion forums and announcements.

In terms of future enhancements to support their learning, the most common tools mentioned were Echo360, Discussion Forums, Dialogue, Online Assignment Submission, Gradebook, Calendar, Quizzes Wikis, and UNITSbut most student comments were tempered by technical or pedagogical conditions for use.

Echo 360: There was the expressed view for Echo360 to be used across all units but at the same time it should be used consistently. Issues of reliability and stability were raised for example: “Echoworksspasmodically; itneedstobemorereliable”;“EchoRecordingsneedtobemoreuserfriendly”;“Echo360needstobemoreaccessible,Ifindthatmoreoftenthannotitisnotworkingwhichmakesitdifficulttolistento”.

Discussion Forums: Student would like to see more use of discussion forums but at the same time this was qualified with statements like: “Iwouldlovetoseemorediscussionboardsineveryunit,especiallymannedbylecturers/ortutorsforfeedbackandanswers.Somedo,andthisisextremelyhelpfulandtimely,butsomedon’t”and“Thediscussionforumsarevaluabletoolforexternalstudentsanditwouldbegreatifthesewerefurtherutilisedbystudents”.While many students value discussion forums there were calls to make them more user-friendly and more organised. There were also concerns that discussion forums are not always monitored closely enough nor were their consistency across units. Some students said that there is also a lack of fellow student participation in some cases.

Dialogues: Students, the same as staff, would like to see improvements in all iLearn communication, in particular, the dialogue module, in most cases they would like it to operate more like email. It is seen as a valuable function:“Thedialoguetoolsareespeciallyusefulforcontactingtutorswithquestionsnotansweredintheforums”but at the same time, students would like to see it improved upon, for example,“Ihaveoftenfoundthedialoguesectioneitherconfusingoritdoesn’tsend.Thiswouldbegoodifitmadesoweknewforsureitsent”.

Online Assignment Submission: This is prized by students but there were several concerns regarding the stability of the system otherwise most students claimed that they would like to use more of it, as well as turn-it-in. Some indicative comments included: “Wouldlikemoreonlinesubmissionsofassignments.Itisactuallymucheasier to lodge thisway insteadofby submittingahardcopy,plus Ihave theknowledgeof receivinganemail toconfirmthesubmission”;“Onlineassignmentsubmission-useofthiswithclearinstructionsofhowtouseitandalsoadirectpersonofcontactwhenerrorsoccur(e.g.systemcrashesisamajorissue)”and “Iwouldabsolutelylovetohaveonlineassignmentsubmissionbutnoneofthepsychologyunitshaveit”.

Gradebook: Students want convenors and tutors to use Gradebook and those that do use it, students would like them to update it more frequently. There was also one comment that the layout of Gradebook is confusing and some students would also like to be sent an alert when new marks are available.

Calendar: Students would like to see the calendar improved so that it is aligned with the unit guide as well as exam and assignment due dates. This request was also made in Session 1 survey. Some indicative comments included: “I’dwouldliketousecalendarmore,butit’stosmallandmostoftheeventsarerepresentedbythesamesymbol,soit’sveryhardtotellwhichiswhich.Ithinkifyoucanmakeitmorecolourfulormorefun,that’swillbegreat”;“I’dloveallunitstousethecalendareffectivelyandefficientlysoyoucanseeallyourassignmentsintheonecalendar,wouldbehandy”and“Acalendarwithallsubjectsandactivitieslistedthatdoesnotneedtorefreshtheentirepagewhenchangingmonths”.

33

Quizzes: Students valued short and regular quizzes as a way of measuring their progression and attaining regular feedback. Some indicative comments included:“Ireallyliketheonlinequiz’s,Ifindthattheyareagreatwayofcompletingquiz’sasyoucandothemathomeandnotfeelpressuredinclass”;“I’mhappywiththelevelofinvolvementofvariousfeatures- itwouldbehelpfultoseesomemorenon-assessablequizzestoseehowyouaregoingwiththecontent”and“OnlinequizzesarereallyhelpfulintestingwhetherIhaveunderstoodthetopic”.

Wikis: Student responses listed wikis in the iLearn functions that they would like to use in the future. There was also the suggestion of wiki alerts when content is added and ‘student controlled’ wiki spaces: “Itwouldbegoodtohavestudentcontrolledwikispresentinilearncourses”;“I’veputNotApplicableagainquiteafewoftheseitemsbecauseI’mnotawareofthem(that’snottosaytheydon’texistinthedeliveryofmycourse,butjustthatIhaven’tusedornoticedthem).I’dbeinterestedtoknowhowthewikiwouldwork”and “TheWikicouldgivealsoalertswhenitisupdated”. It is difficult to compare the two Sessions considering that the question changed from scaled to open-ended but the information in Session 2 provides a richer understanding of what students would like to use in the future and the reasons why.

UNITS: In general, students are more positive than staff with regard to the Unit Guide and UNITS. Some indicative comments included: “TheUnitGuideisgreat”;“Itisextremelyhelpfulfororganisingandco-ordinatingmystudy”;“Theunitguideworkswellandishelpful”;“Theunitguidesareusuallyveryclearandlinkedclearlytocontent”;“Verygoodusability.Overallsatisfied”;“Prettymuchperfect”.Students made several suggestions with regards to layout, for example, some students said: “Shorterunitguideswithclearer lay-outofessential information: suchasassignmentdatesandweighting,contactinformation”;“Standardisedtemplateshouldbeencouragedacrossalllearningareastomake foreasieraccess to informationrequiredbystudents”; “Bigger font forweeklynotes”; “Moresubject-specific informationratherthangeneric information,e.g.onplagiarism.Moreprecisedateswithregardstoassessmentduedates etc”.Student would also like timely availability of Unit Guides as well as hyperlinks to make the Unit Guides more interactive. Students spoke about graduate capabilities, for instance, one student said: “dowe reallyneedall thewaffleaboutgraduate capabilities?” One major issue that students outlined is inconsistency in the information that is provided such as old Unit Guides with incorrect dates for assessment and the availability of two unit guides.

There were also student requests for:

• a printer friendly version;

• linking of Unit Guides with the iLearn Calendar so that dates of exams and assessments can be in sync;

• easier access from iLearn;

• a UNITS iPhone application (onlymentionedtwice).

Standardisation: Overall, students would like to see standardisation. Their comments indicated that many of these functions were not used by convenors; some of their iLearn units were much better than others; that they had experienced difficulties in finding various functions in some iLearn units because the layout was different in each; thus many students requested the coordination of content and where it is placed in all iLearn units. This concern should be addressed through the work of the iLearn Style Guide Working Group, which is based on the visual representation of all iLearn units.

34

5.3 Issues and Future DirectionsIn Session 1 survey, the iLearn functions that the students found the most valuable were also the functions that they would like to see improved. This correlation could be due to the students’ high exposure to certain iLearn functions, which then became the functions that are seen as adding value to their learning. Equally, because they are exposed to them over an extended period they can then see pathways for improvement.

Identified issuesThe identified issues from these data sources show:

• Students use of iLearn functions is reliant upon staff use thus if the more interactive functions are not used widely then there will continue to be a low level of agreement in how they contribute to everyday learning experiences. In 2012, the iLearn functions that provide a sense of ease or convenience are the ones that are relied upon. Transformation of online learning will be a slow process, with respect to all the iLearn functions being used to their full capacity in learning and teaching.

Students, and staff, indicated that there are layout issues within iLearn, for example, Dialogue Module needs to function like email, these ongoing enhancements, to all functions not just the Dialogue Module will require adequate resourcing.

Student satisfaction levels are slowly increasing with regards to learning systems support but so too are the expectations of that level of support.

The introduction of UNITS and online Unit Guides has produced several issues like inconsistency in the information that is provided such as old Unit Guides with incorrect dates for assessment and the availability of two unit guides. This is a quality assurance issue.

Network upgrade is needed, especially considering the fact that all units now have an online presence thus there will be more individuals access them 24/7.

35

6. CONCLUSON

The implementation of iLearn during 2012 is the beginning rather than the end of a project aimed at delivering transformation change. During this first year, the core technical and organisational infrastructure for online delivery has been established and by year’s end there were 3506 active online units on iLearn. This in itself is a major achievement, given that this has not only involved a change from Blackboard-CE6 to Moodle, it has also encompassed the integration of 13 other systems or applications. While the development of iLearn is still in progress the foundations have been laid to provide the agility and flexibility to stay abreast of change. The governance, management, policy, planning and evaluation structures and processes that have been in place for iLearn have been effective during the development and implementation phases. Whether the aspired agility and flexibility of the new infrastructure and capacity to support innovation in learning and teaching will be fully realised will, to some degree, depend on how these structures and processes are adapted and further developed to accommodate the post implementation ‘business as usual’ phase where the high levels of project funding are no longer available.

The issues and implications for post implementation structures and processes have been identified in the various sections throughout the report on Issues and Future Directions. The following recommendations developed by the Evaluation Working Group capture these issues at a broad level.

1. Further development of learning technologies.There needs to be a continued focus on the University’s learning and teaching infrastructure and further alignment work is needed on many of the technologies that form the iLearn platform such as the integrating UNITs and iTeach, resolution for video hosting and web conferencing, the new content management system iShare. For the future clear and transparent strategies and pathways are needed for:

• capturing the expectations and needs of students and staff;

• assessing requests for potential ‘add-ons’, such as web conferencing and a solution for videos;

• adequately resourcing the development and implementation of enhancements.

2. Development of the University’s leadership role within the Moodle and Open Source Communities.Further development of the University’s leadership role within the Moodle and Open Source communities of practice (COP) is required. A planned approach is needed which could include strategies for creating new networks, developing and maintaining current networks, hosting events like iShare Symposium in December 2012 and disseminating research findings.

3. Sustainable funding models for core learning and teaching systems. It is essential that the funding models for core learning and teaching systems are clarified, timely and planned with resources available within the regular funding cycle for routine maintenance and development as well as support for innovation within the university and the broader open-source community.

4. Future planning for the University: online learning strategy.In the absence of a university-wide IT and web strategy, iLearn has been seen as the default solution for communicating and facilitating various student and staff services which has raised a number of governance, management and resource implications. In view of this, it is timely for the university to develop a more comprehensive and transparent approach which aligns online learning with a University web strategy and a broader IT development plan.

5. Ongoing evaluation review and reporting of iLearn.Systematic evaluation and reporting on the performance and effectiveness of iLearn is essential for the continued development of systems and processes that enhance the student learning experience and outcomes. The various feedback and reporting strategies that have been developed through the iLearn development and implementation phases need to be consolidated into a comprehensive

36

evaluation plan which has a clear focus, systematic data collection, dissemination pathways and strategies for action and reporting on outcomes.

6. Ongoing development of University policy related to online learning.The iLearn Implementation highlighted a number of gaps in the current University policies, for example there are no policies for social media and there is ongoing work needed for policies relating to copyright and privacy. The policy landscape in relation to online learning requires constant monitoring, and this needs to be recognised in the form of defined roles, responsibilities and lines of communication.

Working closely alongside infrastructure development has been a major change management program to enhance the quality of teaching and learning – another of the guiding principles. The program aimed to support staff in the development of the skills and capabilities to use iLearn and reconceptualise their teaching. The former is well underway with 78% of staff surveyed in Session 2 confident when using iLearn in their teaching and 68% believing they have developed skills to effectively use iLearn, however, there is still some way to go. Reconceptualisation and transformation of teaching takes longer because of the cultural change involved, however there were signs at the end of Session 2 that this was taking place. Having confidence and skills provides a solid platform for change and by the end of the year, 88% of survey participants indicated they would like to utilise more of the tools in iLearn in the future.

Another of the guiding principles was the enhancement of the student learning experience. There is a positive move in this direction. From the student perspective, their overall satisfaction levels with iLearn are markedly higher when compared to those from the SEET survey in 2010, and there was a noticeable increase from Session 1 to Session 2. Of particular interest has been the finding that during the first Session, the satisfaction levels of continuing students making the transition from Blackboard to Moodle were lower than for first year students. This initial reaction was short lived, and by Session 2 the satisfaction levels of continuing students had risen to that of their first year counterparts.

Student satisfaction is related to the interrelationship between the technology and the use being made of it by staff. One of the factors that has led to this successful combination has been the faculty-based teams who have led and facilitated staff engagement with the change process. Their role in the change management process has been outlined along with issues and implications for future practice, and this leads to a recommendation for:

7. Continued support for transforming learning and teaching. The iLearn faculty-based teams were very successful and these structures and relationships need to be maintained in the future. There needs to be ongoing commitment to the professional development of staff including sessional staff, to continue both technical and pedagogical capability and confidence.

REFERENCES

Akerlind, G.. (2007). Constraints on academics’ potential fro developing as a teacher. StudiesinHigherEducation,32:1,pp. 21-37.

Blueprint for the Future (2011). Macquarie University. Available at: http://www.mq.edu.au/learning-and-teaching/activities/research/blueprint.html.

Chesterton, P. and Cummings, R. (2011) ALTCProject Evaluation Resource. Available at: http://www.olt.gov.au/evaluation.

Gosper, M. & Jacenyik-Trawöger, C (2010). ‘Student Experiences and Expectations of Technology’ North Ryde, Macquarie University: 1-33.

Gosper, M., Malfroy, J., & McKenzie, J. (In Press). Students’ experiences and expectations of technologies: an Australian study designed to inform planning and development decisions.Australian Journalof EducationalTechnology

37

APPENDICES

Appendix 1. Students and staff Surveys for Session 2

38

39

40

41

42

43

44

45

46

47

Appendix 2. iLearn Platform in 2012

Technology or System Function Description

Echo360

Echo360 is used for all venue based recordings. The Echo360 System can be used for recording lecture series, one-off lectures, special events, seminars, and conferences in the supported venues

iShare (Equella)

iShare (Equella) is a Learning Content Management System (LCMS) that is used for the creation, storage and management of resources that are usually delivered in a learning management system.

Turnitin

Turnitin is a text-matching software that compares electronically submitted papers to databases of academic publications as well as other papers that have been submitted into the system. Turnitin assignments can be set up through iLearn using Moodle Direct Integration. This enables students to login to iLearn to submit assignments directly into Turnitin.

iTeach

iTeach is Macquarie’s online unit creation and enrolment system, allowing Faculties, Departments and unit conveners to manage the availability of their iLearn units, unit versions, student enrolment, and staff access.

UNITS

The UNITS system is the online repository for Macquarie University Unit Guides and curriculum mapping tool. This system enables academics to write, publish and view Unit Guides and map units against the University’s requirements for graduate capabilities. It is also designed to provide publically available information about the coursework related to any unit at Macquarie University.

OneID LDAPMacquarie University OneID Lightweight Directory Access Protocol (LDAP) is part of the iLearn authentication process.

Australian Access Federation (AAF)

AAF is another authentication process that provides a framework and support infrastructure to facilitate trusted electronic communications and collaboration within and between universities and research institutions in Australia.

DataswitchDataswitch is the Macquarie University product for integration and asynchronous messaging between applications.

Learning Activity Management System (LAMS)

LAMS is an authoring and delivery system designed to support collaborative, ‘activity-driven’ learning.

Wimba Voice ToolsWimba Voice Tools assists educators to help students improve their language, linguistic or translating skills and/or develop a sense of online community.

Help.iLearn (FAQ engine)

Help.iLearn is a contextualised help tool thus wherever a person is in iLearn they can click on help and the relevant information will be presented. This information was generated through frequently asked questions at the iLearn help desk.

EvasysIs the evaluation system used by Teaching Evaluation for Development Service (TEDS).

Crocodoc APICrocodoc API allows students to annotate and comment on primary sources in their original forms.

48

Appendix 3. Minimum Standards for Faculties in 2012

The iLearn Implementation included the compliance of online ‘minimum standards’ in each faculty and control entity for the first time at Macquarie University. At the beginning of 2012 were as follows:

Faculty of ArtsLink to UNITS (UG units) or Unit guide supplied as file (PDF) for PG unitsTeaching Staff details provided in block (right column)Assessment details provided that are not in Unit GuideCommunication Channels defined (how to contact convenor, Forum use, etc.)ECHO block if applicable (or deleted if not required)Any content/labels etc that are not required to be deleted or hidden

Faculty of Business and EconomicsContact details of at least one of the unit’s teaching staffThe unit guide – if the 2012 version has not yet been finalised, you can upload the most recent version and update it when the new one is available.The minimum requirements should ideally be met by the end of 2011 for all Session 1 2012 units.

Faculty of Science Teaching Staff details (set up your convener details in the units you are convening)Communication Channels defined (set up channels of communication in your unit such as announcements, forums and dialogues)Attended iLearn Basics training or acquired the skills to be able to do the above tasksDesirable - Given consideration to your unit structure By 24 February 2012, you will also have: - Linked your Unit Guide Desirable - Linked Echo 360 (iLecture) and Turnitin

Faculty of Human SciencesTeaching Staff details (set up your convener details in the units you are convening)Communication Channels defined (set up channels of communication in your unit such as announcements, forums and dialogues)Attended iLearn Basics training or acquired the skills to be able to do the above tasksDesirable - Given consideration to your unit structure By 24 February 2012, you will also have: - Linked your Unit Guide Desirable - Linked Echo 360 (iLecture) and Turnitin

ASSAMAccessible unit content aligned with the University’s accessibility guidelines.

MGSMThe blended learning unit design templateThe built-in standard template communication features – The Lounge chat room, FAQ Forum and Student Website Bulletin Board The built-in MGSM branding of the unit template.

Open University Australia (OUA)OUA template- each unit must follow the OUA template. The template involves the unit guide, assessment summary and communications information. Each unit needs to have accessible content in any of the acceptable formats: html or pdf files.Each unit must incorporate any of the communications online tools available: forums, blogs, wikis, dialogue, web conferencing, and so on.Each unit must have an assessment method: online submission, COE submission, Turn it In or any other form of digital submission.DesirableEach unit SHOULD have an assessment report tool for example the gradebook but as the final grades are delivered via e-Student I have decided not to enforce the use of any tool.

49

Appendix 4. Staff comments and responses to the scaled questions (survey questions 8-11)

iLearn functions Summary of Staff responses to Q9 - “The iLearn platform was effective in meeting my needs for facilitating student engagement with the unit content” and Q 11- What would enable you to use the tools more effectively in your teaching?

Linkstoreadingsand/orexternalwebsites

Overall, 82.4% of staff agreed or strongly agreed that the Links to readings and / or external sites facilitated student engagement with the unit content. There were 2 references to Links to readings and/ or external websites that were based on the organisation of the links inside iLearn, for example, one staff member said:“Fileobjects(orlinks)shouldbeinsertableinsidelabels.Itisannoyingtousemultiplelabelstosurroundfiles”.

Linkstoe-Reserve Overall, 63.6% of staff agreed or strongly agreed that the Links to e-Reserve facili-tated student engagement with the unit content. There was only 1 reference to eReserve, which related to the integration of eReserve and iLearn:“ShouldbebetterintegrationwitheReserve”.

Echo360 (lecture re-cordings)

Overall, 64.3% of staff agreed or strongly agreed that the Echo360 facilitated stu-dent engagement with the unit content. There were 2 references to Echo360, both comments related to training on how to give students more information about the lectures in this system. One staff member said: “Iwouldliketheabilitytobeabletoname,writealecturesummaryandattachlectureslidestotherecordingsitebeforetherecording.Inthisway,studentsarepreparedforwhatiscomingup,andtheyaccessonesitefortheirlecturematerials,ratherthanECHOplusalecturenotefolder.ManyofmystudentshavetoldmethattheonlyreasontheyconsultiLearnistogettheirlecturenotes”.

Videos(YouTubeClipsetc)

Overall, 65.5% of staff agreed or strongly agreed that the Videos facilitated student engagement with the unit content. There was 1 reference to Videos (YouTube Clips etc) but no further information on how it could be used more effectively in the staff members’ teaching.

Bookmodule Overall, 53.1% of staff agreed or strongly agreed that the Book module facilitated student engagement with the unit content. There were no references in question 11 to the Book module.

Glossary Overall, 50% of staff agreed or strongly agreed that the Glossary facilitated student engagement with the unit content. There was 1 reference to the Glossary and how this function needs to expand its ‘import’ function:“Itriedtousetheglossary/databasestosetupthephotolibrariesIhavegatheredinthepast8yearsofdeliveryofmyunit.Whileitwasstraightforwardenoughtoallowstudentstoaddtheirphotos,therewasnowayoflettingmegroupuploadorimportallthephotosIhavepreviouslyprovidedforthisunit.IfoundthisbothdisappointingandamajorflawinthestructureanddesignofiLearngiventheemphasisonsocialmediaandsimilarstyleoftoolswhereimagesareakeycomponent”.

50

iLearn functions Summary of Staff responses to Q9 - “The iLearn platform was effective in meeting my needs for facilitating student engagement with the unit content” and Q 11- What would enable you to use the tools more effectively in your teaching?

Linkstoreadingsand/orexternalwebsites

Overall, 82.4% of staff agreed or strongly agreed that the Links to readings and / or external sites facilitated student engagement with the unit content. There were 2 references to Links to readings and/ or external websites that were based on the organisation of the links inside iLearn, for example, one staff member said: “Fileobjects(orlinks)shouldbeinsertableinsidelabels.Itisannoyingtousemultiplelabelstosurroundfiles”.

Linkstoe-Reserve Overall, 63.6% of staff agreed or strongly agreed that the Links to e-Reserve facilitated student engagement with the unit content. There was only 1 reference to eReserve, which related to the integration of eReserve and iLearn: “ShouldbebetterintegrationwitheReserve”.

Echo360 (lecturerecordings)

Overall, 64.3% of staff agreed or strongly agreed that the Echo360 facilitated student engagement with the unit content. There were 2 references to Echo360, both comments related to training on how to give students more information about the lectures in this system. One staff member said: “Iwouldliketheabilitytobeabletoname,writealecturesummaryandattachlectureslidestotherecordingsitebeforetherecording.Inthisway,studentsarepreparedforwhatiscomingup,andtheyaccessonesitefortheirlecturematerials,ratherthanECHOplusalecturenotefolder.ManyofmystudentshavetoldmethattheonlyreasontheyconsultiLearnistogettheirlecturenotes”.

Videos(YouTubeClipsetc)

Overall, 65.5% of staff agreed or strongly agreed that the Videos facilitated student engagement with the unit content. There was 1 reference to Videos (YouTube Clips etc) but no further information on how it could be used more effectively in the staff members’ teaching.

Bookmodule Overall, 53.1% of staff agreed or strongly agreed that the Book module facilitated student engagement with the unit content. There were no references in question 11 to the Book module.

Glossary Overall, 50% of staff agreed or strongly agreed that the Glossary facilitated student engagement with the unit content. There was 1 reference to the Glossary and how this function needs to expand its ‘import’ function: “Itriedtousetheglossary/databasestosetupthephotolibrariesIhavegatheredinthepast8yearsofdeliveryofmyunit.Whileitwasstraightforwardenoughtoallowstudentstoaddtheirphotos,therewasnowayoflettingmegroupuploadorimportallthephotosIhavepreviouslyprovidedforthisunit.IfoundthisbothdisappointingandamajorflawinthestructureanddesignofiLearngiventheemphasisonsocialmediaandsimilarstyleoftoolswhereimagesareakeycomponent”.

51

iLearn functions Summary of Staff responses to Q10 – “The Learn platform was effective in meeting my needs for engaging my students in learning activities”and Q11- What would enable you to use the tools more effectively in your teaching?

Dialoguetool Overall, 49.8% of staff agreed or strongly agreed that the Dialogue tool engaged their students in learning activities. There were 22 references to the Dialogue tool, most comments called for changes to allow the same message to be sent to more than one student and that it should operate more like an email system. Some indicative comments included:“Thedialoguetoolshouldallowyoutosendthesameemailtomorethanonestudent-inoneunitstudentshaveagroupassignmentandthedialoguetoolwasuselessfordiscus-sionswithmeregardingthis”.“TheDialoguetoolisnotuserfriendlyatall.Itshouldoperatejustlikenormalemail”.“Unsatisfactorydialogue-thereneedstobeabetterwayofidentifyingnewdialoguesratherthanhavingtoscrolldownawholelisttolookfortheunreadone-thisisridiculousinalargeunit”.All of the comments indicated a shortage of time thus multitasking is required when communicating with students, especially those that ask the same question. Also, time pressures experienced by staff members they need a communication tool that orders, archives and highlights new posts (like email systems) in order to organise their conversations with students.

Discussionforums Overall, 66.1% of staff agreed or strongly agreed that the Discussion forums engaged their students in learning activities. There were 25 references to Discussion forums. Some indicative comments included:“Moreflexibleoptionsfororganizingpostsinthreads.Thereshouldbeatleastoneviewthatpreservestheorderoftheoriginalthreadssothatonecaneasilyfindoldpostsifdesired.Itwouldalsobeusefultobeabletoselectindividualstudentsandidentifyallthepoststhattheyhadmadeacrossalldiscussionforums”.“Abilitytocloseweeklyforumsandalistofthenumberofstudentposts”.“Theforumsshouldworkwithgroups.Mostofthemdon’t”.The focus of these comments was on the organisation of the discussion threads. Discussion forums had the highest amount of comments by staff members thus these enhancements and/or further training are viewed as significant in allowing staff to use this iLearn function more effectively in their teaching.

Blog Overall, 34.5% of staff agreed or strongly agreed that the Blog engaged their students in learning activities. There were 2 references to the Blog, one staff member simple said that it needed to be improved, while the other comments on transformation of learning and teaching from the perspective of staff and students: “Ihaveusedblogsinthepast,studentsdidn’tlikehavingtodoitandadditionaleffortdidnotseemworthwhileforanyparty”.

Chat Overall, 29.5% of staff agreed or strongly agreed that the Chat engaged their students in learning activities. There were 2 references to Chat both relating to how staff members had used this iLearn function in their units however students did not make use of it.

52

iLearn functions Summary of Staff responses to Q10 – “The Learn platform was effective in meeting my needs for engaging my students in learning activities”and Q11- What would enable you to use the tools more effectively in your teaching?

Quizzes Overall, 56.2% of staff agreed or strongly agreed that the Quizzes engaged their students in learning activities. There were 21 references to Quizzes all of the staff comments were calling for further enhancements. Some indicative comments included:“Markingthequizzesispainfullyslow.Toomanyscreenshavetobecycledthroughwhenoverridingamarkoraddingacomment,anditappearstobeimpossibletomarkquizzesbygroupingallanswerstothesamequestionratherthanmarkingentirequizzesonestudentatatime”.“Quizzesneedatotaloverhaul.Thesystemisawful.Forastart,haveawayofcopyingabasictemplatequestionratherthanhavingtostartalloveragaineachtime”.“Thesettingupofthequizzesiscumbersomeandnotparticularlyintuitive.Needtobeabletocreatecategoriesandwritequestionsallintheonespot”.

Wiki Overall, 26.4% of staff agreed or strongly agreed that the Wiki engaged their students in learning activities. There were 8 references to Wikis, a few staff comments related to the wiki being unavailable during a critical time thus there was calls for stability. Whereas most of the staff responses referred to the ‘look and feel’ and further training. Some indicative comments included:“thewikigoingdownatacrucialtime”“Amoreelegantwiki-itdidn’tprovidethe“container”IwantedfortheshowcaseofICTactivities”“moreflexibilityintheorganisationofwikisforcollaborativelearning”

Database Overall, 35.9% of staff agreed or strongly agreed that the Database engaged their students in learning activities. There were 2 references to Databases, which indicated that some staff members would like further training: “IamkeentoknowhowtouseDatabases”.

Workshops Overall, 48.1% of staff agreed or strongly agreed that the Workshops engaged their students in learning activities. There were 2 references to Workshops, one staff member said that this tools is not relevant to their unit, whereas the other would like to see further enhancements:“Theworkshoptooldidn’tquitemeetmyneedsforpeerassessingastudentshowcase.Iwouldhavelikedtheabilityforstudentstoassess,buttheirassessmenttoremainconfiden-tial”.

Twitterfeeds Overall, 24.1% of staff agreed or strongly agreed that Twitter feeds engaged their students in learning activities. There were 3 references to Twitter feeds, requiring enhancements to make the system more “robust” and the other staff comments expressed both dislike as well as confusion as to what it could be used for in the teaching context.

53

Appendix 5. A list of iLearn Evaluation reportsThe following formal reports were presented to the iLearn Portfolio, iLearn Steering Committee and MACALT during 2012, however these groups were also given regular Evaluation updates and results.

• Staff iLearn Experience Survey Report;

• Student iLearn Experience Survey Report.

In order to provide feedback to the survey participants and University community the following two-page summaries were publically available on the iLearn website in 2012:

• iLearn Staff Experience Survey;

• iLearn Student Experience Survey.

In an effort to share the survey results with the project team members the survey results from the Staff and Student iLearn Experience Surveys were disseminated to the appropriate teams:

• Summary of ‘iLearn functions future use’;

• Summary of ‘UNITs feedback’;

• Summary of ‘most valuable iLearn functions’;

• Summary of ‘iLearn support and online resources’.

The results of the Session 1 iLearn Staff and Student Experience Survey was also presented at Learning and Teaching Week, 2012. If you would like a copy of any of these documents please contact Jayde Cahir: jayde.cahir@mq.edu.au