![Page 1: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/1.jpg)
Evaluation Research into the effectiveness of university learning environments
Rob PhillipsAcademic Consultant
Educational Design Project, School of Education, Curtin University, PerthAdjunct Associate Professor, School of Education, Murdoch University, Perth
![Page 2: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/2.jpg)
My Background
• 1980s– Research scientist – theoretical chemistry and computer
science – Germany and Australia• 1990s
– Interactive multimedia designer and project manager• 2000s
– Educational designer• 2010s
– Learning and Teaching Policy work– Academic consultant – mentoring and educational design
![Page 3: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/3.jpg)
Introductions
• Why are you here?
![Page 4: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/4.jpg)
Book advert
![Page 5: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/5.jpg)
Disclaimer
• The book is largely about e-learning• The ideas apply well to any learning
environment– almost all include technology these days
![Page 6: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/6.jpg)
Overview
• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project
![Page 7: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/7.jpg)
Fundamentals of Research
• Inquiry into the behaviour of phenomena• Exercise of human curiosity• Attempting to solve problems in the world• Attempting to understand the world
![Page 8: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/8.jpg)
In Broad Terms
• What sort of research do you do?• What do you investigate?• What are the main characteristics of this
research?
• Work in pairs
![Page 9: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/9.jpg)
Fundamentals of Research
• Inquiry into the behaviour of phenomena• Exercise of human curiosity• Attempting to understand the world• Attempting to solve problems in the world
![Page 10: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/10.jpg)
What is a phenomenon?
• Inquiry into the behaviour of phenomena• What is a phenomenon?
– an instance of something – an artefact or event that is known through the senses
![Page 11: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/11.jpg)
Natural Phenomena
• They exist naturally in our world– the seasons, gravity and waves on the ocean– we can observe and measure them
• Other natural phenomena cannot be directly observed– the inner workings of atoms and molecules– their behaviour can be measured
![Page 12: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/12.jpg)
Artificial Phenomena
• Phenomena/ artefacts which we create through a design activity
• Tangible – aircraft, cars, etc.• Intangible – computer software
![Page 13: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/13.jpg)
Event Phenomena
• An artefact or event that is known through the senses
• In Education– a coming-together of teacher, learners and various
resources in a particular setting, usually following some design activity
– Learning Environment
![Page 14: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/14.jpg)
Summary
• Natural phenomena• 1 phenomenon – many instances
• Artificial phenomena– Many phenomena – many instances
• Event Phenomena– Many phenomena – single instances
![Page 15: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/15.jpg)
Paradigms vs Ideology
Paradigms• The set of practices that define a scientific
discipline at any particular period of time (wikipedia)
• A world view underlying the theories and methodology of a particular scientific subject (http://oxforddictionaries.com)
![Page 16: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/16.jpg)
Ideology
• The set of beliefs characteristic of a social group or individual (http://oxforddictionaries.com)
• A comprehensive and coherent set of basic beliefs about political, economic, social and cultural affairs that is held in common by a sizable group of people within a society. (http://www.socialsciencedictionary.com)
![Page 17: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/17.jpg)
Paradigms vs Ideology
• Is your research approach based on a paradigm or an ideology?
![Page 18: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/18.jpg)
Overview
• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project
![Page 19: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/19.jpg)
E-learning Artefacts and Environments
• We design and use e-learning artefacts (intangible artificial phenomena)– learning objects, learning management systems,
e-portfolios• e-learning environments (event phenomena)
add the human context– an interaction between learners, teachers and
resources in a set context
![Page 20: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/20.jpg)
Inquiring About the World (IATW)
• New science unit for a new primary and early childhood degree– First offered in 2013
• Teaching problem – How to make science interesting and engaging
• Design– Authentic learning approach– Students investigate a self-chosen environmental issue over
the whole semester
![Page 21: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/21.jpg)
Learning Outcomes• Develop lifelong learning skills including: analytical thinking,
information management, technology skills and communication skills
• Apply processes of inquiry to carry out a scientific investigation
• Report on the outcomes of the processes of inquiry using a range of Information and Communication Technologies
• Evaluate the outcomes of the inquiry processes in the context of the science and humanities content from the Australian Curriculum
![Page 22: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/22.jpg)
Assessment1 Project Plan 15%
2 Report Draft 20%
3 Report 45%
4 Presentation 20%
![Page 23: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/23.jpg)
Video
• IATW Flash video
![Page 24: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/24.jpg)
Baseline
The Learning LifecyclePhase
Design
Design
Develop
Evaluate
Trial learning env’t
Learning environment
Mature Learning environment
Learning lifecycle
![Page 25: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/25.jpg)
The IATW Lifecycle
• On- campus and Open Universities Australia (OUA)
Date Offering
July 2013 Curtin Semester 2
May 2014 OUA Study Period 2
July 2014 Curtin Semester 2
Nov 2014 OUA Study Period 4
May 2015 OUA Study Period 2
![Page 26: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/26.jpg)
Overview
• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project
![Page 27: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/27.jpg)
My Invention
• The Kugelschreiber• A better way to write!
![Page 28: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/28.jpg)
Researching the Kugelschreiber
• I know that this technology is better than old, pencil technology
• Compare the two, in writing a test• Treatment and control groups
![Page 29: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/29.jpg)
Researching the Kugelschreiber
• I know that this technology is better than old, pencil technology
• Compare the two, in writing a test• Treatment and control groups
Kugelschreiber results were slightly worse than pencil results, but the difference was not significant at the 90% confidence level. The Kugelschreiber is no worse than a pencil.
![Page 30: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/30.jpg)
Inquiry into Phenomena
• Natural– Goal: observing how natural phenomena work
• and ultimately understand them
• Artificial– Goal: design artefacts, improve their design and use
them • and ultimately understand them
• Event– Goal: design events, improve their design
• and ultimately understand them
![Page 31: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/31.jpg)
Goals of Research
• Be clear about the phenomenon you are investigating
• Clearly define the goals and questions of your research
• Choose the research approach accordingly
![Page 32: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/32.jpg)
Summary so Far
• We have thought about the phenomenon of e-learning
• We have different research goals at different stages of the e-learning lifecycle
• How will we conduct this research?
![Page 33: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/33.jpg)
General Research Approaches
• Pure versus applied research• Hard versus soft research• Research and development• Research then development
Becher, T. (1989). Academic tribes and territories: intellectual enquiry and the cultures of disciplines. Buckingham: Society for Research into Higher Education.Jones, C., Zenios, M., & Griffiths, J. (2004). Academic use of digital resources: Disciplinary differences and the issue of progression Proceedings of the 4th Networked Learning Conference. Lancaster University, England, UK.
![Page 34: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/34.jpg)
Boyer’s Four Scholarships
• Scholarship of discovery • Scholarship of integration• Scholarship of application• Scholarship of teaching
Boyer, E. L. (1990). Scholarship reconsidered - Priorities of the professoriate. New Jersey: Carnegie Foundation for the Advancement of Teaching.Boyer, E. L. (1996). The scholarship of engagement. Journal of Public Outreach, 1(1), 11-20.
![Page 35: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/35.jpg)
Pasteur’s Quadrant
Stokes, D. E. (1997). Pasteur's Quadrant: Basic science and technological innovation. Washington, D.C.: The Brookings Institution.
![Page 36: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/36.jpg)
What is evaluation?
• The term is used imprecisely• Evaluation implies judgement and decisions• Formative evaluation:
– judgements about improving something• Summative evaluation:
– judgements about the merit or worth of something
![Page 37: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/37.jpg)
Investigating a New Learning Environment
• Questions:– Does the environment function as designed?– How can the environment be improved?
• These are evaluation judgements• Is it any good???
![Page 38: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/38.jpg)
Investigating a Mature Learning Environment
• A study of a mature learning environment may shed light on – how learners engaged with the designed learning
environment – why some learners achieved at different levels– how some learners used the learning environment
to achieve a deeper understanding• These are educational research questions• The goal is increased understanding
![Page 39: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/39.jpg)
Evaluation vs Research
• Studies of learning environments involve a mixture of– Evaluation: making judgements about improvement and
value and worth– Research: increasing understanding
• Continuum between judgements and understanding– Does it function as designed?– How can it be improved?– How does this lead to improved student learning?
• Similar methods – different goals
Judgments
Understanding
![Page 40: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/40.jpg)
IATW - currently
• Pre- and post-test– Confidence with ICT– Knowledge of scientific inquiry processes
• Anonymous survey of students– Did they like it?
![Page 41: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/41.jpg)
IATW questions to ask
• Was the website usable?• What things didn’t work as well as they should?• Was there evidence that students engaged with the
authentic task?• Will the learning design scale up for 2000 OUA
students?
![Page 42: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/42.jpg)
Overview
• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project
![Page 43: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/43.jpg)
Baseline
The Learning LifecyclePhase
Design
Design
Develop
Evaluate
Trial learning env’t
Learning environment
Mature Learning environment
Learning lifecycle
![Page 44: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/44.jpg)
Baseline analysis
Evaluation Research across the Learning Lifecycle
Design evaluation
Design
Develop
Evaluate
Formative evaluation
Formative evaluation
Summative evaluationEffectiveness research
Learning lifecycle
![Page 45: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/45.jpg)
e-learning Research Approaches
• Need to apply to an artificial phenomenon• Reflect the e-learning life cycle
• Cyclical research approaches– Design-based research – Action Inquiry
![Page 46: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/46.jpg)
Design-based Research• Focus on a designed learning environment• A cyclical approach that originated in engineering and
other design fields
• Consistent with – a mixture of evaluation and research– Stokes’ views on quest for understanding and consideration
of use
![Page 47: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/47.jpg)
Action Inquiry
• Focus on personal practice• Local focus• Little concern with theory
![Page 48: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/48.jpg)
Drawing the Threads Together
• e-learning environments are designed phenomena– they follow a lifecycle
• Cyclical research approaches• Mix of evaluation and research
– make sure it works first• Now – onto the practical stuff
![Page 49: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/49.jpg)
Overview
• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project
![Page 50: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/50.jpg)
Breaking Down the Complexity
• How to use all this information:– It’s a complex situation, with– many elements, which– evolves over time
• It requires multiple research methods• Use divide and conquer techniques and a
template to simply the issue
![Page 51: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/51.jpg)
Divide and Conquer Techniques
• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle
– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-
research plan
![Page 52: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/52.jpg)
Evaluation-research Process
![Page 53: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/53.jpg)
Evaluation-research Process 1
![Page 54: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/54.jpg)
Design Principles
• relatively humble ‘proto-theories’• not ‘grand’ theories of learning • Two roles for theory:
– Drawing on theory to support evaluation-research goals and questions across the e-learning lifecycle
– Building theory from research outcomes
![Page 55: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/55.jpg)
Builds theory
Draws on theory
Baseline analysis
Evaluation Research across the Learning Lifecycle
Design evaluation
Design
Develop
Evaluate
Formative evaluation
Formative evaluation
Summative evaluationEffectiveness research
Learning lifecycle
![Page 56: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/56.jpg)
Evaluation-research Process 1
![Page 57: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/57.jpg)
The LEPO Framework
• Learning Environment, Process, Outcomes
Educational Context
Learning environments
Learning processes
Learning outcomes
Students Teachers
facilitate
design facilitate
assess
lead todetermine
work within
engage in
demonstrate
interact with
![Page 58: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/58.jpg)
IATW Conceptual Framework
• Model for inquiry-based learning• Model for information literacy
• Herrington’s 9-stage authentic learning model
![Page 59: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/59.jpg)
Broad Evaluation-Research Questions
• The broadest questions that can be asked in the study
• Open-ended, exploratory questions • IATW examples:
– How can the unit design be improved for a large, online OUA cohort?
– How effective is the unit at developing inquiry skills in students?
![Page 60: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/60.jpg)
Evaluation-research Process 2
![Page 61: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/61.jpg)
IATW
• Stakeholders– Centre for Teaching and Learning– Head of School– Standards bodies
• Participants– Students– Teaching staff– LMS support staff
![Page 62: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/62.jpg)
Activity
• Fill in page 1 of the template• Discuss it with your partner
![Page 63: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/63.jpg)
Divide and Conquer Techniques
• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle
– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-
research plan
![Page 64: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/64.jpg)
Evaluation-research Forms• Baseline analysis: Document the teaching and learning
context; needs analysis; assumptions, literature• Design evaluation: Evaluate the documented design of
the learning environment• Formative evaluation: Make judgments about, and
suggest improvements to, the e-learning environment• Effectiveness research: Develop understanding of how
learners engage with learning tasks to demonstrate learning outcomes
• Project management evaluation: judgements about the conduct of a project
![Page 65: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/65.jpg)
Evaluation-research Process 3
![Page 66: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/66.jpg)
Specific Evaluation-research Questions
• Broad evaluation-research questions are hard to answer
• Specific evaluation-research questions should be answerable– identify the evidence which is needed to answer
them• Synthesise the evidence from the specific
questions to answer the broad questions
![Page 67: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/67.jpg)
Examples of Specific Research Questions
Formative evaluation of the e-learning environment • How do learners use the learning environment?• To what extent do learners engage with learning
tasks as intended?• Are the learning tasks appropriate for the target
learners?• How useful do learners find the e-learning
environment?
![Page 68: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/68.jpg)
Divide and Conquer Techniques
• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle
– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-
research plan
![Page 69: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/69.jpg)
Evaluation-research Process 3
![Page 70: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/70.jpg)
Evaluation-research Matrices
• To map data sources and methods against the research questions
• Examples from IATW
![Page 71: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/71.jpg)
Baseline Analysis
![Page 72: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/72.jpg)
Design Evaluation
![Page 73: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/73.jpg)
Formative Evaluation
![Page 74: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/74.jpg)
Effectiveness Research
![Page 75: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/75.jpg)
Divide and Conquer Techniques
• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle
– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-
research plan
![Page 76: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/76.jpg)
Evaluation-research Plan
• Use the template• Work through the evaluation matrices and
develop questions which are relevant to your project
• Specify your data sources– Interview schedules– Survey questions
![Page 77: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/77.jpg)
Do the evaluation research
![Page 78: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/78.jpg)
Overlap between Forms
![Page 79: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/79.jpg)
Overview
• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project
![Page 80: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/80.jpg)
Evaluation-research Plan
• Use the template to start to develop your own evaluation research plan
• Work through the evaluation matrices and develop questions which are relevant to your project
![Page 81: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/81.jpg)
Application of the Approach
• PhD research• Teaching improvement• Preparation for grant applications
![Page 82: Evaluation Research into the effectiveness of university learning environments](https://reader031.vdocuments.us/reader031/viewer/2022012917/56815fd6550346895dced802/html5/thumbnails/82.jpg)
Where to publish your work?Journal of Perspectives in Applied Academic Practicehttp://jpaap.napier.ac.uk/index.php/JPAAP International Journal for SoTLhttp://academics.georgiasouthern.edu/ijsotl/v5n1.htmlInvited essays, Research papers, essays MERLOT Journal of Online Learning and Teachinghttp://jolt.merlot.org Journal of University Teaching and Learning Practice http://ro.uow.edu.au/jutlp/ International Journal of Innovation in Science and Mathematics EducationIJISME is dedicated to encouraging and supporting early career researchers as well as academics in science and mathematics who are new to education research.http://ojs-prod.library.usyd.edu.au/index.php/CAL/index