Effective dissemination and evaluationEffective dissemination and evaluation

Workshop on Teaching and Learning Grants and Carrick Schemes, 2008

Merrilyn Goos and Clair Hughesm.goos@uq.edu.au; clair.hughes@uq.edu.au

Session overview• A project scenario• Dissemination and evaluation

defined• Purposes of dissemination and

evaluation• Small group development of

dissemination and evaluation strategies

• Resources

A project scenario

• “Introduction to University Teaching and Learning” program: aims, objectives, participants, teaching team

• New course structure and content from 2008• How to disseminate this innovation - not just to inform

potential participants and other stake-holders but to embed good practice?

• How to evaluate the course in its first offering in order to monitor and improve its implementation?

Dissemination definedDissemination is more than distribution of information or making it available in some way. Dissemination also requires that some action has been taken to embed and upscale the innovation within its own context and to replicate or transform the innovation in a new context and to embed the innovation in the new context.

Carrick Institute Dissemination Framework

Evaluation definedEvaluation is the process of determining the merit and/or worth of things for a range of purposes.

Consider funding agency requirements for evaluation:• Ensure that evaluation is undertaken at multiple points throughout the project and is understood and reported within an evaluation framework.

Carrick Competitive Grants program: Guidelines and Supporting Information – 2008 (p.23)

• Must provide a strategy for the evaluation of the project with information on outcomes for students, whether there has been a noticeable improvement or not in the expected outcomes for students and how the project outcomes will be disseminated to others within the discipline.

UQ Teaching and Learning Small and Large Grants Scheme Guidelines

Purposes of project dissemination

Dissemination for awareness(scattering the seed to the wind)Information provision

Distributing information through reports, publishing papers, conference presentations, sending emails, developing websites.One way information process.

Dissemination for understanding(sowing the seed in prepared soil)Information provision

Distributing information in a more purposeful manner, e.g. workshops, visits.Requires interaction with others who are involved in the project.

Dissemination for action(propagating, breeding, growing, grafting)Engaged dissemination

Purposeful, directed, systematic, proactive engagement, evidence based, involves adaptation and implementation, resulting in changes in practice.

Purposes of project evaluation

Formative(to inform ongoing project decision-making)

•Monitor project progress•Revise/enhance project processes and outcomes•Engage key stakeholders early and during project to:

–demonstrate respect–intensify participation–collaborate in identifying key indicators of success–progressively expose to new perspectives, or information

•Make interim reports

Summative(to determine/support project findings and outcomes)

•Report on success/results/value•Demonstrate impact•Account to sponsors•Sustain project outcomes•Support related submissions (eg grants for funding for further related projects; promotion; teaching and learning awards)

Enlightenment(to learn)

•Provide opportunities for project team to maximise the learning developed through participation in the project

Adapted from “The Learning Partnership” 2007 and W. K. Kellogg Foundation

Discussion

• In groups of 3 or 4:– address the following dissemination and evaluation

questions in the context of the project scenario presented earlier (new “Introduction to University Teaching and Learning” program)

– Consider how you would address dissemination and evaluation for a project in your own educational context

– Share your responses with the larger group

Dissemination questionsWho are the users of and other stakeholders in this program?

What do we want to disseminate (process, ideas, products)?

How could we make users and other stakeholders aware and involved?

How could we engage users and other stakeholders so they use and/or support the program?

How could we obtain feedback from users during the project?

How will we know there has been change in practice?

Evaluation questionsWho is our project audience (stakeholders)?

Select two stakeholders/groups and list one or more questions each may have in relation to this project.

How could we collect data to answer these questions? (methods)

From whom or where (stakeholders and others) could data be collected? (sources)

What will count as evidence? (consider both qualitative and quantitative)

The evaluation framework

Evaluation questions

Sources/Methods

Institutional contacts

Expert reference

group

Existing records/artefacts

What is Generic Graduate Attribute (GGA) policy in Australian universities?

Telephone interview (to clarify, address omissions or current activity)

Web search of university policy documents

What are current barriers to effective GGA implementation?

Focus groups to elicit responses to issues paper based on literature review

Written responses to draft issues paper

Literature review (research into implementation experience)

Analysis of AUQA reports

EXAMPLE

Models of planning for evaluation

Implementation Evaluation

Dissemination

Implementation Evaluation

Dissemination

ImplementationEvaluation

Dissemination

Implementation Evaluation

Dissemination

References• Carrick Institute for Learning and Teaching in Higher Education. (2006). Carrick Institute

Dissemination Framework Retrieved from http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/dissemination_carrickframework_2006.pdf

• Carrick Institute for Learning and Teaching in Higher Education. (2008). Competitive Grants program: Guidelines and Supporting Information – 2008. Retrieved from http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/grants_competitive_guidelines2008_july07.pdf

• Evaluating projects resource  A resource to assist applicants planning the evaluation framework component of their Grants Scheme application has been prepared by Murdoch University <http://www.murdoch.edu.au/teach/carrick_evaluation/index.html>

• Felton, P., Kalish, A., Pingree, A., & Plank, K. (2007). Towards a scholarship of teaching and learning in educational development. In D. Robertson & L. Nilson (Eds.), To Improve the Academy: Resources for Faculty, Instructional, and Organisational Development (pp. 93-108). Boston: Anker.

• The Learning Partnership. (2007). Material distributed during Leading and Managing Projects, a workshop sponsored by the Carrick Institute for Learning and Teaching in Higher Education. Melbourne, 17 – 18 September.

• TEDI Evaluation Services http://www.tedi.uq.edu.au/evaluations/index.html• The University of Queensland. (2007). UQ Teaching and Learning Small and Large Grants

Scheme Guidelines. Retrieved 29 October, 2007, from http://www.uq.edu.au/teaching-learning/download/TLStrategicGrantsGuidelines07.pdf

• W. K. Kellogg Foundation. Executive summary: Introducing Program Logic Models. Material distributed during The Professional Development Program for Medical and Healthcare Educators offered jointly by the University of Queensland and Harvard Medical International. 21-25 August, 27-29 November, 2006.