southampton, oct. 2009
DESCRIPTION
Participatory Pattern Workshops: Application to formative e-assessment. Southampton, Oct. 2009. Problem: The Design Divide the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*) . - PowerPoint PPT PresentationTRANSCRIPT
1Southampton, Oct. 2009
Participatory Pattern Workshops:Application to formative e-assessment
2
the critical resource is not the capacity to produce, but the knowledge to do it right.
Problem: The Design Divide the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*)
3
Solution...(in architecture)
4
What is a pattern?
• At is simplest, it is a– Generalised solution to a problem– Follows a specific structure
C o n t e x t
Problem Solution
When, Where, Who
What are we trying to achieve / solve?
Cookbook: ingredients, procedure, expected
outcomes
5
A burda pattern..
Season: Fall For: Women Garment Type: Dress Style: Classic, Evening Wear, RomanticMaterial: Taffeta
“if I copy a dress, I can only create the same dress. If I have a pattern, I can create many dresses” (Yim Ping LENDEN)
6
As for software
7
Participatory Methodology for Practical Design Patterns
• Problem– Acceleration → need for effective protocols for sharing
of design knowledge
• Context– interdisciplinary communities of practitioners
engaged in collaborative reflection on a common theme of their practice.
– blended setting: co-located meetings + on-line collaborative authoring system.
Son, this was my dad's mobile. I want you to have it.
8
The Participatory Pattern Workshops Methodology
9
Collaborative reflection workshopFacilitate on-going design-level conversation between designers and practitioners involved in diverse aspects of the problem domain.
Open, trusting and convivial.
And at the same time
Critical, focused and output-directed.
10
Collaborative reflection workshop
11
Case Stories Workshop
Engender collaborative reflection among practitioners by a structured process of sharing stories of successful practice.
12
Pattern Mining Workshop
Shift from anecdotes to transferable design knowledge by identifying commonalities across case stories, and capturing them in a semi-structured form.
13
Future Scenarios Workshop
Validate design patterns by applying them to novel real problems in real contexts.
14
the cycle of design research
15
the cycle of design research
formative e-assessment: case stories, design patterns, and future scenarios
Caroline Daly, Harvey Mellar, Yishay Mor, Norbert Pachler,
Institute of Education, University of London
http://feasst.wlecentre.ac.uk/
Overview
• Scoping study commissioned by JISC
• Short term, small budget, intended to inform future funding frameworks
• Established a commited user group of higher-education teachers & researchers
• Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform
Methodology
Desk research
Literature review
Comparing frameworks
5 Practical Enquiry Days
Combination of collaborative reflection, report back from team, and guest plenaries
Launch day, 3 Planet workshops, developers' day
What is formative e-assessment?
• No consistent view in the literature
From “anything test before the final” to “synonymous with learning”
• The use of digital means to support formative assessment
• Formative features of assessment, which are afforded by specific features of digital media
OK, so what is Formative Assessment?
“An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence”
(Dylan Wiliam)
Formative = feedback + moments of contingency
"... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another."
(Leahy, Lyon, Thompson, and Wiliam 2005)
Teacher
Learner
Peer
Instruction
Tasks
Actions
Tasks
Actions
Wiliam's 5 stratagies
Conversational Framework (Laurillard)
Evidence Centred Design (Mislevy)
• Highly developed
• Pattern based
• Oriented to large scale, automated systems
• Measurement centric
• Light on theory
• Less suitable for open activity designs
A few cases
• Creature of the week
• CoMo
• Post 16 String Comparison
• Open Mentor
• ...
Creature of the week (Judy Robertson)
Situation
large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life.
Task
Engage and motivate the students show examples of good work which others could
learn from show students their work is valued. build a sense of community.
http://purl.org/planet/Cases/creatureoftheweek
CoMo (Niall Winters, Yishay Mor)
Situation Royal Vet College. Hospital rotations as part of their training.
Task Allow students to capture critical incidents in text and
image. Support sharing of clinical experiences and co-
reflection.
http://purl.org/planet/Cases/CoMo
Post 16 string comparison (Aliy Fowler)
Situation Grammar school been piloting the ‘string comparison’ approach to
language teaching at post-16 for AS and A2 level students.
Sixth Form level, grammatical consolidation and whole-sentence translation.
Task Allow students to practise written language
independently and receive feedback on errors in order to improve their language skills.
http://purl.org/planet/Cases/Post16stringcomparison
Solution
A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.
Open mentor (Denise Whitelock)
http://purl.org/planet/Cases/OpenMentor
A few patterns..
• Try Once, Refine Once
• Feedback on Feedback
• Classroom display
Try Once, Refine Once
(Aliy Fowler)
http://pul.org/planet/Patterns/TryOnceRefineOnce
ProblemLack of immediate feedback for students leads to fossilisation of errors and misconceptions
providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.
Context
Class size Large (30-300)
Content Skills \ facts
Mode of instruction Blended / on-line. Computer tested.
Solution
Feedback on
Feedback
(Linda McGuigan)
http://purl.org/planet/Patterns/FeedbackonFeedback
Good feedback should -
Alert learners to their weaknesses.Diagnose the causes and dynamics of these.Include operational suggestions to improve the learning experience.Address socio-emotive factors.
Tutors know this, but are pressed for time. Or not aware of their feedback strategies
Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching
Problem
Context
Large scale, technology supported, graded courses many tutors instructing many students.
Feedback is mediated by technology that allows it to be captured and processed in real time
Topic of study is subject to both grading and formative feedback.
SolutionEmbed a mechanism in the learning and teaching system
that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms.
In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.
Classroom Display
http://purl.org/planet/Patterns/Classroomdisplay
Problem
Rewards participation.Relates to learner's personal experiences.Window on student conceptions.
Using learner Using learner generated content..generated content..
Needs to collate works in a single easy to access location.Learners uncomfortable about presenting their work in publicLegal or other restrictions on sharing work.
Context
Class size: Small / medium (6-60)
Mode of instruction: Blended (preferable)
Time frame Continuous, over a period
Pedagogy Involves construction / media production
Solution
Augmented domain map
Example scenarioWhen using Try Once Refine OnceTry Once Refine Once, there is a risk
that high-achievers do not receive feedback.
So -
Use Showcase Learning Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors.
Use Feedback on Feedback Feedback on Feedback to alert tutors to the problem.
Conclusions
Tip of the iceberg
Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios
Collaborative elicitation of patterns from cases could be a potent form of professional development.
50
Thank youThe pattern language network project:
http://patternlanguagenetwork.org
The learning patterns project:
http://lp.noe-kaleidoscope.org/
The formative e-assessment project:
http://feasst.wlecentre.ac.uk/
This presentation:http://www.slideshare.net/..?
Yishay Mor
people.lkl.ac.uk/yishay
@yishaym
Niall Winters
www.lkl.ac.uk/naill
@nwin
Harvey Mellar
lkl.ac.uk/people/mellar.html