Learning Environment Online
Doing less with more
Dr Stephen DannSenior Lecturer, Research School of Management
Australian National University
Optimisation
Index• About the author• A Teaching Assessment Journey• Reframing Marking• Decisions to make to improve your time invested in teaching• The Workload Time Model of Assessment • Giving better feedback• Turnitin: Feedback Machine• Contact
About Dr Stephen Dann• PhD from 1998• Been in the business since 1997
• Business School Students• Compulsory first year courses• Compulsory second year courses• Angry engineering students doing a “Soft Bludge Subject In Commerce”• Really angry quantitative students doing “This Stupid Subject With Words”• Very upset and confused students doing this really hard and scary subject that uses
words, and “how do I sentence not-equation answer?” fears
• “Small” classes of 60 to 130 studentsIndex
Background• ANU Vice Chancellor’s Citation for Outstanding Contribution to Student Learning• ANU Achievement award: Candidate for Citation for Outstanding Contribution to Learning• ANU Commendation for Outstanding Contribution to Student Learning• Pearson ANZMAC Emerging Educator of the Year• ANU College of Business and Economics Award for Teaching Excellence• Senior Fellow, Higher Education Academy (UK)
A journey of assessment, technology, and much
improved outcomes (for me)Because when it’s no fun, it’s no fun for anyone
Index
Where I came from…• Exams were scary
• Undergrad sitting them, academic marking them
• Assessment was a dreaded chore• Stories told slightly differently, slightly wrong, • Grades were on the ordinary end of extra ordinary• Marking was postponed, avoided or generally delayed
• It wasn’t a lot of fun for anyone.• Student complaints about feedback, guidance and assessment load
• I was burning time out of my research quota to perform badly in teaching
If it could be done wrong…• Spending more time on marking than allocated• Getting frustrated by marking• A multiple choice exam of doom• Highest score was 40%.
• The bit where I had to scale… upwards
Where was it all going wrong?
Index
Problems and Solutions!• Problem 1:PEBKAC• Solution: Hack the Workload Model
• Problem 2: Why did I set that task?• Solution: Chekov’s Assessment Task
• Problem 3: Out of Alignment• Solution: Curriculum Design makes the work happier
• Problem 4: Assessment was a stick• Solution: Assessment is a better carrot
• Problem 5: Student Insultation• Solution: Self solving student
Problem 1:PEBKAC• Problem exists between keyboard and chair
• I wasn’t investing time, I was spending time• No upfront time investment• All of the late semester time sinks in marking
• Mechanical and slow marking process
• Teaching was chewing up more than the 40% of time, for less than 40% of the success
• So many student queries about things that I thought I’d covered…
Solution: Hack the Workload Model• All hail the workload model, giver of quantification, bringer of
approximations• 1 hour per student for marking
• 100 students, • 35 hour week, • 2 weeks and 4 days of full time marking to budget
• When do I want to spend my marking?• Oh, it’s how much, right, I’ll be needing this day and that day
• Motive to invest 4 days up front to clarify my own marking tasks
Problem 2: Why did I set that task?• Here’s the scenario• Weekly lecture (2 hours)• Tutorial activity involved discussion questions
• Assessable presentation• Short answer questions culminating in a “Tutorial Kit” of submitted answers
• An extended essay
• So let’s finish a semester with an exam with multiple choice questions which I hadn’t prepared the students to face.• It went badly
Chekhov's assessment task• "If you say in the first chapter that there is a rifle hanging on the wall,
in the second or third chapter it absolutely must go off. If it's not going to be fired, it shouldn't be hanging there."• Anton Chekhov (From S. Shchukin, Memoirs. 1911.)
• “If you’re going to put a style of question in the final exam, teach to the style of question throughout the semester”
Problem 3: Out of Alignment• When somebody said “Constructive alignment”, I heard “blah, blah,
blah”
• I should have heard “Would you like your job to be easier?”
Solution: Constructive Alignment• Biggs• 1) What outcomes do we intend the
students to learn?
• 2) How do we teach them to do that?
• 3) Does the assessment enable the students to do what we’ve taught them to do?• Assessment should then do what the
subject says on the tin
• Your new best friend• Graduate Attributes• Course Level Outcomes• Learning Outcomes
• National Standards
• Anything that says “This is what you can do, or should be able to do, or think, or perform”
Problem 4: Assessment was a stick• Assignments were hoops• “Achieve this. Answer that. Do that. Retell these factoids”
• Exams were punishment• Exams were written to punish students for deviating from the curve
Solution: Align assessment to the course• CA to GA to LO to “hello, this looks fun”!
• Graduate Attribute: • Enthusiasm to search for further knowledge and understanding.
• Learning Outcome: • Demonstrate an ability to search for further knowledge through the use of academic literature in written
essays• Essay Task:
• “Using existing papers from the library database, Google Scholar, draft an updated version of this Wikipedia page”• Marks Criteria: Use of academic sources + Breadth of research
• No more boring (to you) assignments on dull (to you) topics
• Support• LEO links to the Academic Skills • Explain how the literature is the easy path, not the hardest path
Problem 5: Consultation versus Insultation • Students would come to the office to see me about issues I’d thought
I’d covered in the class• Clearly I hadn’t covered those issues
• I thought I had, and I thought the students were being a bit… unhelpful
Solution 5: The Self Service Student• Aligning assessment to the course expectation reduced • “I don’t know what I’m doing” when they meant “I don’t know why I’m
doing”• “Why am I doing this? / “What’s the point? / This is pointless”• “Maybe if we all fail the course, he’ll get into trouble”
Self solving students who can ask “How do I meet this goal you’ve set?” rather than “Why am I do this at all?”
Digging upwardsGetting out of the teaching is burning all my time hole
Index
Reset your headspaceThe job is fun
The job resides within a systemSystems are designed to maximise input:output ratios
Tune up the input
People can enjoy complexity within clear structure…Realisation 1
People do maths for fun.• Video games are hard• Tough Mudder exists
• And charges $165
• Sudoko. Crosswords. IKEA
• People don’t fear challenges when they understand the requirements of the task.
• Clarify the why and how of the assessment
I should be enjoying assignment marking. I’m not, so something isn’t going right.Realisation 2
How to make assessment fun to mark?• Connecting what you expect
someone to do with what you teach/train them to do…
• Ask questions that you want to discuss.
• Pursue research in the classroom as a “What if?” and workshop the ideas
• Assessment as exchange of ideas• You get ideas• They get feedback
• You both win.
• Teach from your research, research from your teaching
Works nicely as well
Why should I enjoy assessment and marking?• I’m going to be doing a lot of it, for a long career
• Death and Taxes are covering “inevitable not fun” shifts
• Students are a lens to see problems in a new light• What if I ask questions for which I don’t know the answer?• What about questions that engage technique change?
• Where else do I get this sort of focus group opportunity?
Changing the Question• Ask questions where you don’t have a fixed answer in mind• Or if you do, there’s several of them• Criteria should govern technique and method
• Assess for technique, and set up so that technique can be showcased through content• Skills sets are graduate outcomes
• Give Feedforward and Feedback to the students • Explain success as much, if not more, than you explain failure
Workload and Marking
Index
Upfront Investment• Assessment Criteria• Know if it can be marked before you set it out to the students• Something I was routinely bad at doing, and it cost days at the marking end
• Set expectations well in advance• Reward the expected. • Do not begrudge success
• Assessment Support Before The First Assignment• You’ve been there before, and they haven’t
Assessment Videos are your friend• Talking through the assignment to camera
• Many takes make for a good solid think through before writing up the task
• Be interviewed by a colleague• Work in pairs to extract the key components
• Explain expectations• Talk about adequate, good and great performance
• Convey enthusiasm for the task• Be genuinely keen to see the assessment
• Students worry about technicalities in class, and engage depth in video• Formatting is a fear founded in modern education
Solving the assessment problem: LEO training
• http://www.acu.edu.au/staff/our_university/training_and_development/academic_staff/leo_learning_materials
• Ballarat , Brisbane , Canberra , Melbourne , North Sydney , Strathfield• Learning and Teaching Centre 1st Feb 2016 – 20th Feb 2016
• Guided by local eLearning Advisors from the Learning and Teaching Centre (LTC), you will learn how to create and organise learning materials in your LEO unit/s; create and style LEO Page and Book activities; upload and access materials in LEO content; and use Kaltura for your video resources.• Intended Outcomes
• At the end of this workshop it is expected that participants will be able to:• create an item in EQUELLA and add a document, or documents to it;• add documents from your EQUELLA items to your LEO units.• style Page and Book activities in LEO• create and upload video materials using Kaltura
The Pointy End of AssessmentMarking
Index
1 hour per student per semester• That’s 6 minutes per 10%
• Six minutes is not enough time to have any fun with assessment
• Minimum cut-off: 20% and 12 minutes.• In 12 minutes, you can
• Read the paper (5 minutes) • Select marks from the criteria (1 minute) • Add QuickMarks for feedback (3 minutes)• Leave a 3 minute voice feedback of personalised instruction (3 minutes)
• 40% Assignments – 24 minutes• 3 minutes of voice feedback (21 minutes left) • Read the paper (10 minutes) (11 minutes left)• Select marks from the criteria (1 minute) (10 minutes left)• Add QuickMarks for feedback (5 minutes) (Change on the clock)
Mandatory Business Unit ExamsUnit Exam Weighting Exam Structure Total Subcomponents Minutes per component
1A 50% 2 x 4 questions 17 1 min 42 sec 1B 50% 3 from 4 questions 4-5 6 – 7 mins 30 sec 1C 40% 6 compulsory 24 1 minute1D 50%
A 12.5%B 37.5%
A – 25 mcqB – 3 compulsory
25 + 11 MCQ – 7.5 minutes for processing, calculation, delivery and inputB – 22.5 total = 2 minutes
1E 45% A – 20 mcqB – 3C - 2
20 + 9 MCQ – 6 minutes for processing, calculation, delivery and inputB – 1 min 26 secC – 4 mins
1F 50% 5 questions 10 3 minutes
Assessment Maths: 25 students• 30% Exam. 1 x 20 mark, 2 x 5% from a choice of 3• 75 moving parts
• 40: Exam with 6 questions, choice of 5, marked from 100, 20 marks each, 5 sub components• 625 moving parts
FeedbackThe single, universal, consistent point of failure in education (according to student satisfaction surveys)
Index
Feedback• the most powerful single moderator that enhances achievement• https://teaching.unsw.edu.au/assessment-feedback
• It’s also really fun (and sometime emotional)• You got to get a little teared up when they do so well
Doing Feedback Well• constructive.• highlighting the strengths and weaknesses of a given piece of work, • set out ways in which the student can improve the work.• Tell the students what they did right, and why it was right
Doing Feedback Well• Timely• Sooner rather than later• Students will have always moved on.• Aim the feedback as feedforward (talk to the next task)
Doing Feedback Well• Meaningful. • Talk to the assessment task• Talk to the subject, and the broader scheme of the assessment• Linked to specific assessment criteria• Show a connection between the actions in the assessment and the learning
outcomes• Move quickly
Why is feedback not done well?• Timely Feedback is hard work• 100 assessment tasks x 10 comments of 5 to 10 words in hand writing = a very long day at the office
• Repetition happens on the marking side, not the student side• “Didn’t I just correct this?”
• Repetition creates Resentment • Resentment is never fun
• Fatigue
Fatal 4 Marking• Fatigue• Tired markers make mistakes
• Frustration • Anger does not help the process either
• Inconsistency and Shifting Expectations• Is this good? Bad? What even was my question or even my answer?
• Mmrmrmrmrmm• Handwriting approaches catastrophic failure
Turnitin: Feedback Machine
For when feedback needs to be timely, industrial strength, and fasthttp://www.acu.edu.au/staff/our_university/training_and_development/academic_staff/leo_assessments
Index
Turnitin and Textmatch
Textmatch is an opportunity• Establish expectations• Explain that a text match is cliché,
and clichéd writing is tacky
• Encourage the deeper learning of using the idea, not the phrase
• Tag and highlight matches with advice on how to write from the ACU Academic Skills Unit• http://students.acu.edu.au/office_of
_student_success/academic_skills_unit_asu
Turnitin buys time backQuickmarks are quick.
QuickMarks are drag and drop
Quickmarks can give depth
Quickmarks can praise
Quickmarks can inform
Quickmarks are more than you’re even going to consider writing by hand
But what about?
I like to mark on the couch/beach/park bench/backpacking in the Andes/McDonalds
But I like the physical paper…• Turnitin provides• Reduced liability for lost assignments• Backups of comments• Security and confidentiality• Can enable blind marking• “Dog ate my homework” proof to p=0.01 level• Reduced physical strain from handwriting• Improved portability with increased security for work-from-home • Much lighter to carry than 100 20 page essays
What does it do for you?• Improves student outcomes• Feedback has impact
• Individual personalised voice feedback• Creates cut through• Really reaches the student• Adds nuance to the conversation• Reduces assessment follow-up meetings
• Buys time back from the workload
Why do it?Because it buys time back in your work-life
