6 Aug 2014 : 07:00AM GMT Designing e-Assessment Activities for Online Courses Mary Jacob, Aberystwyth University Your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[at]rmit.edu.au Dr Mathew Hillier, Teaching and Educational Development Institute, University of Queensland mathew.hillier[at]uq.edu.au

Just%to%let%you%know:%%By#par'cipa'ng#in#the#webinar#you#acknowledge#and#agree#that: The#session#may#be#recorded,#including#voice#and#text#chat#communica'ons#(a#recording#indicator#is#shown#inside#the#webinar#room#when#this#is#the#case).#We#may#release#recordings#freely#to#the#public#which#become#part#of#the#public#record.#We#may#use#session#recordings#for#quality#improvement,#or#as#part#of#further#research#and#publica'ons.

Mary%Jacob,%E,learning%Advisor%Aberystwyth%University%%%%%%%%%%%%%%%http://nexus.aber.ac.uk%%

!  Assessment%in%MOOCs%and%other%online%courses%

! What%makes%a%good%assessment?%

!  Improving%questions%,%activity%!  Auto,marked%quizzes%%!  Peer%assessments%

Have%taught%%

Plan%to%teach%

Might%teach%%

No%plans%to%teach%

A.%

B.%

C.%

D.%

•  Attempted%most%of%the%assessments%%Completed%

•  Attempted%assessments%at%the%beginning%but%then%slowed%or%stopped%Disengaged%

•  Watched%lectures%but%attempted%very%few%assessments%%Audited%

•  Briefly%explored%the%course%by%watching%a%few%videos%%Sampled%

Based%on%Kizilcec%R.,%Piech%C.,%Schneider%E.%Deconstructing%Disengagement:%Analyzing%%Learner%Subpopulations%in%Massive%Open%Online%Courses%[Internet].%[Stanford,%CA]:%%Stanford%University,%2013%%[published%2013%April%02;%cited%2013%May%28].%Available%%from:%http://lytics.stanford.edu/deconstructing,disengagement/%%

A.%

B.%

C.%

D.%

Marked%

• Auto,marked%quizzes%

• Peer,marked%assignments%

Not%marked%

• Discussion%board%activity%

• Posting%to%social%media%with%tags%(Flickr,%Twitter,%etc)%

Video%%

Quiz%%

Video%%

Quiz%%

Short%reading%

Peer%assessment%%

Discussion%board%activity%

A.  Yes,%formally%in%a%training%session%B.  Yes,%informally%by%asking%other%people%C.  Yes,%but%only%by%reading%or%using%online%

resources%on*my*own*D.  I’m%not*sure*if%I%have%or%not%E.   No,%I%haven’t%

Photo%by%Ryan%M.%(Shinalight)%available%on%Flickr%with%Creative%Commons%https://www.flickr.com/photos/shinealight/2220267854/sizes/o/%%

!  Based%on%your%experiences%with%MOOCs…%

! …do%you%think%that%the%writers%had%training%in%writing%effective%questions?%

!  Use%thumbs%up%or%down%for%Yes%or%No*!  Use%chat%window%for%more%complex%responses%%

Based%on%Benjamin%S.%Bloom,%Taxonomy(of(educational(objectives:(the(classification(of(educational(goals.%New%York:%David%McKay%Co.%[London]:%Longman,%1956.%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

! Facts%

Based%on%Heer,%R.%A%Model%of%Learning%Objectives–based%on(A(Taxonomy(for(Learning,(Teaching,(and(Assessing:(A(Revision(of(Bloom's(Taxonomy(of(Educational(Objectives%by%Rex,%Center%for%Excellence%in%Learning%and%Teaching,%Iowa%State%University%http://www.celt.iastate.edu/teaching,resources/effective,practice/revised,blooms,taxonomy/%%%

! Facts%

! Concepts%

! Concepts%

Facts% Concepts%A%

B%

C%

Objectives%%

Learning%activity%

Assessment%%

Biggs,%J.%%(1999):%Teaching%for%Quality%Learning%at%University.%SRHE%and%Open%University%Press,%Buckingham.%

What%level%on%Bloom?%

Future%of%Storytelling%%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

!  Knowledge%reproduction%only%

!  Use%scenario,based%or%problem,solving%questions%%!  Push%higher%on%Bloom’s%taxonomy%

%

What%level%on%Bloom?%

Songwriting%

What%level%on%Bloom?%

Songwriting%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

What%level%on%Bloom?%

Internet%History,%Technology,%Security%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

!  Trivial%knowledge%but%not%key%concepts%

!  Revisit%course%objectives%to%identify%threshold%concepts%

%

!  Implausible%distractors%

! Make%sure%none%of%the%distractors%are%obvious%jokes%or%extreme%options%

!  Use%other%formats%besides%multiple%choice%%

How%to%Read%a%Mind%

What%level%on%Bloom?%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

Intro%to%the%Design%of%Everyday%Things%

What%level%on%Bloom?%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

How%to%Read%a%Mind%

What%level%on%Bloom?%

Evaluation*

Synthesis*

Analysis**

Application*

Comprehension*

Knowledge**

!  ‘Select%all%that%apply’%when%the%correct%answer%is%always%all%of%the%options%

!  Vary%the%response%options%%

!  Patterns%that%lead%students%to%the%right%response%without%knowing%anything%

! Watch%out%for%distractors%with%‘all’,%‘never’%etc%! Make%all%options%the%same%length%!  Check%for%subject,verb%agreement%%(if%subject%is%plural%then%verbs%in%all%options%should%be%plural)%

%

! Misalignment%between%objectives,%learning%activity,%and%assessment%criteria%

!  Look%closely%at%learning%objectives%%!  Draw%a%chart%mapping%objectives,%learning%activities,%and%peer%assessment%tasks%to%identify%gaps%

!  Criteria%are%missing%or%unclear%

!  Give%students%clear%and%complete%marking%criteria%for%every%peer%assessment%%

!  Provide%an%example%of%an%excellent%submission%

It%is%harder%to%provide%clear%criteria%when%peer%feedback%is%via%a%discussion%board%instead%of%online%form.%

!  Criteria%are%judgement,based%(‘how%well%did%the%student%do…’)%

!  Break%the%peer%assessment%task%into%several%components,%using%unambiguous%questions%for%each%one.%

!  Knowledge%reproduction%tasks%only%

!  Push%higher%on%Bloom’s%taxonomy%–%have%students%demonstrate%that%they%understand%and%can%apply%key%concepts,%not%just%repeat%them%back%to%you%

!  Solve%problems,%create%something,%analyse%something%

!  To%the%best%of%your%ability,%evaluate*the*student's*site*inventory.%A%student%will%earn%full%marks%for%this%section%if%their%response:%%!  meets%the%required%length%(300,500%words)%!  incorporates%one%or%more%research%filters%!  examines%all%aspects%of%the%site%thoroughly%!  presents%a%clear%and%thoughtful%argument*%!  includes%terminology%and/or%examples%from%the%lecture.%

!  Minor%spelling%and%grammar%errors%should%be%overlooked%here.%Overall,%if%the%student's%answer%makes%sense%to%you,%and%more%or%less%follows%the%guidelines%as%above,%you%should%award%it%full%marks%here.%%

!  *A%rough%guideline%for%a%clearly%organized%argument%would%be%as%follows:%the%site%and%selected%research%filter%would%be%stated%at%the%beginning,%followed%by%appropriate%evidence/facts/observations%about%their%site%that%supports%the%selected%approach,%and%concluding%with%some%explanation%or%analysis%of%why%the%approach%was%chosen.%

0:%The%response%is%blank.%%%1:%The%response%doesn't%engage%any%of%the%research%filters%or%questions,%or%is%too%short.%%2:%The%response%engages%a%research%filter,%but%needs%organization%to%articulate%its%ideas%more%clearly.%%%4:%The%response%is%an%adequate%examination%of%the%site%through%one%of%the%research%filters%above.%%%6:%The%response%is%a%clear%and%thorough%examination%of%the%site%through%one%or%more%of%the%research%filters%above.%%%%8:%The%response%is%a%clear%and%thorough%examination%of%the%site%using%one%or%more%of%the%research%filters%above%AND%includes%terms/examples.%%

GOOD* IMPROVABLE*

! Mathew%Hillier,%Transforming%Assessment%!  Kenji%Lamb,%Soffed%!  Kate%Wright%and%Shelley%Upton,%Information%Services,%Aberystwyth%University%%

Contact%info:%%Mary%Jacob,%mhj@aber.ac.uk%%http://nexus.aber.ac.uk%%

Photo%by%Jen%Collins,%available%on%Flickr%with%Creative%Commons%https://www.flickr.com/photos/jenosaur/4051305996/sizes/o/%%

With thanks from your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[at]rmit.edu.au Dr Mathew Hillier, Teaching and Educational Development Institute, University of Queensland mathew.hillier[at]uq.edu.au