Introducing  Analy0cs  at  UTS  Example:  Academic  Wri0ng  Analy0cs  

Simon Buckingham Shum Director, Connected Intelligence Centre Professor of Learning Informatics @sbuckshum #LearningAnalytics

utscic.edu.au

Stakeholders to build writing analytics tool

2

Ágnes Sándor Xerox Research Centre Europe

Grenoble (linguist: modelling

academic writing)

Xerox Incremental

Parser

Rosalie Goldsmith Institute for Multimedia Learning (researcher in reflective writing in engineering)

Philippa Ryan Faculty of Law (academic teaching civil law)

Reflective writing rubrics

Civil Law student writing

Simon Buckingham Shum Shawn Wang, Simon Knight

Connected Intelligence Centre (learning analytics researchers and web developer)

User requirements, interface design and evaluation, data collection & analysis

AWA: Analytical Academic Writing

3

Roll over sentences with Fkeys for a popup reminding you of their meaning

Analytical academic writing (Civil Law)

4

Engaging Law academic in tuning AWA

5

1.  Trigger  curiosity  through  CIC  briefings  

2.  Submit  collabora0on  proposal  to  CIC  

3.  Agree  0melines  

4.  Think  aloud  user  interface  walkthrough  

5.  Preliminary  text  analyses  

6.  Systema0c  text  analyses  

7.  Refine  parser  

8.  Design  student  pilot  

9.  Gather  student  feedback  

6

From informal rubric to formal rhetorical pattern

AWA: reflective academic writing

7

Validating the tool: metrics

8

human highlighting automated highlighting

Structured annotation from the civil law academic to tune the parser for her discipline

9

Validating the tool: student feedback

10

“takes  the  emo+on  out  of  having  your  work  scru+nized”  

“it  was  not  embarrassing  in  the  way  that  it  can  be  when  a  tutor  or  marker  gives  feedback”  

“I  definitely  found  it  useful.  It  also  made  me  realise  that  I  tend  to  use  bold,  certain  language  in  making  my  point  towards  the  end  of  each  paragraph  rather  than  up  front  at  the  beginning  (when  introducing  my  point).”  

“I  realise  now  what  descrip+ve  wri+ng  is  -­‐  the  soCware  had  quite  a  bit  to  say  about  my  lack  of  jus+fica+on  -­‐  also  true  -­‐  pressed  for  +me  and  difficult  circumstances  have  caused  this  for  me  in  this  instance  -­‐  good  to  see  it  sampled.”  

 

Validating the tool: academic trust

11

An issue of “Algorithmic Accountability” Critical to forge trust between all the stakeholders: Educators, Analysts, Students

We aim to ensure that academics feel that:

their expertise is respected

their feedback has an impact

they understand the tool sufficiently to trust it

ideally they get esteem and research outputs from engaging with analytics