USING LEARNER ANALYTICS

TO SUPPORT THE ACADEMIC WRITING

IN HIGHER EDUCATION

DUYGU SIMSEK

Academic and Professional Literacies Forum, The Open University, UK 07th May, 2014

people.kmi.open.ac.uk/simsek

duygu.simsek@open.ac.uk

simsekduygu_

Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo

Dr. Ágnes Sándor, Xerox Research Centre Europe

ABOUT ME

Ankara, Turkey

BA&MA in Computer and Instructional Technologies

Bilkent University, Turkey

MSc. Software Engineering

University of Southampton, UK

2nd year PhD Research Student KMi

The Open University, UK

Duygu Simsek, APL Forum Talk 2

OUTLINE –

Using Learner Analytics to Support

the Academic Writing in Higher Education

Research Aim

Where this research sits?

Academic Writing

Learning Analytics

Computational Text Analysis

Research Questions

Research Methods

Duygu Simsek, APL Forum Talk 3

RESEARCH AIM

To investigate

whether computational techniques can automatically identify

the attributes of good academic writing in as correlated with

grades of the essay and as identified in the literature

if this proves possible, how best to feed back actionable analytics to support students and educators

whether this feedback has any demonstrable benefits

Duygu Simsek, APL Forum Talk 4

WHERE THIS RESEARCH SITS?

Duygu Simsek, APL Forum Talk 5

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

WHERE THIS RESEARCH SITS?-

ACADEMIC WRITING

Key aim of academic

writing is to convince

readers about the validity

of the claims and

arguments put forward through an effective

narrative.

Duygu Simsek, APL Forum Talk 6

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

WHERE THIS RESEARCH SITS?-

META-DISCOURSE

This effective narrative is

signalled through meta-

discourse!

Duygu Simsek, APL Forum Talk 7

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

META-DISCOURSE

Meta-discourse refers to the features of text that convey the author’s intended

meaning and intention. It provides cues to the reader which explicitly express a

viewpoint, argument and claim, and signals the writer's stance.

Duygu Simsek, APL Forum Talk 8

Fig. 1 Meta-discourse that convey summary statements

Cues t

o S

um

ma

ry

sta

tem

ents

EXAMPLES OF META-DISCOURSE CUES THAT

SIGNAL ACADEMIC/ANALYTICAL RHETORICAL MOVES

BACKGROUND KNOWLEDGE:

Recent studies indicate …

the previously proposed …

… is universally accepted

Duygu Simsek, APL Forum Talk 9

NOVELTY:

New insights provide direct evidence… …suggest a new approach… Results define a novel role ...

OPEN QUESTION:

Little is known …

… role … has been elusive

Current data is insufficient…

TENDENCY:

... emerging as a promising approach

Our understanding ... has grown exponentially ...

Growing recognition of the

importance ...

CONTRASTING IDEAS:

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE:

studies ... have provided important advances

... is crucial for ... understanding

valuable information ... from

SURPRISE:

We have recently observed ... surprisingly

We have identified ... unusual

The recent discovery ... suggests intriguing roles

SUMMARISING:

The goal of this study ...

Here, we show ...

Our results ... indicate

WHERE THIS RESEARCH SITS?-

META-DISCOURSE

In order to assess students’ writing therefore, educators will be examining students’ use of meta-discourse which make their students’ thinking visible.

However, students find it challenging to learn to write in an academically sound way.

They need to learn how to make their thinking visible by recognising and deploying meta-discourse.

Duygu Simsek, APL Forum Talk 10

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

WHERE THIS RESEARCH SITS?-

COMPUTATIONAL TEXT ANALYSIS

Meta-discourse cues

are automatically

identifiable.

This PhD investigates

whether it is possible to

provide automatic

meta-discourse analysis of student writing through the use of a

particular rhetorical

parser, XIP. Duygu Simsek, APL Forum Talk 1

1

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

(XIP)

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

EXAMPLE OF A RHETORICAL PARSER: INCREMENTAL PARSER (XIP)

Natural Language Processing (NLP)* product which includes a

rhetorical parser detecting meta-discourse in academic texts.

XIP extracts salient sentences based on their rhetorical functions:

Background Knowledge

Summarising

Tendency

Novelty

Significance

Surprise

Open Question

Contrasting Ideas

Duygu Simsek, APL Forum Talk 12 *language processing by computers that enable computers to derive meaning from natural language input

Student Writing Analysed by XIP

Duygu Simsek, APL Forum Talk 13

CONTRAST

SUMMARY

RHETORICAL FUNCTIONS CLASSIFIED BY XIP

BACKGROUND KNOWLEDGE:

Recent studies indicate …

the previously proposed …

… is universally accepted

Duygu Simsek, APL Forum Talk 14

NOVELTY:

New insights provide direct evidence… …suggest a new approach… Results define a novel role ...

OPEN QUESTION:

Little is known …

… role … has been elusive

Current data is insufficient…

TENDENCY:

... emerging as a promising approach

Our understanding ... has grown exponentially ...

Growing recognition of the

importance ...

CONTRASTING IDEAS:

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE:

studies ... have provided important advances

... is crucial for ... understanding

valuable information ... from

SURPRISE:

We have recently observed ... surprisingly

We have identified ... unusual

The recent discovery ... suggests intriguing roles

SUMMARISING:

The goal of this study ...

Here, we show ...

Our results ... indicate

Fine for

researchers or

machines but it is

not

learner/educator

friendly

XIP’s Output

Duygu Simsek, APL Forum Talk 15

WHY XIP? –

KEY FEATURES OF ACADEMIC WRITING?

Duygu Simsek, APL Forum Talk 16

Relevance

Understanding & Knowledge

Structure & Organisation

Linguistic Accuracy

Illustrations

Referencing

Argumentation

WHY XIP?

There is a mapping between good and strong features of academic

writing and the XIP’s rhetorical functions.

Duygu Simsek, APL Forum Talk 17

WHERE THIS RESEARCH SITS?-

Learning Analytics

XIP is a parser with potential, if it can be

embedded in a more

complete learning

analytics (LA) approach.

Duygu Simsek, APL Forum Talk 18

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

(XIP)

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

WHERE THIS RESEARCH SITS?-

Learning Analytics (LA)

Learning moves online (i.e. pure online courses –OU, MOOCs –FutureLearn, Coursera)

Digital learners leave data trails in online activities

Learner-produced online student data (i.e. data from Learning Management Systems)

Collecting traces left behind to improve learning

LA is “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”.[1]

Duygu Simsek, APL Forum Talk 19 [1] Call for Papers of the 1st International Conference on Learning Analytics & Knowledge (LAK 2011). Retrieved 12 February 2014.

WHERE THIS RESEARCH SITS?-

Learning Analytics (LA)

LA stakeholders can be educators, learners and administrators.

educators can learn about student activities and progress

learners can get feedback about their own progress,

understand their own learning habits and clearly see the

impact of the activities helping them more

administrators can make departmental and institutional

level adjustments based on the data collected and

analysed

The most common use of LA is to spot learners who appear to

fail and to make interventions in order to help those students.

Duygu Simsek, APL Forum Talk 20

WHERE THIS RESEARCH SITS?-

Discourse-centric Learning Analytics

How should a DCLA

approach be validated?

Duygu Simsek, APL Forum Talk 21

ACADEMIC

WRITING

LEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

(XIP)

Discourse

Centric

Learning

Analytics

(DCLA)

Meta-

discourse

in Student

writing

MAIN RESEARCH QUESTION

Duygu Simsek, APL Forum Talk 22

To what degree can computational text analysis

and visual analytics be used to

support the academic writing of

students in higher education?

TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE

AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD

ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY

THE GRADE, AND BY EDUCATORS?

Duygu Simsek, APL Forum Talk 23

XIP

Evaluates Accuracy & Sufficiency

Any correlation between Grades &

XIP output?

XIP’s Highlights vs. Marker’s

RQ1

TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE

AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD

ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY

THE GRADE, AND BY EDUCATORS?

Duygu Simsek, APL Forum Talk 24

RQ1

XIP Highlighted Student Writing

Any correlation between

the final grade of writing & XIP findings?

Pearson for

Total number of salient sentences vs. Grade

Generalised Multiple Regression

How strongly each rhetorical sentence type influences the final grade

Grades

The OU’s S288 Practical Science

1st year undergraduates

5 strands:

Physics and Astronomy

Chemistry and Analysis

Environmental Sciences

Earth and Environment

Biology and Health

Collaborative scientific writing assignment

Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh

July 5, 2013

DATA

Duygu Simsek, APL Forum Talk 25

~300 1st year undergraduate students in 2012 (33 reports)

~600 in 2013 (69 reports)

Not a meaningful correlation between total number of

sentences found and grades.

Weak negative correlation between summary & grades.

Pearson: -0.28

Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh

July 5, 2013

PRELIMINARY RESULTS

Duygu Simsek, APL Forum Talk 26

TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE

AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD

ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY

THE GRADE, AND BY EDUCATORS?

Duygu Simsek, APL Forum Talk 27

RQ1

What is the overlap between XIP’s output and

how tutors judge quality?

Tutor Highlighted Student Writing XIP Highlighted Student Writing

IN WHAT WAYS SHOULD XIP OUTPUT BE DELIVERED TO END

USERS (STUDENTS AND EDUCATORS)?

Duygu Simsek, APL Forum Talk 28

XIP

Evaluates Accuracy & Sufficiency

Any correlation between Grades &

XIP output?

XIP’s Highlights vs. Marker’s

Output

RQ2

IN WHAT WAYS SHOULD XIP OUTPUT BE DELIVERED TO END

USERS (STUDENTS AND EDUCATORS)?

1st Year Pilot study

Duygu Simsek, APL Forum Talk 29

RQ2

1 min. Intro. Video: http://goo.gl/WGIkDs 5 mins. Demo: http://goo.gl/Km1di5

TO WHAT EXTENT DO EDUCATORS VALUE THE RESULTS OF XIP’S ANALYSIS OF AN INDIVIDUAL STUDENT OR COHORT’S WORK

WHEN THE PRIMARY FOCUS IS ON ASSESSMENT?

Duygu Simsek, APL Forum Talk 30

XIP

Evaluates Accuracy & Sufficiency

Any correlation between Grades &

XIP output?

XIP’s Highlights vs. Marker’s

Output

What educators think

RQ3

TO WHAT EXTENT DO EDUCATORS VALUE THE RESULTS OF XIP’S ANALYSIS OF AN INDIVIDUAL STUDENT OR COHORT’S WORK

WHEN THE PRIMARY FOCUS IS ON ASSESSMENT?

Duygu Simsek, APL Forum Talk 31

XIP

Evaluates Accuracy & Sufficiency

Any correlation between Grades &

XIP output?

XIP’s Highlights vs. Marker’s

Output

What educators think

RQ3

TO WHAT EXTENT DO STUDENTS VALUE THE RESULTS OF XIP’S ANALYSIS AS FORMATIVE FEEDBACK ON THEIR WRITING?

Duygu Simsek, APL Forum Talk 32

XIP

Evaluates Accuracy & Sufficiency

Any correlation between Grades &

XIP output?

XIP’s Highlights vs. Marker’s

Output

What educators think

What students think

RQ4

YOUR VIEW ON…

1. Your view on the distilled summary of key features of academic

writing & argumentation.

2. Any graded student writing you have or any OU module that you

can suggest me to contact for data collection?

3. Any feedback on my studies with ALs?

Duygu Simsek, APL Forum Talk 33