expectation, feedback and performance amongst … thu 1520 b1.pdf · 2016-12-16 · first year...

Australian International Education Conference 2007 www.idp.com/aiec 1

Expectation, Feedback and Performance Expectation, Feedback and Performance Amongst Students Speaking Amongst Students Speaking

Languages Other than English in Languages Other than English in First Year Human Biology Units in First Year Human Biology Units in

Western AustraliaWestern Australia

Kayty Plastow, Jan Meyer, Georgina Fyfe, Kayty Plastow, Jan Meyer, Georgina Fyfe, Sue Fyfe, Mel Sue Fyfe, Mel ZimanZiman

Background Background –– the projectthe project

►► Produce feedback for online tests in first year Human Produce feedback for online tests in first year Human BiologyBiology

►► Three WA universities (mainly Three WA universities (mainly BScBSc students) students) –– A, B, CA, B, C

►► Large (>2000 per year) and diverse student cohortLarge (>2000 per year) and diverse student cohort

►► Importance of feedback Importance of feedback –– meaningful, immediate, meaningful, immediate, individualindividual

►► Benefits of online feedbackBenefits of online feedback


The Project The Project -- to Produce Feedback for Online Tests & to Produce Feedback for Online Tests & Construct an Online Instrument of Reflective PracticeConstruct an Online Instrument of Reflective Practice

Stage 1 Stage 1 –– the studentsthe studentsWho were we writing feedback for?

How did they view feedback?

Experience of Feedback?

What did they do with feedback?

What feedback were they expecting?

15 Focus Groups

Glover (2004)-based Paper Questionnaire

Trialled on level 3 HB students

Reduced, Refined, Administered

AnalysedEnrolment Mode - 98% full-time

LOTE - 22% of students spoke a language other than English at home – 2/3rds exclusively so. 42 different languages other than English

LOTE & Bilingual at Home by Institution

27%

18%

11%0

150

300

450

UniA UniB UniC

Native EnglishLOTE & bilingual

Average Age by Language Status

17.5

18

18.5

19

19.5

20

20.5

Native English LOTE Bilingual

Aver

age

Age

(yea

rs +

/- s.

e)

a a

b ANOVA (Institution as blocks)

F=3.28, df 2 & 1068, p=0.038

Proportion LOTE and Bilingual Speakers by Institution

0

5

10

15

20

25

UniA (70) UniB (84) UniC(79)

Institution (average TER cutoff 2003-2007)

% o

f Res

pond

ents

LOTE

Bilingual

Distribution of Language Groups (%)

African

W European

Indian subcontinent

E European

Aramaic BaseOtherAsian

ChineseEnglish LOTE

Language GroupsLanguage Groups


Work & Language StatusWork & Language Status►► Fewer speaking LOTE at home in paid work (and more Fewer speaking LOTE at home in paid work (and more

declining to answer the question) (22% sample LOTE)declining to answer the question) (22% sample LOTE)

Krause et al 20051999 LOTE 37.5% (12.58 hrs/wk)

Engl 54.9% (12.64 hrs/wk)2004 LOTE 45.9% (11.46 hrs/wk)

Engl 58.1% (12.79 hrs/wk)

Work

0

10

20

30

40

50

60

70

80

No work Work Unanswered

Perc

enta

ge o

f Lan

guag

e G

roup

EnglishLOTE & bilingual

χ2= 85.10, 2 d.f, p < 0.001Answering vs not χ2=31.01, 1 d.f. p < 0.001 Even including not answering as working = 49% LOTE vs 71%

none 1-10hrs 11-20hrs>20hrs

ECU

UWACURTIN

020

40

60

80

100

120

140

160

Over 65% in paid work – of those working49% worked <10hrs/week41% work 11-20 hrs/week

9% (30% at UniA) worked >20hrs/ week

UniC

UniB

UniA

Hours of Paid Work by Language Status

02468

101214161820

ECU UWA CURTINA

v. H

rs W

orke

d pe

r Wee

k

English onlyLOTE onlyBiling with English

UniA UniB UniCF native English vs any LOTE=53.46,df 1&1065, p<0.001

English Av 9.3

LOTE 4.9 hrs

Language Group and ExpectationLanguage Group and Expectation

Distribution of Language Groups (%)

African

W European

Indian subcontinent

E European

Aramaic BaseOtherAsian

ChineseEnglish LOTE

Anticipated Mark by Language Group

60

62

64

66

68

70

72

74

76

78

80

82

Chi

nese

Engl

ish

Oth

er A

sian

Aram

aic

Base

Afric

an

E Eu

rope

an

W E

urop

ean

Indi

ansu

bcon

tinen

t

Ant

icip

ated

Mar

k (%

)

ANOVA (Institutions as blocks & Age as covariate)F=4.65, df 7 & 1060, p<0.001

International students have different International students have different learning experiences and expectations learning experiences and expectations ((RamburthRamburth and McCormick, 2001)and McCormick, 2001)


Factors influencing expectationFactors influencing expectation

Expected Mark by Level of Paid Work

64

65

66

67

68

69

70

71

72

none 1-10hrs 11-20hrs >20hrs

Hours per week

Expe

cted

Mar

k %

ANOVA (age as covariate) F LOTE =11.72, df1 &1066, p<0.001F Institution =14.69, df1 &1066, p<0.001

ANOVA (Institutions as blocks, Age as covariate)F Work =4.6, df 1 & 1036, p= 0.032F Language =6.17, df 1 & 1036, p= 0.013

Anticipated Mark by Age

65

66

67

68

69

70

71

72

73

74

16-18 19-21 >21

Age Category

Ant

icip

ated

Mar

k (%

)

Native English

LOTE or bilingual

Anticipated Course Mark by Work and Language Status

66

67

68

69

70

71

72

73

74

Nonworker Worker

Ant

icip

ated

Mar

k (%

)

LOTE & bilingualNative English

ANOVA (Institutions as blocks) F LOTE =8.36, df1 &1065, p = 0.004

Anticipated Mark by Institution

54

56

58

60

62

64

66

68

70

72

74

UniA (70) UniB (84) UniC(79)

Institution (av TER cutoff 2003-2007)

Ant

icip

ated

Mar

k (%

)Native EnglishLOTE or bilingual

LOTE students:

•Were more optimistic about value of feedback

•Valued most sorts of feedback more

•Valued explanations of correct answers significantly more than non-LOTE students

Attitudes to FeedbackAttitudes to Feedback


The Project The Project -- to Produce Feedback for Online Tests & to Produce Feedback for Online Tests & Construct an Online Instrument of Reflective PracticeConstruct an Online Instrument of Reflective PracticeStage 2 Stage 2 –– The feedbackThe feedback

Who were we writing feedback for?





What sort of feedback was needed?

What sort of errors were being made?

What was the point of our questions?

Wrote 6072 feedback comments for MCQ tests for modules at each institution

Focus Groups


Trialled on level 3 students


Analysed

Examined past MCQ responses with analyses of who made errors, why? discriminating power etc

Administered tests with feedback

Guiding Principles for FeedbackGuiding Principles for Feedback►► Address to Address to ‘‘frameworkframework’’ rather than specific level rather than specific level ––

““whywhy””, not , not ““whatwhat””►► Meaning of key terms in parentheses as Meaning of key terms in parentheses as

encountered, otherwise plain languageencountered, otherwise plain language►► Affirmation for correct responsesAffirmation for correct responses►► Where need to choose incorrect statement reWhere need to choose incorrect statement re--

iterate in affirmation (iterate in affirmation (““Yes! That statement was Yes! That statement was untrue.untrue.””) and as a general comment) and as a general comment

►► Personal voice for correct responses, Personal voice for correct responses, impersonal for pointing out error impersonal for pointing out error

►► Give mental images to remember by,Give mental images to remember by,avoid meaningless mnemonic aidsavoid meaningless mnemonic aids

►► 11--2 sentences2 sentences

Having correct and incorrect. Makes it very easy to misread what the question is asking especially if u glance at the question quickly.(UniB)

Excellent use of brackets within the feedback. (UniA)


The Project The Project -- to Produce Feedback for Online Tests & to Produce Feedback for Online Tests & Construct an Online Instrument of Reflective PracticeConstruct an Online Instrument of Reflective PracticeStage 3 Stage 3 –– The impactThe impact









Wrote 1605 feedback (4467 more)

comments for MCQ tests for one module at each institution

Focus Groups




Analysed



Was the feedback seen as helpful?

What did they learn from it?

Did it improve performance?

Online Reflective Practiceinstrument

Analysed Test/ Retest & Final Examination Marks for Effects of Test with Feedback

Directed to online survey

Test Evaluation

Reflective Practice Reflective Practice

Expectations

(approx 10 options plus “other”)

Problem Areas

Technique

Strategies

About 5 minutes with test evaluation

where do you find out feedback of your test technique? All I viewed was the results and information about why it was wrong/right etc. (UWA)


Response to Test Response to Test

Errors of Test Technique

0

5

10

15

20

25

30

35

40

45

50

[Non

e Give

n]

misrea

d

noprobs

timeman

age

toinc

orrect

leftuna

ns

LOTE & bilingual

Native English

Problem vs No Problem χ2 =1.33, df=1, p=0.248Misread vs Changed to Incorrect χ2 =4.90, df=1,

Helped me to understand questions I got right

0

10

20

30

40

50

60

70

80

90

100

agreement <agreement agreement <agreement

read right didn't read right%

of l

angu

age

x re

adin

g gr

oup

LOTE & bilingual

Native English

Overall χ2 =9.18, df=1, p=0.002Amongst non-readers χ2 =5.11, df=1,

No difference from Native English speakers in judgments that feedback helped them to understand

• what they got wrong (88.7% vs 87.5%, χ2<1, NS)

• other questions in similar area (64.5%, 63.9%, χ2<1, NS

helpfulness of reflective practice (64.5% vs57.7%, χ2<1, NS

Type Read by Language Background

0 10 20 30 40 50 60

scroll

right

w rong

all

Type

Rea

d

Percent of Language Category

Native English

LOTE & Bilingual

Language, Expectation & AchievementLanguage, Expectation & AchievementTest Mark by Language Status

50

52

54

56

58

60

62

LOTE & bilingual Native English

Test

Mar

k (%

)

Test mark

Age Adjusted Test Mark

ANOVA (Institutions as blocks, Age as covariate)F=0.32 NS

Perceived Accuracy of Test as Reflection of Knowledge by Language Status

0

10

20

30

40

50

60

70


% o

f Lan

guag

e G

roup

accurateinaccurate

χ2 =4.82, df 1, p=0.028

Test Score in Relation to Expectations by Language Group

0

10

20

30

40

50

60

70


% o

f Lan

guag

e G

roup

higherexpectedlower

χ2 =14.05, df 2, p<0.001

Expected Mark by Language Status

67

68

69

70

71

72

73


Expe

cted

Mar

k (%

)

ANOVA (Institutions as blocks, Age as covariate)F=8.74, df1 &1068, p=0.003


Test mark and work statusTest mark and work statusTest Mark by Work Status

LOTE

& b

ilingu

al

Nat

ive

Eng

lish

59

60

61

54

56

58

55

57

Age

-Adjus

ted

Test

Mar

k (%

)

worker

SE of Difference of Means

nonworker

ANOVA

Work or not F 7.321, p= .008

Expectation F =3.178, p=.077

Interaction F =5.512, p=.02

workernonworker

Paid Work Status

80

75

70

65

60

55

50

Tes

t Sco

re (

per

cen

t)

LowerHigherExpected

Expectation

Expectation, Test Scores and Paid Work

Test an Accurate Reflection of Knowledge

0

10

20

30

40

50

60

70


% o

f Wor

k-La

ngua

ge G

roup

nonw orker

w orker

InteractionInteraction

high

er th

anex

pect

ed

low

er th

anex

pect

ed

Nat

ive

Eng

lish

wor

ker

LOTE

& b

iling

wor

ker

01020304050607080

Language, Work, Expectation & Performance

Native EnglishworkerLOTE & bilingworker


Test Pace and BackgroundTest Pace and BackgroundTest pace and Performance

UniC S2 2006

01020304050607080

Test

Sco

re%

Test

Rel

ated

MC

Q

Oth

er M

CQ

Test

Rel

ated

Sho

rt A

nsw

ers

Oth

er S

hort

Answ

ers

Test

Sco

re%

Test

Rel

ated

MC

Q

Oth

er M

CQ

Test

Rel

ated

Sho

rt A

nsw

ers

Oth

er S

hort

Answ

ers

Faster Than Designed Slower Than Designed

Ave

rage

Sco

re (%

+/-

SE)

European NameNon-European Name

• Used family name as a proxy for language background, which could only be directly linked to outcomes for survey completers at smallest institution (“European” ≅ “Native English Speaker”)

• In Western Australia most likely error inclusion of native English speakers in “NonEuropean”

•Most non-Commonwealth European immigration 1948-1960s, minority of student exchanges•Most other migration post-1975, O/seas students

Test Pace and Background

0

10

20

30

40

50

60

70

Faster Than Designed Slower Than DesignedTest Pace

Perc

ent o

f Pac

e G

roup

European NameNon-Euro Name

n=148 n=74 n=19 n=29

χ2 = 12.27, 1df, p<.001

European Name 20min 47sec (41.6s/Q)NonEuropean 23min 48sec (47.6s/Q)F=9.8, 1&268, p=.002Those with Non-European names took longerMost “Slower than Design” Non-European names

Test Pace, Background and Exam Outcome

Test Pace>=1min/Q< 1min/Q

50

52

42

46

54

44

48

40

Average M

ark (percen

NonEuropean Name

Av. SE of Difference

European NameAcross Components Interaction term F=9.90, 1&1076df, p=.002

Students with NonStudents with Non--European Names Gain Less European Names Gain Less Advantage from FeedbackAdvantage from Feedback

Feedback Advantage and Population Background

Fail

30

Hig

hDis

t

40

Cre

dit

50

60

Dis

tinct

ion

Pas

s

70

Fee

dbac

k ar

ea M

CQ

mar

k fin

al e

xam

(%)

euro

noneuro

unknownSE of difference of means

Unbalanced ANOVA, Institutions as blocks Fnametype= 4.67, df1&1163, p=0.031

Feedback Area Advantage by Population Background

0

10

20

30

40

50

60

70

80

90

100

NONSTARTER INCOMPLETE COMPLETE

Feedback Test Completion

Feed

back

are

a fin

al M

CQ

/ Non

feed

back

A

rea

(%)

euro

noneuro

Advantage of feedback indicated by value of “completed” cf “nonstarter”

Significantly greater for those with European-derived surnames


SummarySummary

►► More likely to have lower marks than expected and to have difficMore likely to have lower marks than expected and to have difficulty ulty accepting lower marks as accurate reflections of their knowledgeaccepting lower marks as accurate reflections of their knowledge

NESB students more likely to receive marks lower than expected, NESB students more likely to receive marks lower than expected, to find to find workload too heavy and to fear failing. Less likely to have expeworkload too heavy and to fear failing. Less likely to have expectations ctations regarding university met (Krause et al., 2005)regarding university met (Krause et al., 2005)

►► Significant numbers of LOTE students are in paid work, which hasSignificant numbers of LOTE students are in paid work, which hasgreater impact upon their performancegreater impact upon their performance

International students more problems than domestic International students more problems than domestic LOTEsLOTEs (Krause et al., (Krause et al., 2005)2005)

►► Greater expectations of online resources and greater appreciatioGreater expectations of online resources and greater appreciationn-- disproportionate to what is delivered?disproportionate to what is delivered?

Greater reliance on online resources (Krause et al., 2005)Greater reliance on online resources (Krause et al., 2005)

ImplicationsImplications

►►LOTE students as a group have higher LOTE students as a group have higher expectations and are more likely to be expectations and are more likely to be disappointed in their marksdisappointed in their marks

Help with development realistic expectationsHelp with development realistic expectationsLOTE students in particular would benefit from LOTE students in particular would benefit from early assessments and feedbackearly assessments and feedbackMultiple assessment tasks (with feedback) Multiple assessment tasks (with feedback) important in first semesterimportant in first semesterChances for detection of need for intervention Chances for detection of need for intervention in pattern of use of online resourcesin pattern of use of online resources


AcknowledgementsAcknowledgements

►► Leith Sly, IT support Curtin UniversityLeith Sly, IT support Curtin University

►► Support for this study has been provided by The Support for this study has been provided by The Carrick Institute for Learning and Teaching in Carrick Institute for Learning and Teaching in Higher Education Ltd, an initiative of the Higher Education Ltd, an initiative of the Australian Government Department of Education, Australian Government Department of Education, Science and Training. Science and Training.

►► The views expressed in this presentation do not The views expressed in this presentation do not necessarily reflect the views of The Carrick necessarily reflect the views of The Carrick Institute for Learning and Teaching in Higher Institute for Learning and Teaching in Higher Education.Education.

The Project The Project -- to Produce Feedback for Online Tests & to Produce Feedback for Online Tests & Construct an Online Instrument of Reflective PracticeConstruct an Online Instrument of Reflective PracticeStage 1 – The students and feedback






15 Focus Groups


Trialled on level 3 HB students


Analysed

Personal, Authoritative[most dislike] “general feedback that doesn't directly co-incide with me it may not specifically apply to me and therefore may be useless” (CURTIN) “written feedback from your assessor” (CURTIN)

Positive[most dislike] “comments that make me feel stupid makes me don't wanna learn” (UWA)

Explanatory[most dislike] “just give the correct answer without any explanation or say nothing at all because I do not understand at all even though I have found the correct answer myself”(CURTIN)

Pertinent, Digestible“huge comments that don't directly relate to learning objective doesn't help learning”(CURTIN)

Only 30% thought they “always” or “mostly” received enough feedback

Rated feedback more useful with increased

•age •feedback experience•expected performance

(LOTE)(less employment)


The Project The Project -- to Produce Feedback for Online Tests & to Produce Feedback for Online Tests & Construct an Online Instrument of Reflective PracticeConstruct an Online Instrument of Reflective PracticeStage 2 Stage 2 –– The feedbackThe feedback









Wrote 1605 feedback (4467 more)

comments for MCQ tests for one module at each institution

Focus Groups




Analysed



Found

• the clutter of ages – cleaned up obvious duplicates & problems

• many more terminology-based questions than we thought we had

• fairly frequent ‘misreading’ errors

•most obviously correct option in a “choose the incorrect”question)

•“hypo”/ “hyper” confusions

Problem Area Problem Area --LanguageLanguage

hard to find the correct statement among all the jargon (UWA)

I would often scan through the options quickly without having an idea of the kind of answer I was looking for, thus leaving me confused about the options, especially if there's minor variations in wording. (UWA)

it used words i was unfamilar with and it confussed me and i lost cncentration all togethr(CURTIN)

10.5% of all students nominated “terminology”as a problem area highlighted by the test (commonest response)

We were told that the information we needed to know was in the lec notes, yet the way the questions were worded did not reflect this. (CURTIN)

Hard to remember some names and functions etc. (CURTIN)

just that some multiple choice answers were similar, it kinda confused me. therefore, i took a guess of the two (CURTIN)

Having different names for differrentparts is also confusing (ECU)

get put off reading long questions, lose sight of what their

trying to say (UWA)

I have trouble understanding the terminology. Need to study this more. (CURTIN)


Initial QuestionnaireInitial Questionnaire►► DemographicsDemographics

Age, sex etcAge, sex etcPaid Work (hrs)Paid Work (hrs)LanguageLanguageAnticipated markAnticipated mark

►► Satisfaction with Satisfaction with past feedbackpast feedback

AmountAmountTimingTiming

►► Use of FeedbackUse of Feedback►► Anticipated Anticipated

FeedbackFeedback►► Helpfulness of Helpfulness of

feedback feedback

12 types of feedback

“Helpfulness of Feedback” section

Whole Questionnaire took about 15 minutes to complete, 1099 responses

Tests with FeedbackTests with Feedback

►► UniAUniA50 questions50 questions

No variationNo variationBlackboardBlackboard

►► UniBUniB & & UniCUniC30 questions30 questionsEach chosen Each chosen randomly from randomly from topic sets of topic sets of 5 5 --15 questions15 questionsWebCTWebCT

Which of the following statements is CORRECT?

Designed to take about 1 minute per question


Student Evaluation of FeedbackStudent Evaluation of Feedback

Helped Understand Wrong Answers

0 10 20 30 40 50 60

Strongly Disagree

Disagree

neutral

Agree

Strongly Agree

Agre

emen

t

Percent of Responses

Apply to Other Human Biology Modules

0 10 20 30 40 50 60

Strongly Disagree

Disagree

neutral

Agree

Strongly Agree

Agr

eem

ent


Length of Feedback Messages

0 10 20 30 40 50 60

Too Long

Too Short

Full

Brief

Just Right

Agr

eem

ent


They are a great help! Thanks for the time & effort! Definitely it would be great to have access to more similar feedback in the future, it is a fantastic study tool (UniB)the feedback comments really help me alot. all other biology units should have this system to help students to understand better.(UniC)

Found them helpful for the questions I got wrong as I could find out right there and then how I went wrong. Before, I had to try to remember what the question was to then be able to look it up in the book. Please keep them for next year. (UniC)

they helped a lot ,as the next time I did the test I got 48/50!! (UniA)

workernonworker

Workornot!

60

40

20

0

Cou

nt

Bar Chart

innaccurateaccurate

AccurateYN!

Nonworker Worker

20

0

80

40

100

60

Count

AccurateInaccurate

Whole Sample

workernonworker

Paid Work Status

64

62

60

58

56

54

52

50

Test

Sco

re (p

erce

nt)

InnaccurateAccurate

Accurate Evaluation?

Expectation, Test Performance and Paid Work

ANOVAworkF=7.321, 1&127df, p=.008

ANOVAexpectF=7.321, 1&127df, p=.077

ANOVAintF=7.5.572, 1&127df, p=.020

Perceived Accuracy of Perceived Accuracy of Assessment of UnderstandingAssessment of Understanding

►► Workers more likely to feel that their test results do not accurWorkers more likely to feel that their test results do not accurately ately reflect their understandingreflect their understanding

►► Those with lower marks more likely to consider inaccurateThose with lower marks more likely to consider inaccurate►► Workers who accept accuracy have lower scores than Workers who accept accuracy have lower scores than nonworkersnonworkers

UniC - Only one under test conditions

χ2 =4.833, 1df, p=.028


Feedback does not compensate for Feedback does not compensate for disadvantage of disadvantage of ““languagelanguage”” backgroundbackground

Fail

Pas

sC

redi

tD

istin

ctio

nH

ighD

ist

none

uro

euro

FU

LL T

ES

T

none

uro

euro

PA

RT

TES

T

non

euro

euro

NO

TE

ST

0102030405060708090

100

Fina

l Exa

m M

CQ

%

Final Grade

Feedback Area Performance Final Exam

Unbalanced ANOVA (institutions as blocks) Fnames = 10.41, df=1&1139, p<0.001; Ftests = 5.511, df=2&1139, p=0.004; Fnames.grades.test =2.90, df8&1139, p= 0.003

expectation, feedback and performance amongst … thu 1520 b1.pdf · 2016-12-16 · first year...

Documents