Assessing IFP students’ needs assessment

Florencia Franceschina

Manchester Metropolitan University

F.Franceschina@mmu.ac.uk

Inform Conference,

University of Reading, 17 July 2010

1. Introduction

• Increased weight of the ‘student voice’ in HE (e.g., NSS, International Student Barometer, etc.)

• The ‘student voice’ is valued in different ways:– Students as a customers– Students as individuals capable of self-directed

learning

• Learning needs self-assessment relevant to either view

Defining self-assessment

“Self-assessment refers to the involvement of learners in making judgements about their own learning […]”.

Boud and Falchikov (1989: 529)

Why assess learning needs?

Because it can help us to identify:

• learners’ strengths and weaknesses

• learners’ goals and expectations

Issue #1: How useful is

learning needs self-assessment?

• Useful for policy makers

• Useful for curriculum developers

• Useful for teachers

• Useful for students

“Whether it be a junior high school student wondering whether to crack open his social studies textbook one more time before the test or a medical student deciding whether to practice her intubating technique before her next shift, students make more effective decisions about where to apply their learning efforts when they can actually discern their strengths and weaknesses.”

(Dunning et al., 2004: 85)

Issue #2: How valid is

learning needs (self-)assessment?

Do students know what they need?

Do even experts know which needs we should explore and aim to address? For example, the relevance of some needs may not be apparent at the time of assessment (Grant, 2002: 157).

Issue #3: How reliable is

learning needs self-assessment?

• Are all learners equally able to make accurate judgements about their learning needs?

• How good are IFP students at it?

2. Brief review of the literature

Self-assessment …

• … is difficult.

Meta-studies reveal typically low correlation between self-views and actual performance:

.21 (Hansford and Hattie, 1982)

.39 (Falchikov and Boud, 1989)

Similar findings by Dunning et al., 2003; Dunning et al., 2004; Powers, 2002; Ross, 1998.

Self-assessment …• … is difficult.• … is not always valued by learners

(e.g., compared to summative assessment)

Self-assessment …• … is difficult.• … is not always valued by learners

(e.g., compared to summative assessment)

• … is modulated by culture.

Self-regulation is subject to the influence of educational/cultural background factors

(e.g., Purdie and Hattie, 1996; Olaussen and Bråten, 1999)

Self-assessment …• … is difficult.• … is not always valued by learners

(e.g., compared to summative assessment)

• … is modulated by culture.• … is modulated by level of study.

Students in advanced courses tend to be more accurate self-assessors than students on introductory courses (Falchikov and Boud, 1989).

Self-assessment …• … is difficult.• … is not always valued by learners (e.g.,

compared to summative assessment)• … is modulated by culture.• … is modulated by level of study.• … is modulated by academic ability level.

“High-performing students were accurate, with accuracy improving over multiple exams. Low-performing students showed moderate prediction accuracy but good postdiction accuracy. Lowest performing students showed gross overconfidence in predictions and postdictions. Judgments of performance were influenced by prior judgments and not prior performance. Performance and judgments of performance had little influence on subsequent test preparation behavior.”

Hacker et al. (2000: 160)

Self-assessment …• … is difficult.• … is not always valued by learners (e.g.,

compared to summative assessment)• … is modulated by culture.• … is modulated by level of study.• … is modulated by academic ability level.• … is differently accurate depending on area

of study.

Students in science tend to be more accurate self-assessors than students in other areas (Falchikov and Boud, 1989).

Self-assessment …• … is difficult.• … is not always valued by learners (e.g.,

compared to summative assessment)• … is modulated by culture.• … is modulated by level of study.• … is modulated by academic ability level.• … is differently accurate depending on area of

study.• … can be developed as a skill.

Although student self-assessment remains inaccurate after training, it does seem to be subject to some improvement. For example:

McDonald (2010) has shown that involving students in meta-assessment training can help mathematics students to improve their test performance.

Chen (2008) found that through feedback and practice students’ self-assessments of English speaking skills become more aligned with tutor assessment.

Wide range of data collection methods available

• Formal / informal

• Planned / opportunistic

• Individual / group

• By self / peer / tutor / outside expert

Some commonly used tools

• Questionnaires• Focus groups• Online discussion fora• Staff-student liaison committees• Diaries• Objective tests• Observation• Action research

3. Methodology

• Cohort of 51 IFP students

• Min. language entry requirement: IELTS 5.5 or equivalent

• Followed up for the duration of IFP (2009-2010)

Mixed nationalities

• Saudi Arabia (22%)• Bahrain (14%)• Nigeria (14%)• Pakistan (8%)• Turkey (8%)• UAE (8%)• China (4%)• India (4%)

• Iran (4%)• Sri Lanka (4%)• DR Congo (2%)• Hong Kong (2%)• Kenya (2%)• Turkish Cyprus (2%)• USA (2%)• Zimbabwe (2%)

Mixed linked degrees

Science / Engineering

53%Business31%

HLSS14%

Other2%

Data collected

• Learning needs questionnaire– Student version (individual self-assessment)– Staff version (group assessment)

• Diagnostic language test• Attendance records• Coursework and exam results (all IFP

modules)• Staff interviews• End-of-year module feedback (EAP module)

Analyses

• Perceived needs vs. observed academic performance

• “Objective” diagnostic measures vs. observed academic performance

4. Findings

Overview

1. “Objective” measures

2. “Subjective” measures

3. Improvements in judgements?

Correlations between ‘objective’ measures

Lang test at entry

EAP unit mark

IFP mark

EAP attend.

IFP attend.

Lang test at entry

- .22 .36

EAP unit mark

- - .81 .67

IFP

mark

- - - .65

EAP attend.

- - - -

IFP attend.

- - - - -

Correlation between language test at entry and EAP unit marks

0

10

20

30

40

50

60

70

80

0 20 40 60 80 100 120 140 160 180

OPT scores at entry

EAP unit marks

Correlation between language test score at entry and overall IFP mark

0102030405060708090

100

0 20 40 60 80 100 120 140 160 180

OPT score at etnry

Overall FY mark

Correlations between ‘objective’ measures

Lang test at entry

EAP unit mark

IFP mark

EAP attend.

IFP attend.

Lang test at entry

- .22 .36

EAP unit mark

- - .81 .67

IFP

mark

- - - .65

EAP attend.

- - - -

IFP attend.

- - - - -

Correlation between EAP unit mark and IFP overall mark

0

10

20

30

40

50

60

70

80

0 10 20 30 40 50 60 70 80 90

EAP unit mark

IFP overal mark

Correlations between ‘objective’ measures

Lang test at entry

EAP unit mark

IFP mark

EAP attend.

IFP attend.

Lang test at entry

- .22 .36

EAP unit mark

- - .81 .67

IFP

mark

- - - .65

EAP attend.

- - - -

IFP attend.

- - - - -

Correlation between EAP attendance and EAP mark

0102030405060708090

100

0 10 20 30 40 50 60 70 80

EAP mark

EAP attendance

Correlation between IFP attendance and overall IFP mark

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90

IFP overall mark

Attendance %

How did “objective” measures work?

• Language test at entry was poor predictor or later performance. This could be due to a number of reasons. (Similarly surprising patterns reported for UCAS tariff points/FY performance elsewhere.)

• Attendance was reasonably good predictor. This confirms findings in previous studies (e.g., Hughes, 2009).

• EAP unit was best predictor of IFP performance.

Staff perceptions about students’ needs at entry

Tutors most frequently identified general language weakness as an issue to be addressed.

“when marking essays, it is obvious that many students have a very weak grasp of syntax sometimes, and this will hold them back in the future.”

“Sometimes there have been some difficulties in their ability to express themselves effectively in written work but this is not always the case. The main problem I have found with international students who do have language difficulties is their lack of commitment in addressing the issue or them denying the fact there is a problem despite overwhelming evidence and continuous advice to attend support classes. Some students believe their English skills are better than they truly are.”

“Required much repetition on basic tasks appear not to understand instructions.”

“there are others who have a very poor command of the English language and struggle with their research and assignments.”

Tutors and support staff thought the IFP students’ numeracy skills and subject background knowledge was on the whole the same or better than in UK students.

Relative difficulty of IFP modules (MMU ‘units’)

Mean unit marks per unit type

0

10

20

30

40

50

60

70

L-type units X-type units N-type units

Mean unit marks

Spread of unit marks per unit type

0102030405060708090

100

1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113

Individual unit marks

L-type unit marks X-type unit marks N-type unit marks

Students’ perceptions about their needs at entry

Indirect method of calculating match between students’ perceived needs and actual performance:

1. Self-assessment questionnaire answers coded as 1 if student ticked ‘I need to study more of this’/’I need help with this’, and 0 if student ticked ‘I don’t want much of this’/’I don’t need help with this’

2. EAP unit summative assessment marks were coded as 0 if mark was above mean and 1 if below mean.

3. Match between 1 and 2 above was checked.

Maximum possible score for successful matching of needs and actual performance: 5

(W, R, S, L, IT)

IFP students’ average score: 2.5

Perceived vs observed order of difficulty of subskills

• Predicted by students:W < IT < L < S/R

• Observed:W (38%) < R (43%) < S (50%) < L (52%) < IT (80%)

Correlation between correct prediction index for subskills and actual IFP overall marks

0

1

2

3

4

5

0 10 20 30 40 50 60 70 80 90

IFP overall mark

Correct prediction index

Students’ perceptions about their needs at exit

Needs questionnaire (at exit) answers match performance in subskills at an average score of 2.3 (no improvement)

End-of-year student feedback on EAP unit

0

1

2

3

4

5

6

7

8

9

10

Q1a. vocabulary quizzes

Q1b. glossaries and essay planning notebook

Q1c. vocabulary notebook

Q1d. textbook

Q1e. IT (orange) bookletQ1f. materials on WebCT

Q1g. speaking practice and assessmentQ1h. listening practice and assessmentQ1i. reading practice and assessmentQ1j. writing practice and assessment

Q1k. IT practice and assessment

Question

Average score

How useful?

How enjoyable?

Learning need as judged by students at the end of the IFP

Higher Lower

Speaking Listening

Reading

Writing

IT

EAP unit performance

Assessment area

Mean scores

SD Median

IT 80 27 87

Listening 52 14 53

Speaking 50 15 52

Reading 43 16 47

Writing 38 21 40

5. Summary and Conclusion

• Not all objective measures that seem relevant make useful predictions.

• Tutors and support staff’s global, subjective assessment of students’ learning needs seems to be accurate.

• These IFP students do not seem to be very good at assessing their learning needs in detail, but they are relatively better when their judgements are considered at a more general, group level.

• Given how unreliable IFP students’ judgements seem to be, we should be careful when using these judgements to inform our practice.

• The highest value of learning needs self-assessment is probably for the students themselves.

6. Practical suggestions

• Bear in mind that learning needs self-assessment is difficult, especially for IFP students.

• When interpreting IFP students’ needs self-assessment, remember that it can be influenced by:– cultural factors – academic ability level (good vs. weak students)– area of study (science vs. other)– assessment context (formative/summative; high/low

stakes)

• Students become more accurate with training and practice, so provide them with opportunities to develop this skill.

• Close the feedback loop with staff and students, for example through:– As part of regular assessment feedback– Websites, VLEs, notice boards– Programme committee meetings– Informal reporting by staff in class

• Whenever possible, combine self-assessment with other sources of information about your students’ learning needs.

• Take advantage of the wide range of data collection tools available.

• Methodological tip: ask students to make self-assessment judgements using same scales as ‘objective’ performance measures you will use to compare. (This has its own problems, though.)

ReferencesBoud, David, and Falchikov, Nancy. 1989. Quantitative studies of student self-assessment in higher

education: a critical analysis of findings. Higher Education 18:529-549. Chen, Yuh-Mei. 2008. Learning to self-assess oral performance in English: a longitudinal case study.

Language Teaching Research 12:235-262. Dunning, David, Heath, Chip, and Suls, Jerry M. 2004. Flawed self-assessment. Implications for health,

education and the workplace. Psychological Science in the Public Interest 5:31-54. Dunning, David, Johnson, Kerri, Ehrlinger, Joyce, and Kruger, Justin. 2003. Why people fail to recognize

their own incompetence. Current Directions in Psychological Science 12:83-87.Falchikov, Nancy, and Boud, David. 1989. Student self-assessment in higher education: a meta-analysis.

Review of Educational Research 59:395-430.Grant, Janet. 2002. Learning needs assessment: assessing the need. British Medical Journal 324:156-

159. Hacker, Douglas J, Bol, Linda, Horgan, Dianne D, and Rakow, Ernest A. 2000. Test prediction and

performance in a classroom context. Journal of Educational Psychology 92:160-170.Hughes, Nicola T. 2009. Attendance as a measure of student motivation and engagement. InForm 3:7-8.McDonald, Betty. 2010. Improving learning through meta assessment. Active Learning in Higher

Education 11:119-129. Olaussen, Bodil S, and Br 蚯 en, Ivar. 1999. Students' use of strategies for self-regulated learning: cross-

cultural perspectives. Scandinavian Journal of Educational Research 43:409-432.Powers, Donald E. 2002. Self-assessment of reasoning skills. ETS Research Report RR-02-22.

Princeton, NJ: Educational Testing Service (ETS). Purdie, Nola, and Hattie, John. 1996. Cultural differences in the use of strategies for self-regulated

learning. American Educational Research Journal 33:845-871. Ross, Stephen John. 1998. Self-assessment in second language testing: a meta-analysis and analysis of experiential factors. Language Testing 15:1-20.