The validity of ICT The validity of ICT assessment assessment

at 16: the student perceptionat 16: the student perception

A PhD enquiryA PhD enquiry

Pete BradshawPete Bradshaw

pete.bradshaw@ntu.ac.ukpete.bradshaw@ntu.ac.uk

NTU School of EducationNTU School of Education

Research Seminar 6/11/08Research Seminar 6/11/08

Starting pointsStarting points

► What do you think of What do you think of this? this?

► Marks out of 100?Marks out of 100?► Grade A*-G?Grade A*-G?► Level?Level?

► What other information What other information would we need?would we need?

► Is it ICT anyway?Is it ICT anyway?

jef safi (2006), rhızomıng thε planεs of rεsılıεncε . . [online] available at http://flickr.com/photos/jef_safi/301035761/ accessed 5/11/08 (CC licensed)

Grades?Grades?

► National Database National Database of Accreditedof AccreditedQualificationsQualifications

► GradesGrades► PointsPoints► Contribution toContribution to

thresholdthreshold

Student voiceStudent voice

I just did AQA GCSE a few days ago and i am sure anyone I just did AQA GCSE a few days ago and i am sure anyone else who did will agree it is shamefully and embarrassingly else who did will agree it is shamefully and embarrassingly easy for GCSE.easy for GCSE.

The only reason everyone seem to get bad grades is the 60% The only reason everyone seem to get bad grades is the 60% coursework - no offense but in my experience from my ICT coursework - no offense but in my experience from my ICT lessons the sort of people who take ICT are the sort who lessons the sort of people who take ICT are the sort who won't work. I took it because it was either ICT or German and won't work. I took it because it was either ICT or German and as i already was doing two other languages i didn't want the as i already was doing two other languages i didn't want the three and always regarded it as my dos lesson.three and always regarded it as my dos lesson.

Also the test is very easy but everyone finds it easy so the Also the test is very easy but everyone finds it easy so the grade boundaries are incredibly high. It's gonna be like 90+% grade boundaries are incredibly high. It's gonna be like 90+% for an A* and 85%+ for an A this year.for an A* and 85%+ for an A this year.

‘‘addonai’ (2007)addonai’ (2007)

Student voice – Simon’s viewStudent voice – Simon’s view

Millwood (2008)

Validity = valueValidity = value

► Dochy & Moerkerke (1997Dochy & Moerkerke (1997) ) “…“…students are expected to students are expected to experience authentic asseexperience authentic assess-ment… ss-ment… because they realise because they realise the relevancy and usefulnthe relevancy and usefulness of it for their future ess of it for their future lives”lives”

► Watts (2008) – analogy Watts (2008) – analogy with moneywith money

‘smaku’ (2006) Back to School… [online] available at http://flickr.com/photos/smaku/154520364/ accessed 5/11/08 (CC licensed)

‘Roby72’ (2008) Money Back Guarantee [online] available at http://flickr.com/photos/roby72/2401722298/ accessed 5/11/08 (CC licensed)

Discussion 1Discussion 1

► Think of an assessment in which you are involved. Think of an assessment in which you are involved.

► What is it that gives it value?What is it that gives it value?► To whom does it give that value?To whom does it give that value?► Is the value different for different people, contexts?Is the value different for different people, contexts?

Some notions of validitySome notions of validity

► Different types of validity (eg Cohen, Manion and Different types of validity (eg Cohen, Manion and Morrison, 2007; Ripley, 2007; Gipps and Murphy, Morrison, 2007; Ripley, 2007; Gipps and Murphy, 1994; Messick, 1988)1994; Messick, 1988)

► A unitary concept (Gronlund, 2005)A unitary concept (Gronlund, 2005)

► To do with predictive value, authenticity (Tomari To do with predictive value, authenticity (Tomari and Borich, 1999)and Borich, 1999)

► Encompasses reliability (Gronlund, 2005)Encompasses reliability (Gronlund, 2005)

Face validityFace validity

► Test recognisabilityTest recognisability

► Relates to perceptions – how appealing is a test to Relates to perceptions – how appealing is a test to students? students?

Watts (2008)Watts (2008)

Construct validityConstruct validity

► Does the test assess what it sets out to assess? Does the test assess what it sets out to assess? ► Also related to marketing research methodologies Also related to marketing research methodologies

(Chisnall, 2005)(Chisnall, 2005)

► Discussion 2: What should ICT assessment set out Discussion 2: What should ICT assessment set out to assess? What are the constructs?to assess? What are the constructs?

The research The research

The research The research

The research The research

The research The research

Research aimsResearch aims

► To critically analyse the ways in which students To critically analyse the ways in which students aged 16 construct their learning of ICT capability aged 16 construct their learning of ICT capability in formal and informal contexts;in formal and informal contexts;

► To explore the relationship between formal and To explore the relationship between formal and informal learning within the field of ICT;informal learning within the field of ICT;

► To explore the methodologies of assessment of To explore the methodologies of assessment of ICT capability at 16 and how this affects student ICT capability at 16 and how this affects student perceptions of their capability;perceptions of their capability;

► To develop a theoretical base to evaluate the To develop a theoretical base to evaluate the construct validity of assessment of ICT at 16. construct validity of assessment of ICT at 16.

Questions emerging from Questions emerging from aimsaims

► How do year 11 How do year 11 students perceive ICT students perceive ICT capability? capability?

► How does the education How does the education and assessment system and assessment system (in England, at 16) (in England, at 16) perceive it? perceive it?

► How do the two differ? How do the two differ?

► How may they be How may they be aligned?aligned?

Concepts emerging from Concepts emerging from literature reviewliterature review

► LearningLearning

► AssessmentAssessment

► TechnologyTechnology

► PolicyPolicy

Gilbert report – what does this Gilbert report – what does this mean for assessment of ICT?mean for assessment of ICT?

DfES (2006:27)

Assessment and learning: Assessment and learning: perceptions in metaphorsperceptions in metaphors

From Gulikers (2006, 11):

A striking number of metaphors refer to the strong influence of assessment on student learning such as: ► the tail wags the dog► the real test bias► the washback effect ► the pre-assessment effect ► consequential validity 

MethodologiesMethodologies

► Central to the enquiry is student perceptionsCentral to the enquiry is student perceptions

► Interpretivist standpointInterpretivist standpoint► Student-centred (reflexivity) -> feminist research Student-centred (reflexivity) -> feminist research

traditiontradition► Power (Foucault, Habermas) -> symbolic Power (Foucault, Habermas) -> symbolic

interactionisminteractionism► Hermeneutics (Husserl, Heidegger)Hermeneutics (Husserl, Heidegger)► Interpretive phenomenology (Conroy, 2003)Interpretive phenomenology (Conroy, 2003)► Triple hermeneutics (Alvesson and SkTriple hermeneutics (Alvesson and Skőldberg, őldberg,

2000)2000)► NOT action research, ethnography nor case studyNOT action research, ethnography nor case study

Conroy (2003)Conroy (2003)

► Interpretive Phenomenology (in nursing)Interpretive Phenomenology (in nursing)

► Hermeneutic Principles for ResearchHermeneutic Principles for Research

► Spiral of hermeneutics – six stages: working from Spiral of hermeneutics – six stages: working from the interpretations of individuals to development of the interpretations of individuals to development of principlesprinciples

► Offers a possible basis for a modelOffers a possible basis for a model

MethodsMethods

► Pilot groups in two schoolsPilot groups in two schools► Repertory grid analysis (Cohen, Manion & Morrison, Repertory grid analysis (Cohen, Manion & Morrison,

2007) to identify personal constructs of assessment 2007) to identify personal constructs of assessment (after Kelly, 1955)(after Kelly, 1955)

► Elicit constructsElicit constructs► Play back to larger population – Conroy’s HPR model?Play back to larger population – Conroy’s HPR model?► Sample those taking and those not taking ICT Sample those taking and those not taking ICT

qualificationsqualifications► Interview for vignettes of key informants (students): Interview for vignettes of key informants (students):

triangulationtriangulation► Revisit after ‘exams’Revisit after ‘exams’► Analyse policy and awarding body documents for Analyse policy and awarding body documents for

comparatorscomparators

IssuesIssues

► Reflexivity from personal standpointReflexivity from personal standpoint

► Fast pace if change in assessment processes means Fast pace if change in assessment processes means that student perceptions may be coloured by that student perceptions may be coloured by misconceptionsmisconceptions

► Access to students after they taken examinationsAccess to students after they taken examinations

ReferencesReferences► ‘addonai’ (2007), commenting in the thread Just got back from ICT GCSE [online] 25 May 2007, 10.08 GMT available at

http://uk.gamespot.com/pages/forums/show_msgs.php?topic_id=25648227&page=1 accessed 02/10/08

► Alvesson, M & Skoldberg, K (2000). Reflexive Methodology: new vistas for qualitative research, London: Sage

► Cohen, L, Manion, L and Morrison, K (2007), Research methods in education. 7th ed. London: RoutledgeFalmer.

► DfES (2006), 2020 Vision: Report of the Teaching and Learning in 2020 Review Group (The Gilbert Report), London: DfES

► Dochy, F & Moerkerke, G (1997), Assessment as a major influence on learning and instruction, International Journal of Educational Research, 27(5), pp 415-431 cited in Gulikers (2006)

► Gipps, C and Murphy, P (1994) A Fair test: assessment, achievement and equity. Buckingham: Open University Press.

► Gronlund, N (2005), Assessment of Student Achievement (8th edition). NY: Allyn and Bacon

► Gulikers, J (2006)Authenticity is in the Eye of the Beholder: Beliefs and perceptions of authentic assessment and the influence on student learning, Maastricht: Open University of the Netherlands

► Messick, S (1989), Validity, in Linn, R (ed.) Educational Measurement. New York: Macmillan

► Millwood, R (2008) Simon, 15, England [online] available at http://www.futureknowledge.org/youth-voice/simon-15-england accessed 14/10/08

► Ripley, M (2007) E-assessment – an update on research, policy and practice, Bristol: Futurelab available online at http://www.futurelab.org.uk/resources/documents/lit_reviews/Assessment_Review_update.pdf accessed 02/10/08.

► Tombari, M and Borich, G (1999), Authentic assessment in the classroom: applications and practice. London: Prentice-Hall.

► Watts, A (2008) Does assessment do what it says on the tin [seminar] Cambridge Assessment Network, 22/10/08

pete.bradshaw@ntu.ac.ukpete.bradshaw@ntu.ac.uk

http://http://petebradshaw.wordpress.com petebradshaw.wordpress.com