challenging minds? students’ perceptions of computer-based world class tests of problem solving

Challenging minds? Students’ perceptions ofcomputer-based World Class Tests of problem

solving

Mary Richardsona,*, Jo-Anne Bairdb, Jim Ridgwayc,Martin Ripleyd, Diane Shorrocks-Taylore, Malcolm Swanf

aDoublestruck, London, UKbAssessment and Qualifications Alliance, Stag Hill House, Guildford, Surrey GU2 7XJ, UK

cSchool of Education, University of Durham, Leazes Road, Durham OH1 1TA, UKdQualifications and Curriculum Authority, Piccadilly, London WIA 8QA, UK

eSchool of Education, University of Leeds, Leeds LS2 9JT, UKfMalcolm Swan School of Education, University of Nottingham, Jubilee Campus,

Nottingham NG8 1BB, UK

Abstract

World Class Arena is a British government initiative to assess and develop the skills of gifted

and talented children. Part of the strategy is to use computer-based tests. Students attempttasks that require them to engage in higher-order problem solving, often in interactive, realistic,contexts. This study reports observations and interviews in schools. Students found tasks enga-

ging and motivating, despite the unfamiliarity of the problem types and the challenging nature ofthe items. Students had no problems working with computers. They were sometimes distracted byattractive graphics, and sometimes used poor heuristics when attempting tasks. The study pro-vides evidence that a computer environment can provide new ways to assess the problem

solving skills of highly able students. # 2002 Elsevier Science Ltd. All rights reserved.

Keywords: Assessment; Computer-based tests; Gifted and talented; Problem solving

1. Introduction

The British government has pledged to improve educational provision for giftedand talented students, attempting to meet their needs and to offer further support

Computers in Human Behavior 18 (2002) 633–649

www.elsevier.com/locate/comphumbeh

0747-5632/02/$ - see front matter # 2002 Elsevier Science Ltd. All rights reserved.

PI I : S0747-5632(02 )00021 -3

* Corresponding author. Present address: AQA, World Class Tests, Stag Hill House, Guildford GU2

7XJ, UK. Tel.: +44-1483-556038.

E-mail address: [email protected] (M. Richardson).

within the curriculum (Department for Education and Skills, 2001). Among theproposals to improve education for able students is World Class Arena, an initiativeled by the Department for Education and Skills (the government departmentresponsible for education) and the Qualifications and Curriculum Authority (thegovernment’s guardian of standards in education and training within the UK).

World Class Arena is a project designed to promote and extend the education ofgifted and talented students and includes World Class Tests, teaching resources, awebsite and an international educational network. Central to World Class Arena,World Class Tests are computer and paper based assessments that include a numberof features:

� testing mathematics and problem solving (for 9 and 13 year olds);� include computer-based tests;� internationally administered to provide comparative information around the

world; and� identification and recognition of the achievements of gifted and talented stu-

dents. (Ripley, 2002)

The focus of the research reported here is students’ perceptions of World ClassTests of problem solving. These tests differ from other ‘standard’ assessments in theUK in a number of important ways. Some of them are computer-based; they are notlinked to any particular school curriculum; they are created to identify gifted andtalented students; and most importantly, they are designed to provide a stimulatingand challenging experience for the students who take them.

In contemporary literature, ‘problem solving’ has been used to cover both well-practised word problems, and tasks which are novel to students and present unfa-miliar challenges. The World Class Arena definition of ‘problem solving’ is addressedby an elaborate domain definition, beyond the scope of this article. For presentpurposes, it is sufficient to note that ‘problem solving’ in the World Class Arena isconstrained to problem solving in mathematics, science and technology. Tasks mustpresent students with unfamiliar challenges—that is they should not be recognisablypart of the existing curriculum. To perform well, students must engage in extendedchains of reasoning, not a series of short exercises; and students are often asked tojustify their answers (an essential component of problem solving). Sample tasks areshown later.

The World Class Tests are predicated in part on an international perspective. Thispresupposes that it is possible to define a mathematics and problem solving curri-culum and assessment domain for the most able 10% of 9 and 13 year olds whichmeet with a good deal of consensus around the world. The approach taken has beento write assessment tasks, which are plausible examples of such a domain, and then,through field testing and expert review, to determine the extent to which these tasksand the domain definition meet with general approval. A significant volume of thesetasks has now been written and extensively field trialled in schools in England; someof the materials have been trialled in the USA, New Zealand and Australia withfurther work in a wider range of countries planned for 2002.

634 M. Richardson et al. / Computers in Human Behavior 18 (2002) 633–649

Data are now available from the first sample of students drawn from five coun-tries: Australia, England, Hong Kong, New Zealand and the USA. Analyses ofthese data will enable us to contrast student performance data with some of theviews already drawn out from the expert reviews carried out in a range of countries.This study examined students’ perceptions of the tasks they have been set, drawingon research carried out with a sample of students from England. These analyses helpus to understand some dimensions of the face validity of the tasks we have designedand assist us in the refinement and presentation of our domain maps.

Terman (1925), Renzulli (1977), and George (2000), amongst others, characterisegifted students as possessing high ability, high motivation and high creativity espe-cially when faced with problem solving. Freeman (1998) argues that the thinking ofgifted students is often closer to that of a subject expert and this deeper knowledgeand comprehension could develop outstanding performance. To motivate able stu-dents, it is necessary to meet their needs by creating innovative challenges becauseresearch also suggests gifted students are often bored by work in school (George,2000; Lens & Rand, 2000). Gifted students can be hard taskmasters, demandingintellectually challenging tasks for themselves.

Research has been an integral part of the development of computer-based testingworld-wide and recent findings have indicated an ongoing need for investigation intostudents’ perceptions of and interaction with computer-based assessments (Bur-khardt, 2002; Mogey & Watt, 1999). World Class Tests can only be effective if theirdevelopment and subsequent application meets the needs of key stakeholders; giftedand talented students worldwide. Addressing the perceptions and responses of indi-viduals is crucial to the success and growth of computer-based test questions. In thepast, debate about students’ use of computer technology was frequently conducted‘over the heads of the students themselves’ (Buckingham, 1999). In order to producematerials students want to use, we (as test providers) must talk to them, find out whatmotivates them and allow our design ideas to be influenced by their suggestions.

1.1. Problem solving

It has been argued elsewhere (e.g. Ridgway, 2001) that ready access to technologyactually changes the sorts of cognitive skills that are valuable—the movement fromindustry-based economies to knowledge-based economies brings with it a shift in thedefinition of desirable knowledge. Computers are both agents of cognitive changeand a vehicle for assessing new kinds of cognitive competence. Tests have beendesigned to assess some of these new skills. Examples include handling complex setsof data, decision making in situations where factors interact, and those ‘discoveryskills’ much advocated by science educators. The Mathematics Assessment ResourceService teams at the Universities of Durham and Nottingham have developed WorldClass Tests, problem solving tasks focussed on mathematics, science and technology.Problem solving in these domains requires a wide range of skills and so a widevariety of task types have been developed. In science and technology, contentreflects everyday experience rather than England’s curriculum. Students are requiredto use extended chains of reasoning and to justify their answers. Task types include

M. Richardson et al. / Computers in Human Behavior 18 (2002) 633–649 635

investigating open situations, reasoning fromevidence/scientific exploration/designing/planning or evaluating/recommending as well as problem solving in abstractdomains, notably mathematics. Current work focuses on developing the range of‘task types’ further.

Table 1 describes the Problem Solving Tasks used in this study. While theMathematics Assessment Resource Service are exploring ways of deducing reason-ing processes from data collected by the computer, this is not the primary reason forcomputer use. The computer allows us to pose problems that we would be unable topose using paper-based tests; it both sets the scene and allows the student to interactwith problems. For example, students click on a party guest to see what they haveeaten (Feeling Unwell), enter numbers into a diagram to see what effect they have onthe result (Small Pyramids, Make 100, Chasm) or watch an animation (Bicycles). Tosolve a problem, the student has to organise their exploration, decide what and howdata must be recorded, perform appropriate calculations and communicate reason-ing on paper.

World Class Arena exploits the unique features of computer environments,while not abandoning the virtues of paper. The computer enables tasks to be setwhich involve animation, full colour and interactivity—design features such as the

Table 1

Problem solving World Class Tests

Problem Description

Sunflower (for 9 year olds) Pupils investigate how to grow the ‘world’s tallest’ sunflower

using two types of plant food. A successful solution involves

controlling these two variables systematically.

Feeling unwell (for 9 year olds) Pupils identify the food that has caused some partygoers to

feel unwell. A successful solution involves recording data

systematically and eliminating foods that were eaten by people

who were not taken ill.

Small Pyramids (for 9 year olds) A number pyramid is presented to pupils. Pupils investigate

how changing one number in the bottom row of the pyramid

affects the value of the number at the apex. This is a pure

number task involving the discovery and use of a linear function.

Bicycles (for 13 year olds) Pupils observe a computer animation of a moving bicycle and

explain why a number of revisions to the bicycle design are

inferior. Pupils are then asked to investigate how changing gear

affects the rate at which the cyclist must pedal.

Make 100 (for 13 year olds) A sequence of six numbers is presented. The pupil may change

the first two numbers. Succeeding numbers are each the sum

of the two preceding numbers. Pupils must find every pair of

starting numbers that make the final number 100. This requires

pupils to control two variables, record solutions and look for

the linear relationship within these solutions.

Chasm (for 13 year olds) Pupils investigate how its breadth, width and thickness affect the

weight that may be supported by a plank bridge. This is a

demanding problem involving the discovery and use of linear,

quadratic and inverse functions.


background, border, titles and icons follow a consistent colour scheme. It is ideallysuited for posing problems which require students to experiment in a controlled‘microworld’, search for patterns and relationships then use these to make inferencesand predictions. Computers are not suited to interpreting sophisticated studentresponses nor are they a natural medium for students to work in, therefore scoringof reasoning is still left to human markers and students use paper for sketchingdiagrams, developing ideas and writing explanations. During the test, students workwith both a computer and a booklet that supports the material presented on screenand provides space for written responses. From the viewpoint of test administration,booklets and computer responses have to be cross-referenced, and at the time ofmarking, trained markers are presented with both computer recorded (sometimescomputer scored) responses and student responses on paper (Figs. 1 and 2).

In most tests, questions develop so that early questions are accessible to all stu-dents, and the demands increase as progress is made. With the more substantialproblems used in World Class Arena each task progesses differently. This designprinciple is important both pedagogically and morally—the designers’ intention is toleave the student feeling they have made progress on every task, and have been ableto show what they can do (as opposed to showing what they cannot do). A studentcan perform poorly in terms of scores obtained (since success on easy parts of a taskmay be given little reward compared with later parts) while remaining positive about

Fig. 1. Bicycles: for 13 year olds. Students observe a computer animation of a moving bicycle and explain

why a number of revisions of the bicycle design are inferior. Students are then asked to investigate how

changing gear affects the rate at which the cyclist must pedal (page 1 of 4).


the test experience. Tests are assembled from tasks with known psychometric prop-erties, using the Mathematics Assessment Resource Service domain definition toprovide appropriate balance of content and process. A test has two components—one paper and one computer based. Each test is designed so that the sum of all thetasks—paper and computer based—provides an adequate sample of the domain.

Claims about the ‘World Class’ nature of the tests are based on two ideas. Firstthat the tests sample hitherto un-assessed aspects of performance, now seen as keyintellectual skills in many statements of educational goals, world-wide; and secondon the basis of the awards procedure, where expert judges decide on the appropriateperformance required of students before they are deemed to be ‘world class’.

Attempting tasks, the student is challenged to represent the problem in appro-priate ways, bring a variety of problem solving strategies to bear under appropriatecognitive control and to explain the answers they obtain. The Mathematics Assess-ment Resource Service have aimed to design problem solving tasks which requirestudents to display elements of performance identified in problem solving literature(e.g. Oliver and Omari, 2001; Polya, 1945; Schoenfeld, 1984) as being essential tosuccessful problem solving. Many tasks have a strong link to the real world, ensur-ing their relevance to the students and presenting them with a tangible challenge.Thus the purpose of this study was an exploration of student understanding ofproblem solving tasks, discussion about task design, human–computer interaction

Fig. 2. Bicycles: for 13 year olds. Students observe a computer animation of a moving bicycle and explain

why a number of revisions of the bicycle design are inferior. Students are then asked to investigate how

changing gear affects the rate at which the cyclist must pedal (page 4 of 4).


issues and their response to computer-based assessment compared to a traditionalpaper and pencil approach.

2. Method

2.1. Sample

An opportunity sample of 12 students was used for each of two tests, making atotal of 24 students (eight boys and 16 girls) included in the study. The students werefrom seven primary and secondary schools in the south of England. Teacherswere asked to select pupils in the top 10% of their year group, and whom theywould categorise as ‘gifted and talented’.

2.2. Design and materials

The computer-based versions of six trial World Class Test problem solving itemswere used, three for nine year olds and three for 13 year olds. Tests were presentedon laptops and students interacted with the computer using the keyboard and amouse. An incomplete randomised presentation order was used, in which each stu-dent completed two tasks from a test.

The World Class Tests are offered as Problem Solving or Mathematics tests—candidates can sit for either or both. There is a paper-based test and a computer-based test for each subject; each lasting 1 h 15 min (13 year olds) and 1 h (9 yearolds). Schools were asked to give the students an opportunity to see example taskson the World Class Arena website1 prior to the interviews. Approximately half ofthe participants accessed the website prior to the interviews.

2.3. Procedure

Interviews were conducted in the students’ schools; typically in the library or in aHeadteacher’s office. A Headteacher is the person responsible for the running ofa school. Students were interviewed individually by one researcher. The tworesearchers involved in the study interviewed students in different schools.

At the beginning of the interview, the purpose of the research was explained to thestudent. They were told that we were interested in their views on the test, but thatthey were not being tested—their results were not important. All of the students saidthat they understood that they were not being tested and some expressed relief aboutthis. To collect data and establish rapport, some friendly conversation about theirpersonal computer use and attitudes to computer uses in general was then includedin the interviews. Good rapport was established with the students and they wereat ease and spoke freely. Students were introduced to the computer-based test bybeing led through a practice example, in which the task itself was discussed with the

1 www.worldclassarena.org.


students, as well as the sorts of interview questions they would be asked. Studentsthen completed two further tasks and were asked questions about the task design,task difficulty and human–computer interaction.

The second section of the interview used an adapted repertory grid method(Fransella & Bannister, 1990) designed to elicit students’ perceptions of task design.Students were shown the same questions but presented as printed task dyads andthen asked to describe as many differences between the tasks as they could think of.Once they had run out of different concepts, students were asked to identify as manysimilarities between the tasks as they could think of. The interviewers did notprompt students, so if they could not see differences they were asked to spot simila-rities. This particular presentation technique was adapted from Green, Hamnett,and Green (2001). Two of the 13-year-old students found it difficult to produce asingle difference or similarity but most, especially the 9 year olds, found it a verystraightforward task. Students were then asked which of the questions they pre-ferred and for an explanation for their choice and the interview ended with a dis-cussion about whether computer-based or paper and pencil tests were preferable. Atthe end of the interview, students were asked whether they had any questions thatthey wanted to ask and were thanked for participating in the research. Participatingschools have been provided with the findings of the research.

The interviews differed from the live version of the tests in several ways: notablythe number of problems presented, the uncontrolled ‘testing’ situation and in thatthere was no paper answer booklet available for the tasks in which responses—suchas answers and explanations—had to be written. Instead, students talked throughthe answers they would have given and used blank paper for their ‘working out’.One student commented that she was not sure about computer-based testingbecause it felt too informal. Thus, some of the responses are inevitably a reflection ofthe research situation.

The development and use of this kind of interviewing procedure is not without itsmethodological problems, and these informed thinking along the way. Key pro-blems are those related to the observation process itself and issues surrounding theinterview process. Students were observed as they worked with the questions onscreen, and the interpretation processes involved in observations are well docu-mented. The very fact of being observed and under the ‘spotlight’ as it were, givesrise to what Foster (1996) refers to as ‘reactivity’, where the knowledge of beingobserved itself influences the behaviour of the individual being observed. Theremedy for this is usually familiarisation, so that the person being observed gradu-ally forgets the observation situation and begins to react in more usual ways. Similarissues have also been raised by, among others, Pellegrini (1996) and Sharman (1999).In the work being reported here, efforts were made to put the students at their ease,but there is no way of fully knowing if the process of being observed and questioneditself has influenced the outcomes.

Interviewing also clearly presents difficulties as a response-elicitation tool, espe-cially where quite young students are involved. The assumption is that asking aquestion will elicit a response that is both detailed and true, and the subsequentresponse interpreted un-problematically by the interviewer. A great deal of the


methodological literature, well summarised in Hayes (2000), indicates that this isunlikely to be the case. Without great care, leading questions may be asked, certainwords may be emotively loaded or tone of voice may itself signal the kinds ofresponses being sought. The interpretation process is also likely to be influenced bythe experience and expectations of the interviewer. In the research being reportedhere, the researchers were well aware of these potential pitfalls, seeking to avoidthem as far as possible.

3. Results and discussion

3.1. Attitudes towards computers and computer-based tests

Students generally enjoyed participating in the research, saying things like ‘‘Thatwas fun.’’ or ‘‘ I enjoyed that, can I do another one?’’ Only two of the studentsbegan to get frustrated when they could not easily complete the tasks. In thoseinstances the researcher discussed the problem with the student and gave clues abouthow to solve it, as responses were not being scored and it was important not todamage students’ self-esteem for the purposes of research.

3.2. Familiarity with computer technology

The process of test development has involved trials on large student samples(several thousand students, across a series of trials). Students involved in a trialcompleted a questionnaire on their reactions to the tests. Comments on their unfa-miliarity with computers, or on difficulties with the interface are extremely rare.

In this study, the students were highly familiar with using a keyboard, mouseand navigating in internet-style software, this competence improving the students’confidence—the ‘everyday’ use of computers enhancing a capability in handlingcomputer-based technology (Brosnan, 2001; Mumtaz, 2001; Paine, 2001). However,it is important not to confuse competent computer use with ability to solve pro-blems; data from trials indicated that computer literacy does not necessarily relate toproblem solving competency in a computer environment.

Some of the students pointed to human–computer interaction aspects in thedesign of the software, when it did not comply with standard Internet presentation(e.g. there was no ‘back’ button). Such observations further support the observa-tions of the students’ competence at computer use and a sophisticated attitude tocommenting on all aspects of the tasks they were shown. Computers were used athome and at school by the vast majority of the students included in the study. Typi-cally, students used computers for homework, for surfing the net, or for playing games.

At the end of the interview, the students were asked whether they thought theywould prefer computer-based or paper and pencil tests. Twenty-one of the studentssaid they would prefer computer-based tests. The three who could not decide orselected paper and pencil were all 13 year olds. The tasks on that test require a lotof recording and working on paper even where the response is entered into the


computer. Two of the students said that they would prefer to be able to ‘jot thingsdown’, while the third said that he could not type fast and would not like to do acomputer-based test in English. When candidates sit the computer-based WorldClass Tests they have a paper booklet to support the test, one that requires them toanswer further tasks and also gives them the opportunity to show their working.Students’ reasons for their preferences for computer-based tests are given in Fig. 3.

Students mentioned that it was easier to give answers on the computer becausetheir hands wouldn’t ache, the writing would be neater or there would not be anyspelling errors.

I like using computers—paper gets all smudged and your hands get sweatywhen you write a lot. On computers your hands are in control.

In World Class Tests, it may well have been easy to give responses on computerand students felt that they benefited because they did not have aching hands at theend of the test. Many students (14) mentioned that they simply liked computers,elaborating that they were more modern, that the tasks were novel, they enjoyed theinteraction or that tasks were fun.

Tests are usually very boring, so doing them on-line would be exciting

Paper and pencil can’t do things like make the sunflower grow.

Students preferred the use of pictures and colour comparing them to paper-basedtests that were described as generally not very colourful. Of course, there are

Fig. 3. Reasons given for preference of computer-based tests over paper and pencil (number of children

who gave that reason in italics).


exceptions to these observations, but in general students’ perception of this differ-ence between paper and computer-based tests demonstrates what has been describedas the ‘holding power’ of computers (Computers in Teaching and Learning, 1999).The following quotations support this:

Children take more notice of things on computer because writing is quite boring

On computer you get pictures, but writing is only in black and white

Children think it’s boring, but on a PC it’s bright and colourful.

Another finding that emerged from the research is that some students (6) do notlike erasing things and preferred to be able to revise their answers on computer. Thisis probably as a result of the emphasis that teachers place on presentation in school,where completion and presentation often seem to take precedence over comprehen-sion and reasoning (Desforges & Cockburn, 1987). Others (4) mentioned that it wasquicker to do computer-based tests and one student even mentioned that therewould be administrative advantages:

Information can be recorded better and sent faster than mail.

Six students thought that the computer tells you if the answer is right or wrong orthat it helps you. Several tasks had interactive facilities that did support the students toget the answer correct. In most cases, the solution was only possible using some formof interaction with the computer. One student mentioned that the computer-based testwas less anxiety provoking than paper and pencil tests, describing her reaction:

On paper you feel [that there are] things to do, you feel tense. On computer youwouldn’t get that scared.

This could be a function of the interactive nature of the research environment, butduring pilot work for this research, a 12-year-old girl described sitting paper andpencil tests as ‘‘a very isolating experience’’ saying that interaction with the compu-ter ‘‘made you feel less alone’’ during a test.

Some students said that they preferred computer-based tests because they had achoice about the order of the tasks, they could go back to previous tasks (3) andthey could delete mistakes (4). All of these supposed advantages are features sharedwith paper and pencil tests, but the computer-based format might make them moreobvious to students. One student thought that people could not look over and seewhat he was doing however this was not found to be the case in large scale trials ofcomputer-based questions undertaken as part of the question development process.One student felt that the computer helped her to do things at her own pace. Thecomputer-based test did not assist this, but this is an advantage often cited forcomputerised instruction and it may be that this student had heard about thispotential advantage, as may be seen in Fig. 4.


Students could see many potential disadvantages of using computers for testingsome of which were very sophisticated. One boy mentioned health and safety con-cerns, saying that one could get eyestrain because there are no breaks in exami-nations. Computer-based tests will have to be shorter than some traditional paperand pencil examinations, which are often 3 hours long. Otherwise, there will be ser-ious health and safety implications. A few students mentioned technical problemsthat could arise (viruses, malfunction, security and connection speed) and somestudents pointed out that there were subject-specific reasons that made computertesting more suitable for some subjects than for others.

3.3. Task design

This study focused on students’ perceptions of computer-based problems. Theresponses are often complex but apparently honest because students, when asked,seemed to like an opportunity to ‘tell it like it is’. The study also produced a largeamount of unsolicited, sophisticated commentary that is useful not only to thosewho design and programme World Class Tests but also feeds our understanding ofhow gifted students contextualise what they look at and how they interact with on-screen assessment. Success of task design was investigated and produced responsesthat point to what Papert (1982) refers to as the ability of computers to make sub-jects ‘come to life’ in a way that is accessible and relevant.

Participants in the study were clearly enthusiastic about what they saw andexperienced during interviews, initial discussion about design was peppered withcomments such as ‘interesting’, ‘colourful’, ‘attention grabbing’, ‘enjoyable’ and soon. Three respondents expressed ‘no interest’ in the use of colour or pictures stating

Fig. 4. Potential disadvantages of computer-based tests in general (number of children who gave that

reason in italics).


that images were not always necessary to answer the task. One respondent2 com-mented that colours were not needed and neither were pictures; he liked the picturesbut pronounced them ‘unnecessary’. This corresponds to the distinction taskdesigners make between pictures (‘eye candy’) and figures (which do some work,cognitively). Discussions about the design of tasks led to students commenting onthe context of tasks for example; the task Feeling Unwell (see Fig. 5).

This task elicited comments such as the following:

Types of food at the bottom look good but that makes me hungry

. . .they are all cartoons and look like they’ve eaten something wrong

A banana definitely won’t make them feel unwell

Many of the tasks make use of realistic contexts. It is well known (e.g. Cooper &Dunne, 1999) that realistic contexts can lead to students bringing outside knowledgeto bear which may or may not, be relevant. The problem solving tasks are designed

Fig. 5. Feeling Unwell: for 9 year olds. Students identify food that has caused some partygoers to feel ill.

A successful solution involves recording data systematically and eliminating foods that were eaten by

people who were not taken ill.

2 This student was 9 years old, exceptionally bright and completed the 13-year-old problem solving

tasks scoring 100%.


in such a way that real world knowledge should not interfere with solving theproblem in hand. Nevertheless, students sometimes adopt inappropriate schemasthat lead to poor solutions. For example, in the ‘Feeling Unwell’ problem, studentsfrequently saw the screen and immediately said that the question was about healthyeating. Once this schema had been accessed, students rarely looked for an alter-native possibility. Instead of treating the task as a logic question (what had Garueaten that the other guests had not eaten), they would look for unhealthyfoods eaten by Garu, or healthy foods eaten by the other guests. The assumptionthat a test will always present a politically correct solution (‘bananas good, burgersbad’) is wrong both in terms of the designers’ intentions—some tasks are designedspecifically to see if students can abstract from situations which are emotionallyladen, or can choose a response which is not politically correct—and in terms of thefacts of the matter (bananas can indeed be a source of food poisoning). Such contexteffects are not restricted to computerised tests—these problems exist with paper-based tests.

Students’ accounts of the motivation for doing the tests was roughly split betweenthe initial ‘look’ of the tasks, and once attempted because they presented a ‘chal-lenge’. This attraction is a response that is perhaps due to the sample of studentsinterviewed (i.e. the more able) indicating some of the ‘triadic characteristics’—highability, high motivation and high creativity—that have been ascribed to gifted andtalented students. Gifted students often articulate in ways that differ from otherstudents and the responses recorded in this study hint at skills that allow them tosolve problem solving tasks effectively (Freeman, 2001). World Class Tests seemto support these claims as all of the students in our study stated that they werechallenged and motivated by the tasks, frequently saying how exciting it was or howmuch they had enjoyed doing them.

During the interview, students were asked about preferences for tasks they per-ceived as easier or harder. Results indicated no clear preference suggesting factorsother than task difficulty influence of students’ preferences. Enthusiasm for challen-ging questions suggests construction of a more detailed analysis of their individualcomments/responses in order to understand their thinking and meta-cognitiveresponses to the computer-based tests. Tasks which explicitly ask students to writedown their reasoning do however have the effect of making students stop and reflecton their own problem solving approach and thus encourage meta-cognitive activity.Further analysis of problem solving tasks of World Class Tests will continue as theproject develops, driving test designers and researchers to investigate and respondstudents’s problem solving strategies and provide them with challenges appropriateto their ability.

4. Conclusion

This study differed from much of the current research that has been conducted inthe area of the computer-based assessment because it focused on gifted and talentedstudents and their perceptions of novel problem solving tests designed specifically


for them. Both positive and negative effects of the computerised, problem solvingtests were noted during the study.

One of the aims was to ascertain whether the participants found the tests engagingand motivating. Overall, students enjoyed the tests, despite the unfamiliarity of theproblem types, the medium of testing, and the challenging nature of the items. Stu-dents also approved of the range of colour and pictures used. Not only did thecomputer-based tests capture students’ attention: their attention was sustained. It ispossible the novelty value of sitting computer-based tests fuelled their enthusiasm,but the interactive nature of the tests also seemed to engage students. Two studentswho were initially reluctant to participate in the study showed more interest whenthey saw it was computer based and, once at the computer, quickly became absor-bed in solving the problems. There was no evidence that students’ computer skillsaffected their interaction with the tasks, mainly because they were all skilled com-puter users. Research conducted by teams at Mathematics Assessment ResourceService and at Leeds University has found similar trends within much larger samplesof students who took computer-based tests; we do not expect unfamiliarity withtechnology to be a barrier to assessing high attaining students, in future studies.

It has been claimed that attention-grabbing graphics and interactivity have animportant role as they can motivate students to undertake a task (Stoney & Wild,1998). Here, this study found that some graphics distracted students’ attention awayfrom the instructions and question text, leading to a misunderstanding of the ques-tion. Not reading the question carefully is a common mistake on paper basedexaminations too, but it is suggested that computer-based test designers need to beparticularly sensitive to the potential of graphics and animation to interfere withperformance.

Many of the tasks set problems in realistic contexts. As noted elsewhere (Cooper& Dunne, 1999), realistic contexts can lead students to adopt inappropriate schemasfor the task to be solved. Choosing an inappropriate schema meant that studentswere unable to solve problems effectively; once ‘hooked’ into a particular schema itwas often difficult to persuade them to re-engage. These inappropriate schemas oftenindicated poor problem solving skills (such as a shallow interpretation of the prob-lem) rather than poor task design. Once started on the wrong track, students oftenhad to be prompted to look at what they were being asked to do.

More work needs to be done on the identification and development of problemsolving skills, especially in a context where computer-based assessment is used. Stu-dents often bring their own strategies to bear on problems, which might not be wellsuited to the particular problem presented. For example, many of the World ClassTest tasks required a strategy of systematic trialling akin to the scientific enquiryskills problems described by Bennett and Persky (2002). Students who participatedin the current research study often adopted a ‘guess and check’ strategy. Experi-enced computer users have learned that it is often quicker to guess and check alter-native choices very fast, rather than read the manual, or systematically attemptthings. In this regard, transfer of training from their wider experience of softwareand games may not foster a reflective, systematic approach. It is to be hoped that‘meta-knowledge’ concerning the problems best solved by systematic methods, or


best solved by fast search, will develop as students experience more computer-basedproblems.

References

Bennett, R. E., & Persky, H. (2002). Problem solving in technology-rich environments. In C. Richardson

(Ed.), Assessing gifted and talented children (pp. 19–33). London: Qualifications and Curriculum

Authority.

Brosnan, T. (2001). The ‘third way’: teaching ICT. Athens, Greece: Invited paper at the 1st European

Conference on Modern Education.

Buckingham, D. (1999). Superhighway or road to nowhere? Children’s relationships with digital technol-

ogy. English in education, 33(1), 3–12.

Burkhardt, H. (2002). World class assessment: principles, practice and problem solving. In C. Richardson

(Ed.), Assessing gifted and talented children. London: Qualifications and Curriculum Authority.

Computers in Teaching and Learning, Staffordshire University. (1999). Learning with software: pedagogies

and practice. Available: http://www.education.edu.au/archives/cp/o2.htm [retrieved 29 October 2001].

Cooper, B., & Dunne, M. (1999). Assessing children’s mathematical knowledge: social class, sex and prob-

lem solving. London: Oxford University Press.

Desforges, C., & Cockburn, A. (1987). Understanding the mathematics teacher. London: Falmer.

Department for Education and Skills. (2001). Schools: building on success. Norwich: Her Majesty’s

Stationery Office.

Foster, P. (1996). Observing schools. London: Paul Chapman Publishing.

Fransella, F., & Bannister, D. (1990). A manual for repertory grid technique. London: Academic Press.

Freeman, J. (1998). Educating the very able: current international research. London: Her Majesty’s

Stationery Office.

Freeman, J. (2001). Gifted children grown up. London: David Fulton.

George, D. (2000). The challenge of the able child. London: David Fulton.

Green, C., Hamnett, L., & Green, S. (2001). Children put the national tests to the test. Education 3–13,

29(3), 39–42.

Hayes, N. (2000). Doing psychological research. Buckingham: Open University Press.

Lens, W., & Rand, P. (2000). Motivation and cognition: their role in the development of giftedness. In

K. A. Heller, F. J. Monks, R. J. Sternberg, & R. F. Subotnik (Eds.), International handbook of gifted-

ness and talent (pp. 193–202). London: Elsevier.

Mogey, N. & Watt, H. (1999). The use of computers in assessment of student learning. Available: http://

www.icbl.hw.ac.uk/ltdi/implementing-it/using.htm [retreived 9 April 2001].

Mumtaz, S. (2001). Children’s enjoyment and perception of computer use in the home and the school.

Computers and education, 36, 347–362.

Oliver, R., & Omari, A. (2001). Student responses to collaborating and learning in a web-based environ-

ment. Journal of computer assisted learning, 17(1), 34–48.

Paine, N. (2001). Bringing science alive. Times Educational Supplement; OnLine Magazine (9 November).

London: Times Supplements Limited.

Papert, S. (1982). Mindstorms: children, computers and powerful ideas. Brighton, United Kingdom:

Harvester Press.

Pellegrini, A. D. (1996). Observing children in their natural worlds. Mahwah, NJ: Erlbaum.

Polya, G. (1945). How to solve it. London: Penguin Books.

Renzulli, J. S. (1977). The enrichment triad nature: a guide for developing defensible programs for gifted and

talented. New York: Creative Learning Press.

Ridgway, J. (2001). Tools for change—building the knowledge base for macro-systemic change. In

H. Taylor, & P. Hogenbirk (Eds.), Information and communication technologies in education: the school

of the future. Boston: Kluwer.

Ripley, M. (2002). Introduction. In C. Richardson (Ed.), Assessing gifted and talented children. London:

Qualifications and Curriculum Authority.


Schoenfeld, A. (1984). Mathematical problem solving. London: Academic Press.

Sharman, C. (1999). Observing children: a practical guide. London: Cassell.

Stoney, S., & Wild, M. (1998). Motivation and interface design: maximising learning opportunities. The

Journal of Computer Assisted Learning, 14, 40–50.

Terman, L. M. (1925). Mental and physical traits of a thousand gifted children: genetic studies of a genius

(Vol. 1). Stanford, CA: Stanford University Press.


challenging minds? students’ perceptions of computer-based world class tests of problem solving

Documents