Sandra Johnson, Assessment EuropeSQA October 2013

Any feature in examinations that allows different candidates for the same qualification to achieve that qualification through different assessment routes.

Appears in the context of the question rather than the content

Allows candidates to choose between items of mandatory content

Reflects optional content in courses Embedded within questions Parallel papers/units Class-based tasks

Choose one of the following four writing scenarios on the contexts of employability, culture, culture and learning that you have studied in the course. Write approximately 120-150 words.OPTION 1You have just come back from a summer job in Spain and you have been asked to write a report for your school’s/college’s Spanish webpage.  You must include the following information:

OPTION 2You recently watched a Spanish film at an International film festival.  You have been asked to write a review of the film, in Spanish, for a Spanish website.  You must include the following information:

Where you worked and how you got there everyday What you had to do every day as part of your duties and why you

enjoyed them or did not enjoy them How you got on with your boss and the other employees How you think this experience will help you in the future

An outline of the plot of the film and a description of one of the major scenes A description of the main characters and why you liked or disliked them What the themes were and which one you considered to be the most important Why you would recommend the film to other young people

Attempt BOTH questionsYou should use different examples of media content in your response to each question.1. Media Content in Context

 How audiences respond to genre texts can depend on the mixture of expected and expected elements within them.  Analyse how this statement could apply to media content you have studied. In your response you must cover:a)the ways in which genre markers are evident in narrative structures, codes and/or conventions (10 marks)b)the ways in which genre markers are evident in at least one other key aspect from categories, language or representation (10 marks)c)the ways in which different audiences might respond to expected and unexpected elements of the genre (10 marks) You can use the bullet points to structure your response, or integrate your responses to the bullet points in any appropriate way.   2. The Role of Media

 The media is consistently criticised as being intrusive, out of control or problematic in some other way. Often, the response from the media is that it is simply meeting needs. Discuss this statement with reference to media content you have studied.  

Answer either Question B1 or Question B2 and Question B3

B1

Compare the importance of the Executive in making policy, with reference to two political systems you have studied.

In your answer you should compare three aspects of policy making.

12 marks

B2

Compare the importance of the Judiciary, with reference to two political systems you have studied.

In your answer you should identify three aspects of the Judiciary.

12 marks

8. Answer either A OR B.

A Describe how animals survive adverse conditions.

OR B Describe recombinant DNA technology.

4  Labelled diagrams may be used where appropriate 

Total marks — 40

ECTION 1 — Scottish Text — 20 marksRead an extract from a Scottish text you have previously studied and attempt the questions.Choose ONE text from eitherPart A — Drama Pages 2–7orPart B — Prose Pages 8–17orPart C — Poetry Pages 18–25

Attempt All the questions for your chosen text.

SECTION 2 — Critical Essay — 20 marksWrite ONE critical essay on a previously studied text from the following genres — Drama, Prose,Poetry, Film and Television Drama, or language.Your answer must be on a different genre from that chosen in Section 1.

To allow flexibility in curriculum coverage To maximise learning motivation and test

motivation for candidates To provide opportunities for candidates to build on

their strengths and weaknesses

Should optional questions be used in examinations? Stalnaker, J.W., School and Society, 1935

Optional questions in tests and examinationsDevadson, M.D., Teacher Education, 1963

Test performance and the use of optional questionsDucette, J. & Wolk, S., Journal of Experimental Education, 1972 Question choice in examinations: an experiment in geography and scienceTaylor, E.G. & Nuttall, D.I., Educational Research, 1974

Validity (comparability of demand) Wisdom of candidates’ choices* Reliability Grading (all in pool)

Fairness to candidates

* Willmott, A.S. & Hall, C.G. (1975) O level examined: the effect of question choice. Schools Council.

Candidates do not always make the wisest question choices

Johnson, S., Johnson, R., Miller, L., Boyle, A. (2013) Reliability of Vocational Assessment: An evaluation of level 3 electro-technical qualifications. Coventry: Ofqual.

A B

candidates 11 25

questions 23 6

markers - <1

marker-question interaction 1 <1

candidate-question interaction 52 55

candidate-marker interaction - -

confounded residual 12 12

  Section A

(20 3-mark questions)

Section B

(6 15-mark questions)

  phi 95% CI* Phi 95% CI*

Single marking 0.71 ± 9.4 0.66 ± 18.0

Double marking 0.73 ± 9.0 0.68 ± 17.1

* These are given as marks around candidates’ section total scores

One of several alternative Unit 1 papers Presented three extended response questions candidates were required to answer two questions

so in practice there were three different pathways through the paper

each question was worth 60 marks for a 120-mark paper total

Variance component estimatesand % contributions 

Option 1(q1q2)

Option 2(q1q3)

Option 3(q2q3)

Candidates   41 33 69Questions 2 <1 5Confounded residual 57 67 26 Generalizability coefficients 

     

Relative measurement 0.59 0.50 0.84Absolute measurement 0.58 0.50 0.81Standard errors of measurement      SEM relative 9.4 10.4 8.5SEM absolute 9.6 10.4 9.3Margins of error      ME relative 18.4 20.4 16.7ME absolute 18.8 20.4 18.2

a 2-section paper two three-part open-ended questions in each section Allows

candidates to choose between items of mandatory content

candidates were required to answer one question from each section

so in practice there were four different pathways through the one paper

each question worth 35 marks for a 70-mark paper total

  Pathways through the paper

  q1+q3 q1+q4 q2+q3 q2+q4

2009 0.70 0.63 0.66 0.58

2010 0.60 0.56 0.50 0.56

2011 0.72 0.65 0.59 0.62

* ‘Tariff section’ weights in analyses were 0.571 and 0.429,

respectively, for the 10-mark and 15-mark part-questions.

  q2a q2b q2c 

q4a q4b q4c

Reliability coefficients       

     

Relative measurement 0.44 0.42 0.70 

0.52 0.54 0.69

Absolute

measurement 0.41 0.38 0.66

 

0.49 0.48 0.65

Should optionality be encouraged or eliminated?

If continued in future paper design – what forms would be most appropriate?

How should marker reliability studies be modified to accommodate differential option popularity?

How soon can comprehensive reliability studies be introduced to look at optional pathways?

Without question pretesting, what might be done to give all candidates fair grading outcomes in the context of optionality?