CBA Autumn 2015

Principal Information Seminar

19th, 22nd and 23rd June 2015

Anne Cromie and Dorothee Wagner

1

Learning Intentions

At the end of the session you will be aware of the:

•context for CBA in Northern Ireland for autumn 2015;•findings of independent reviews of computer-based assessments (CBA) including the OECD Review;•developments made to CBA in response to feedback from schools; •reports and outcomes produced and how they can be used to inform planning; and •guidance and support available from CCEA and C2k.

2

Overview

9.00 or 13.00 Registration

9.30 or 13.30 Welcome

Context for CBA Autumn 2015

10.30 or 2.30 Tea/Coffee

10.45 or 14.45 Developments to CBA

CBA Reports and Outcomes

Next Steps

11.45 or 15.45 Questions, Evaluation & Close

3

Focus on Raising Standards

“The introduction of the CBAs is consistent with the national strategy to improve pupil performance within literacy and numeracy.” p61

“…fit well with an official assessment strategy that focuses on assessment for learning and a wider national strategy to reduce inequity in student outcomes.” p61

4

OECD Review of Evaluation and Assessment in Northern Ireland Education 2013

Step Change

5

Step Change

“Literacy and numeracy are bedrocks of a successful education

system. Getting this right significantly enhances a young person’s

employability and lays the foundations for further success

in all areas of learning and life.”

Step Change, October 2014, www.cbi.org.uk

6

Step Change

• At primary school, literacy, numeracy, communication and ICT skills have been made a central focus – and computer-based assessment is increasingly used as a tool for embedding ICT skills into broader curriculum study.

• Helping young people to be confident and familiar with ICT is important.

• However, greater improvement is still needed. On leaving primary school, 18% of students still do not have the expected level of communication and mathematics skills for their age.

• The policy focus on literacy, numeracy, communication and ICT is strongly supported by industry

7

Northern Ireland employers’ top priority areas for action (%)

8

Top priority areas for action (%) in primary schools

• Clearly defined goals on literacy and numeracy 86.54%

• Development of communication skills 53.85%

• Development of character / behaviours 46.15%

• Clearly defined goals on science 30.77%

• A broader definition of achievement at school

covering exam results and other factors essential 17.31%

to success in life and work

9

Recent Evaluation and Review

10

Evaluation and Review of CBA since 2012

Standard Evaluation

Commissioned Review Additional

Annual and ongoing evaluation by CCEA

1.CCEA review the operation of the new assessments.2.ETI review to determine how effectively schools make use of assessment information.

3.Independent review of the technical issues - why they occurred and what lessons can be learnt for the future.

4.Independent review of policy and practice.

5.Department review the policy of statutory CBA.

OECD Review

11

Evaluation and Review of CBA since 2012

• Covered all aspects from procurement to operation – Phased approach in autumn 2013– 2014 mini trial– Phased approach in autumn 2014 and 2015

• Overall findings– Consistent– Reinforce key messages

• Since 2012:– 184 schools chose to use CBA in 2013 – 194 in 2014– 276 in 2015

12

Moving Forward

Procurement Cycle

•Five year procurement cycle

Technical Support/ Transformed Primary Network

•Continued partnership – CCEA/C2k/Capita/suppliers•Enhance CCEA/C2k helpdesk•Monitoring network stability•Ensure ICT equipment set up for CBA•Revised Pre-flight check for IE11

– Support CBA on transformed network

13

OECD Review

14

OECD:Particular Strengths of CBA

OECD has identified the following as particular strengths:

•Standardised to NI school population•Aligned to revised curriculum•Adaptive (difficulty of the questions adapts to the level of the pupil)

– reduces the risk of demotivation– provides nuanced, fine-grained profile of the pupil’s performance.

•Consistent with the national strategy to:– improve pupil performance in literacy and numeracy– focus on assessment for learning

15

OECD:Main Benefits of CBA

OECD has highlighted CBA as a reliable basis for…

•assessing progress and learning needs •planning teaching to meet learner needs•supporting self-assessment and individualised target setting•reducing the workload of teachers as no marking is involved•providing teachers with clear and reliable information for parents on their children’s strengths and weaknesses

16

OECD:Learning for CBA

OECD identified the following regarding CBA:

•Technical challenges– procurement process– screen freezes/sound– physical dexterity (keyboard and mouse)

•Feedback of results– adaptivity– delay in standardised scores– parent reports

•Lack of continuity– procurement process– InCAS– longitudinal data– two providers

•Timeframe

17

Ministerial Statement

• OECD Review positive

‘The practice of having this sort of universal, formative assessment in primary school, mapped to our own curriculum and delivered at the start of the academic year, is noted with approval by the OECD’

John O’Dowd, Oral Statement (11 March, 2014)

18

Legislative Context

In an email to schools on March 26, 2015, the Assessment and Qualifications Team (DENI) noted that:

•Since the introduction of NILA and NINA assessments in 2012, CCEA has worked with both assessment providers to improve both CBAs to best meet school needs. •Participant schools reported increasing confidence in the educational value of the assessments, with over 90% stating that they plan to use them again, if available. •All primary schools are now invited to register their interest in participating in a further voluntary CBA programme in autumn 2015 and the Minister would like to encourage as many schools as possible to consider taking part.

19

NILA and NINA

20

What do they have in common?

•May be completed only by Year 4 to Year 7 pupils•Web-based and pupil responses to questions are saved automatically online•Adaptive and not time-bound•Longer a pupil takes to answer the longer the assessment will take•Can be paused and re-started from that point•Available in English & Irish•Special Entry Points available•Pupils cannot repeat assessments•Similar tables and reports•Reports and outcomes available within 48 hours of completion

NILA and NINA

21

22

Focus on Skills

Aspect of Literacy Skills Focus Pupils’ ability to …

Listening Information Retrieval

Listen to and understand information and ideas.

Reading Sequence Understand information and ideas

Information Find and use information

Features of Texts

Understand different types of texts

Opinions Respond to different layers of meaning

Writing Punctuation Punctuate text

Spelling Recognise accurate spellings

Grammar Use grammar accurately23

The Literacy Assessment

• Single assessment • No time limit• Most pupils complete 46 questions• Average time of 40 minutes• Four difficulties of questions

– age appropriate for the pupil (=), for a younger pupil (<), an older pupil (>) or an even older pupils (>+)

• SEN Entry Point starts with (<)• Question number is displayed at the bottom of the screen• 8 different answer techniques

– cloze, drag and match, drag into order, drag punctuation, drag to hot spot, pick multiple, pick one and select word

• Practice assessment

24

Adaptive Routing in NILA

25

Let’s take a few minutes to look through the CBA Objectives Report

26

Focus on Skills

Skills Focus Pupils ability to …

Strand 1 Number Counting & Understanding Number;

Facts Facts and Mental Arithmetic

Operations Operations with Whole Numbers, Integers and Decimals

Fractions Fractions, Decimals, Percentages, Ratio & Proportion

Problems Problems – including Money, Function Machines & Algebra

Strand 2 Shape and Space

Shape and Space

Measures Measures

Handling Data Handling Data

Skills Focus Pupils’ ability to …

Strand 1 Number Counting & Understanding Number;

Facts Facts and Mental Arithmetic

Operations Operations with Whole Numbers, Integers and Decimals

Fractions Fractions, Decimals, Percentages, Ratio & Proportion

Problems Problems – including Money, Function Machines & Algebra

Strand 2 Shape and Space

Shape and Space

Measures Measures

Handling Data Handling Data27

The Numeracy Assessment

• 2 strands:

– Strand 1 Understanding & Using Number – Strand 2: Shape & Space, Measures and Handling Data

• Neither has a fixed time for completion• Each strand assessed using an adaptive approach• Pupils take an average of 20 minutes per strand• Only mental arithmetic questions are timed• Irish Medium version only available to IM schools or those with IM

units• Pencil and paper should be available to pupils• Practice area• Voiceover available for every question

28

Adaptive Assessment

• Start in their target year - one year below their current school designated year e.g. Year 3 questions for a Year 4 pupil

• SEN Entry Point is 2 years below target year• 3 difficulties of questions

– Easy, medium and difficult

• First question is a ‘medium’ one• First set of 6-8 questions assesses general numeracy• Move up, down or stay in the year• Move up or down can be reversed• Process repeated until assessment determines they are answering

questions appropriate to their ability

29

Adaptive Assessment

30

Let’s take a few minutes to look through the CBA Objectives Report

Promotions and Demotions by Year

31

32

Coffee Time10.30 / 14.30

CBA Developments

33

Developments since 2012

34

SEN Entry Point Both

HI Entry Point NILA

Skills Focus Report Both

Year Group Report NINA

Improved Parent Report Both

SS and AROs available within 48 hours Both

Improved filtering of reports NILA

Improved IM selection Both

Modified pupil checklist Both

Developments since 2012

NILA and NINA administrator’s access/permissions aligned Both

Improved assessment monitoring NINA

Item Analysis Both

Question number added to the bottom of the screen NILA

Reduction in question techniques NINA

Improved practice area NINA

Software adapted to operate on the C2k transformed network Both

Pre Flight Check Both

35

Special Entry Points

SEN (Special Educational Needs) Entry Point:

•for pupils experiencing difficulties with literacy/numeracy– quicker access to easier questions – more positive and potentially shorter assessment experience– offers same access to the full range of questions as other pupils

HI (Hearing Impaired) Entry Point:

•for pupils who are receiving support for a hearing impairment – removes the Listening aspect of the assessment– allows pupils to be assessed in Reading and Writing

Use of the Special Entry Points will not be obvious to pupils.

36

Qualifying for Special Entry Points

The teacher and the Senior Management Team must:

For the SEN Entry Point•reasonably expect that the pupil is performing two or more years below their chronological age; and•be in agreement that an SEN Entry Point is appropriate for the pupil.

For the HI Entry Point•have evidence that the pupil is receiving support for a Hearing Impairment; and•be in agreement that an HI Entry Point is appropriate for the pupil.

Only the Administrator can set a Special Entry Point

37

Setting an SEN Entry Point for a Pupil (NILA)

38

Setting an SEN Entry Point for a Pupil (NINA)

39

NINA SEN Entry Point

Pupil’s Curriculum Year

First Non-SEN Entry Point questions will be from:

First SEN Entry Point questions will be from:

Year 4 Year 3 Year 1

Year 5 Year 4 Year 2

Year 6 Year 5 Year 3

Year 7 Year 6 Year 4

40

General SEN improvements

• Specific guidance included in the NILA / NINA Guidance Documents;

• Improved font size, style and colour;• Language modification;• Setting an SEN entry point for individual pupils;• Setting an HI entry point (removes Listening questions from NILA);• Improved ‘drag punctuation’ questions in NILA;• Voice in NILA and NINA Demos slowed down;• Improved pupil demos (for the practice areas and the real

assessment);• Streamlined NINA Practice area; and• Simplified pupil checklists.

41

Monitor Assessments

NILA EM completed NINA completed

NILA EM started NINA started

NILA IM completed NINA not started

NILA IM started

NILA – to help monitor progress during an assessment the question number is displayed at the bottom of the screen (most pupils answer 46 questions).

NINA – each strand takes approximately 20 minutes.

42

NILA: Monitoring Assessment Progress for the School

Shows that the pupil has started NILA in English

Shows that the pupil has completed NILA in English

Shows that the pupil has started NILA in Irish

Shows that the pupil has completed NILA in Irish

43

NINA: Monitoring Assessment Progress for the School

Shows that the pupil has started NINA

Shows that the pupil not started NINA

Shows that the pupil has completed NINA

44

Item Analysis

NILA•Additional questions added to NILA at the lower and upper end.•Drag punctuation questions:

– Improved ‘grab’ of the box; and– More accurate placement of the punctuation mark.

•Improved graphics and screen display.•Improvements for Hearing Impaired (HI) pupils.

NINA•Reviewed small number of questions with high proportion of incorrect answers.

45

Reduction of Question Techniques

46

Pupils are allowed to use pencil and paper

Cross Out

47

Calculation

48

Multi-drag

49

50

Computers

• Use pupil workstations.• New teacher laptops can be used, if required.• Connect to the C2k network at least 30 minutes

before CBA use.• Computers not been connected to the C2k network

recently may take longer to update.• Pupils must first log into the C2k network with their

individual C2k username and password. • Multiple pupils logged on with the same username

and password will not be able to access NILA and NINA.

51

Check the Sound

NINA•Go to the practice area •Select any practice question and •Click the sound icon•You should be able to hear the sound play

NILA•Go to the practice assessment•Click the sound icon•You should be able to hear the sound play

52

Accessing CBA

53

Teacher ‘MY-SCHOOL’ homepage

• ‘View All’• ‘Curriculum Links’• ‘NILA’ or ‘NINA’

Accessing CBA

54

Pupil ‘MY-SCHOOL’ homepage

• ‘My Links’• ‘NILA’ or ‘NINA’

Pre Flight Check (PFC)

Key Points

•Simple and user-friendly resource for schools•Automatically runs when pupils click on NILA or NINA link via their ‘MY-SCHOOL’ homepage•Completed prior to commencing assessments•Checking correct versions of software are installed•In case of incorrect set-up - provide guidance and advice to schools•When successful takes you straight to the assessment homepage

NINA NILA

http://preflightcheck.ccea.org.uk/nina.html            http://preflightcheck.ccea.org.uk/nila.aspx            

55

Pre Flight Check (PFC)

56

Something’s Wrong

57

Pre Flight Check (PFC)

• CCEA captures all PFC information• CCEA is notified if a PFC has a problem• Monitoring provides frequency of information and helps identify

patterns • Completing the pop up gives details of:

– school, – workstation – teacher’s comment

• Need to include both DENI number and workstation number – little sticker on computer

58

Outcomes and reports

59

What NILA / NINA Tables and

Reports are available?

• NILA / NINA Results Tables provide schools with Age-Related Outcome (ARO) Ranges and Standardised Scores (SS). These are available for individual pupils, classes, year groups and at whole school level. These build over time to show additional data as it becomes available for new academic years.

• NILA / NINA Parent Reports contain the results table for an individual pupil, showing only the Age-Related Outcome (ARO) Range.

• NILA / NINA Objectives Reports provide specific information about the questions asked and whether the pupil answered correctly or incorrectly.  

60

Results Table (School, Year Group and Class)

61

Results Table (Individual Pupil)

62

Parent Report

63

Skills Focus Reports

64

Using Outcomes

CBA “… are intended for diagnostic purposes and the assessment data is primarily used for:

•Assessing pupil progress and identifying their learning needs

•Supporting self-evaluation and target-setting

•Helping teachers shape their teaching to address the learning needs of their pupils

•Providing information for parents on their children’s strengths and weaknesses within literacy and numeracy.”

OECD Review of Evaluation and Assessment in Education (2013) p.54  

65

Next Steps

66

Roles and Responsibilities

Role Responsibility

PrincipalOverview of school CBA

• arranging the SIMS upload to WebXchange• accessing the school username and password • allocating the CBA administrator role

AdministratorManages at school level

• teacher passwords• SEN/HI Entry Points

TeacherManages at classroom level

• pupil usernames and passwords for class• preparing classroom• practising with pupils• administering assessments

Classroom assistantSupports pupils

• continuation of general classroom good practice

67

SIMS Upload to WebXchange

Role to be assigned by Principal

•Quick to complete•Complete well in advance of using assessments•Need to have:

– Instructions sent to principals (approx. 17 August)

•Does not affect use of practice assessment/area by pupils•Last date to complete is 25 September 2015

If you experience any difficulty with SIMS/Webxchange please contact the C2k Service Desk on 0870 6011666.  

68

Moving Forward with the CBA 2015

69

Admin passwords available to schools 12 August

Preparation instructions provided to schools 17 August

Manuals and posters delivered to schools 4 September

Phase 1 assessments completed 7 - 11 September

Phase 2 assessments completed 14 - 18 September

Final date for SIMS upload 26 September

Phase 3 assessments completed 28 September - 11 December

Evaluations live 7 September - 11 December

CBA Passwords

The School Administrator Username is the school DENI number.

Principals can access the new Administrator Password from ‘Principal Documents’ within the ‘MY-SCHOOL’ area of the C2k Exchange.

See CBA guidance documents page 10.

To access teacher usernames and passwords please refer to

page 11.

70

Assessment Preparation

Teachers should:

•use the online course CBA section of www.ccea.org.uk; •refer to the NILA / NINA Information for Schools Guidance Documents;•be familiar with the NILA / NINA pupil demos;•explore the NILA practice assessment and the NINA practice area;•check pupil usernames and passwords;•ensure that pupils can have a break during the assessment/s;•ensure that pupils know that NINA allows them to have the question/s read aloud if required;•consider resources required; and•check the equipment.

71

Assessment Preparation

Teachers should ensure that pupils:

• view the pupil demos as part of a class discussion;

• know how to pause or exit an assessment by clicking on the red x at the top right hand side of the screen and closing the window;

• know how to resume an assessment by clicking “Start Assessment button”. Pupil should log in again with the same username and password and see “Continue (first) assessment”; and

• understand the assessments they will be completing and have had sufficient opportunity to use the NILA Practice Assessment / NINA Practice Area.

NB - Pupils should be provided with pencils and paper for the NINA assessment.

72

Pupil Demos

Pupil Demos

•Accessible via www.ccea.org.uk

•Optional sub-titles available

•Scripts are available

NILA Practice Assessment Demo / NINA Practice Area Demo:

Introduces pupils to NILA / NINA and the techniques they need to be aware of when completing their assessment.

NILA / NINA Real Assessment Demo:

Reinforces this knowledge and prepares the pupils for logging in to complete their actual assessment.

73

The Practice Assessment / Practice Area

Be sure to:

• take your pupils through the various question/response types;

• draw attention to the instructions for each question type;

• demonstrate how to answer using a data projector or smart board;

• give your pupils plenty of opportunity to use the practice assessment (NILA) / practice area (NINA);

• show pupils how to undo/change an answer;

• evaluate how pupils cope with question types and advise; and

• show that an answer can be deleted or changed any number of times before the ‘Next’ (NILA) / ‘Submit’ (NINA) button is clicked.

NB - Pupils can access the Practice Assessment / Practice Area without usernames/password. No log-ins required.

74

Trouble-shooting

Question not fully loaded (images/buttons are misplaced/missing)

 

Step 1: If this issue continues for more than 10 seconds click on ‘F5’ once.

Step 2: If the problem persists for this user, close Internet explorer (IE) and get the pupil to log back into their assessment.

Step 3: If the problem is still not resolved, please remove this device from the assessments and use another workstation if available.

Step 4: If the problem remains, contact the C2k Service Desk on 0870 6011666.

75

Phoning C2k

Please make a note of the following information before calling the C2k Helpdesk:

•date and time of issue •assessment being attempted (NINA or NILA)•description of issue•workstation ID •pupil ‘C2k / NINA / NILA’ account user name•if possible please take a screenshot.

76

CCEA Support and Resources for Autumn 2015

• Face-to-face training (17 August – 28 September)

• Online Training Course (from 17 August)

• Videos available on CCEA website

• FAQs available on CCEA website

• Preparation instructions – SIMS upload, school username/password

• NILA / NINA Guidance Documents, including guidance for pupils with SEN

• ‘How to change an answer before you submit it in NINA’ poster

• Pupils demos and scripts

• NINA practice area

• NILA practice assessment

77

Websites and Helpdesks

78

Resources and Information for Schools•Latest information about CBA and online resources for schools on www.ccea.org.uk

Computer-Based Assessment Helpdesk

Hardware/ IT Difficulties or help with computers/laptops•C2k helpdesk 0870 6011666•C2K Exchange (desktop icon)

Monday – Friday (except for public holidays)

Email cba@ccea.org.uk

Questions, Evaluation and Close

79