evaluation report on the implementation of the incas ... report on the... · scope of the report...
TRANSCRIPT
1
Evaluation Report on the Implementation of the InCAS
computer-based assessments – September 2010
Copyright 2010 CCEA. All rights reserved.
This report is available at: www.nicurriculum.org.uk
This evaluation was conducted by the CCEA Research and Statistics Unit from
September 2009 through to March 2010. For further information about this
evaluation please contact Karen Wiltshire:
CCEA Research & Statistics Unit
29 Clarendon Road
Clarendon Dock, Belfast
BT1 3BG, or alternatively email
2
Scope of the Report
This is the final evaluation report on the phased implementation of the InCAS
computer based assessments in Years 4 to 7. Based on feedback from
primary principals, teachers, pupils and parents this report sets out the scope
for improvement. As with findings from previous research reports, outcomes of
this evaluation will be used to inform future planning and development. This
report is also intended to provide stakeholders with an overview of the overall
implementation of InCAS across the primary sector in Northern Ireland.
A full breakdown of the evaluation process is detailed in the method section of
this report.
3
Acknowledgements
CCEA would like to thank all principals, teachers, pupils and parents who
contributed to this evaluation. We are extremely grateful to all participants for
taking the time to share their experiences of the InCAS computer based
assessments. This cooperation and involvement allows CCEA to have a
detailed and collective understanding of the implementation of the new
assessment arrangements across primary schools in Northern Ireland.
Furthermore, this evaluation has provided schools, parents and pupils with an
opportunity to inform the future planning and development of InCAS.
4
Contents
Section Page
1.0 Executive Summary 5
2.0 Introduction 12
3.0 Method 14
4.0 Results 16
4.1 Principal Feedback 16
4.2 Teacher Feedback 28
4.3 Parent Feedback 48
4.4 Pupil Feedback 59
5.0 Conclusion 66
6.0 Recommendations 68
5
1.0 Summary of Findings Rationale This is the final evaluation report on the phased implementation of the InCAS
computer based assessment. In the 2009 autumn term, InCAS became the
statutory form of assessment for pupils in Years 4 to 7. This evaluation
provided all stakeholders (principals, teachers, parents and pupils) involved in
the final phase of implementation with an opportunity to present their views
and opinions on the new primary assessment arrangements. A total of 182
principals, 469 teachers, 15,838 parents and 11,117 pupils completed
questionnaires. Information collected from this evaluation has been used to
measure the success of the implementation to date and inform future planning
and development.
Key Findings Administration of InCAS in schools Overall, feedback from principals, teachers and pupils on the administration of
InCAS was positive. 93.4% of principals (n = 170) found the implementation of
InCAS manageable. Furthermore, feedback from the majority of teachers
(93.4%, n = 438) and pupils (76.4%, n = 8,484) indicates that pupils found
InCAS user-friendly. Comments from pupils who did not find InCAS easy to
complete suggest that these pupils found the subjects (Reading or General
Maths) difficult as opposed to the usability of the assessment tool.
Despite such positive feedback, over a third of principals (45.9%, n = 83) and
teachers (37.1%, n = 174) encountered challenges administering InCAS. Only
a small number of respondents highlighted particular challenges they
encountered. The three most common challenges commented upon
concerned computers freezing (25 comments), logging pupils on
(24 comments) and insufficient computer capacity (15 comments). Such
findings are consistent to those collected in the previous year’s evaluation.
However, despite such challenges, the majority of teachers who encountered
difficulties administering InCAS indicated that they felt confident at the
prospect of using this assessment tool in the future (92.5%, n = 209).
6
One such teacher stated that “the guidance is clear and if I have a problem I
know how it can be addressed”.
It is important to note that over the last three years a small, but sizeable
number of principals and teachers have highlighted that they have inadequate
computer capacity to administer InCAS. Such respondents have found
administering InCAS challenging.
Support CCEA provides schools with a variety of support and guidance in the form of;
training materials, telephone helpdesk, a dedicated section on the Northern
Ireland Curriculum website, online tutorials and an InCAS pupil demonstration.
Both principals and teachers rated all aspects of support highly. Pupils found
Inky particularly useful, with 84.5% of pupils (n = 9,390) indicating that Inky
showed them how to operate InCAS.
However, just over a third of principals (38.1%, n = 69) felt they would benefit
from further support to assist with the administration of InCAS. Principals
requested funded substitute cover (18 comments) and additional computers
(17 comments). 62.6% of principals (n = 114) revealed that classroom
assistants support schools with the administration of InCAS. A large number
of principals (79.7%, n = 145) and teachers (71.7%, n = 170) felt classroom
assistants would benefit from training on InCAS.
InCAS Outcomes This academic year (2009/10) errors affecting the General Maths age-
equivalent scores and the standardised scores were detected in some of the
outcomes given to schools. Findings from this evaluation strongly suggest that
such errors reduced stakeholders’ confidence with the reliability of InCAS
outcomes. 42.3% of principals (n = 77) did not believe InCAS outcomes
reflected pupils’ level of ability. Furthermore, only 73.9% of teachers (n = 345)
deemed the outcomes from InCAS to be consistent with their professional
judgement. This represents nearly a 20% reduction from the 2009 evaluation
responses (93.5%, n = 314). Closer analysis of feedback from principals,
7
teachers and parents by year group suggests that stakeholders principally
questioned the outcomes produced for Year 4 pupils. A number of principals
(n = 22) and parents (n = 380) felt that InCAS outcomes for Year 4 pupils were
lower than pupils’ actual level of ability.
The 2009/10 evaluation shows teachers’ confidence levels in reporting
assessment outcomes to parents was 79.5% (n = 373) this is a proportional
drop from the 2008/09 responses (91.6%, n = 308). 56 teachers stated that
they were not confident reporting outcomes to parents as they questioned their
reliability.
A sizeable number of teachers (32.2%, n = 151) and principals (28.6%, n = 52)
did not believe the outcomes from InCAS enhanced the information shared at
the parents meeting. Furthermore, 25.8% of parents (n = 3,882) did not
believe InCAS outcomes accurately reflected their child’s level of ability.
Despite such concerns, 80% of teachers who responded to the evaluation felt
that InCAS outcomes were helpful in informing the learning and teaching of
individual pupils (80.6%, n = 378) as well as overall classes (79.3%, n = 372).
Furthermore, the majority of parents responding (91.5%, n = 13,749) found the
outcomes easy to understand and felt they informed them of their child’s
strengths (85.3%, n = 13,133) and areas for development (82.3%, n= 12, 556).
Despite being covered in the teacher training sessions and the InCAS manual,
19 teachers commented that they had difficulties interpreting outcomes
whenever pupils received a * or # symbol. Teachers requested clarification on
how to interpret these scores.
Just over half of pupils (51.6%, n = 5,731) indicated that they were unsure why
they completed InCAS.
School Management Feedback from this evaluation suggests that senior management have started
to establish InCAS into schools. Nearly 90% of principals held staff meetings
8
to determine how to share InCAS outcomes with parents (89.5%, n = 162) and
to determine how InCAS outcomes could be best used in their school
(89%, n = 161). Furthermore, a number of principals indicated that they used
InCAS outcomes for target setting (73.1%, n = 133) and also within their
school development plan (60.4%, n = 110).
Statutory Requirements Findings from this evaluation reveal that a small, but noteworthy number of
schools did not meet all the statutory assessment requirements. 14.8% of
principals (n = 27) did not report InCAS outcomes from previous years.
Furthermore, 13.5% of parents (n = 2,114) did not receive their child’s
outcomes from InCAS in writing.
9
Conclusion
This evaluation reveals that the errors in the InCAS outcomes during the
2009/10 autumn term have reduced stakeholders’ confidence with the
reliability of this assessment. Feedback from respondents indicated that
respondents particularly queried the reliability of outcomes for Year 4 pupils,
with many commenting that scores were lower than pupils’ actual level of
ability. It is imperative that all educational bodies (The Department of
Education, CCEA, and CEM) provide reassurance to schools by outlining the
procedures taken to ensure the reliability of InCAS in the future.
Feedback from this evaluation also indicates that schools encountered
challenges administrating InCAS. However, only a small number of
respondents specified challenges they encountered. The manner in which
InCAS is administered varies greatly between schools, depending upon their
individual setting. This evaluation suggests that schools may benefit from
guidance outlining the ways in which schools can best manage the
administration of InCAS. Furthermore, a review of ICT provision should be
carried out to identify schools in need of additional ICT resources. Feedback
from principals and teachers also suggests that training classroom assistants
on the administration of InCAS would further support schools.
Over 50% of pupils (51.6%, n = 5,731) indicated that they were not sure why
they completed InCAS. In keeping with the aim of the “Big Picture” of the
Northern Ireland Curriculum at Key Stages 1 & 21 which “aims to empower
young people to achieve their potential and to make informed decisions
throughout their lives” pupils should be aware of why they are completing this
assessment. Furthermore, if pupils’ outcomes from InCAS are explained in a
way which is meaningful to them, they should have an understanding of their
strengths and weaknesses. Such information should enable them, in
collaboration with their teacher or parent, to formulate achievable targets for
themselves for the year ahead.
1 http://www.nicurriculum.org.uk/docs/key_stages_1_and_2/Big-PicturePrimary-KS12.pdf
10
Despite the areas for improvement highlighted above, this evaluation has
indicated that schools have found InCAS manageable to deliver; pupils have
found this assessment user-friendly and InCAS outcomes are useful to schools
and parents. Moreover, principals and teachers rated the support provided by
CCEA very highly. CCEA should continue to offer such support to ensure
schools are adequately supported and empowered to manage these new
assessment arrangements.
11
Recommendations
Prior to the administration of InCAS in the 2010/11 autumn term there are a
number of issues documented within this report that should be addressed with
a matter of urgency.
This evaluation suggests that CCEA should:
in partnership with other educational bodies put a communication strategy
in place at the start of the autumn term to rebuild confidence throughout the
primary sector on the reliability of InCAS;
continue to provide schools with guidance on the ways in which schools
can best manage the administration of InCAS;
continue to empower schools to embrace InCAS and develop expertise and
confidence in administering, analysing and making use of the outcomes
produced from InCAS;
continue to work in partnership with C2K and CEM to prevent technical
difficulties occurring during the administration of InCAS;
provide classroom assistants with training on the administration of InCAS;
liaise with schools to ensure that the purpose of the InCAS assessments is
clear to pupils;
encourage teachers to share outcomes from the InCAS assessments with
pupils and set targets with them for the year ahead;
advise the Department of Education to carry out a review of ICT resources
to identity schools with inadequate ICT provision for the administration of
InCAS;
in partnership with the Department of Education, continue to communicate
with parents on the InCAS assessments; and
provide further advice to teachers on the meaning of the * or # symbols
allocated to some pupils’ outcomes.
12
2.0 Introduction
Purpose of InCAS Assessment
InCAS is a diagnostic assessment tool in Reading and General Maths and
optional associated assessments. InCAS is designed to support schools in
identifying pupils’ strengths and areas where they are having difficulties. The
outcomes from the InCAS assessments should therefore help teachers plan
their teaching during the school year to meet the needs of pupils in their
classrooms. The assessments also provide schools with useful information for
monitoring individual pupil progress to ensure that every pupil is reaching his
or her full potential and generally, to inform self-evaluation, development
planning and target-setting.
Requirement to Carryout Diagnostic InCAS Assessments
The Interactive Computerised Assessment System (InCAS) has been
introduced to primary schools in Northern Ireland on a phased basis over three
school years, from 2007/08. In 2007/08 InCAS applied to Year 5 pupils, in
2008/09 it was extended to cover Year 5 and 6 pupils, and from the 2009/10
school year it applied to pupils in Years 4 to 7.
This academic year (2009/10) under the Education (Assessment
Arrangements) (Foundation to Key Stage 3) Order (NI) 2007 (the 2007 Order)
the Department of Education formally specified in circular 2009/142 that InCAS
was the computer based assessment method to be used for carrying out the
statutory diagnostic assessment of reading and mathematics for pupils in
Years 4 to 7 in the autumn term.
The 2007 Order stipulates that the outcomes of the specified computer based
assessments are to be recorded and notified to parents by the end of that
2 http://www.deni.gov.uk/incas_circular_to_schools_-_september_2009-2.pdf
13
term. In accordance with this Order, the Department of Education stipulated
that from the 2009 autumn term, schools are required to:
provide age equivalent InCAS outcomes for Reading and General Maths to
parents in writing during the autumn term for the 2009/10 school year, and
where available the 2008/09 school year. If they wish, schools may also
provide outcomes for the 2007/08 school year and, if parents request
outcomes for 2007/08, this information should be provided;
provide information in a graphical format where two years of InCAS
outcomes are available; and
offer a parents meeting before the end of the autumn term to set the InCAS
outcomes in an educational context as a basis for explaining to parents
their child’s strengths, and any difficulties the child may be experiencing,
and the actions the school intends to take to address these. The meeting
also provides an opportunity to help parents understand what they can do
to support their child’s learning.
Evaluation of the Implementation of InCAS
CCEA’s Research & Statistics (R&S) Unit has been responsible for the
evaluation of the implementation of InCAS. Throughout each roll out year, the
R&S Unit has provided all stakeholders involved in this process with an
opportunity to provide feedback on this assessment. CCEA has used the
feedback collected throughout the evaluation process to determine how to best
support schools to implement InCAS and further develop this assessment tool.
14
3.0 Method
The overall aim of this evaluation was to conduct a robust and reliable
evaluation of the final implementation phase of the new InCAS computer
based assessments.
In order for this evaluation to be as inclusive as possible, information was
collected from all stakeholders involved in this year’s implementation of the
InCAS assessments. This encompassed principals, teachers, parents and
pupils. Excluding special schools, there are a total of 872 primary schools in
Northern Ireland (including preparatory and independent) with an approximate
population of 91,130 pupils in Years 4 to 7 (DE statistics 2008 – 2009).
A separate evaluation has been conducted with special schools, to further
explore the manageability of InCAS for pupils with special educational needs.
Summary of Participants and Research Tools
Questionnaires were used to collect feedback from stakeholders. Principals,
teachers and pupils were asked to complete an online questionnaire.
Additional hard copies of the questionnaires were available to all on request.
Hard copies of the parent questionnaire were posted to schools for distribution.
Questions within all questionnaires were twofold. Scales were used to obtain
quantitative evidence from respondents. Comment boxes were then used to
capture the reasoning behind respondents’ answers to all dichotomous and
scaled questions.
Feedback from Principals
All primary principals (n = 870) in Northern Ireland were invited to complete an
evaluation questionnaire on InCAS. A total of 182 principals completed a
questionnaire, which represents a response rate of 20.9%. Responses were
representative of the five Education and Library Boards.
15
Feedback from Teachers
All Year 4, 5, 6 and 7 teachers required to complete InCAS (Years 4 to 7) were
invited to complete an evaluation questionnaire on InCAS. A total of 469
teachers completed questionnaires, which represents a response rate of
approximately 13.4%. Responses were representative of the five Education
and Library Boards. Feedback was collected fairly evenly across the four year
groups.
Feedback from Parents
There are approximately 91,130 (2008/09 DE statistics) parents with a child in
Years 4 to 7 in Northern Ireland, all of whom were invited to complete a
questionnaire on InCAS. A total of 15,838 completed questionnaires were
returned which represents a response rate of approximately 17.4%. Parental
feedback was evenly received from parents with children in the four year
groups required to complete InCAS and from the five Education and Library
Boards.
Feedback from Pupils
A total of 11,117 pupils across Years 4 to 7 completed a questionnaire about
the InCAS assessments. Based on these figures this evaluation has obtained
a response rate of 12.2% of all Year 4 to 7 pupils in Northern Ireland.
Feedback was collected fairly evenly across the four year groups. Feedback
was representative of pupils in Years 4, 5, 6 and 7.
16
4.0 Results collected from Principal Questionnaires
This section presents the results for each question from the Principal
questionnaire. Results are based on completed questionnaires from 182
respondents. Percentages relate to the number of principals who answered
each question, which may be fewer than the overall total of 182*. A number of
principals reiterated comments throughout their questionnaire; as a result
some comments are repeated throughout this section of the report.
Findings from this year’s evaluation were compared to that collected from the
previous year. No noteworthy differences were noted.
* It should be noted that throughout the report, figures are rounded to one decimal place. Consequently
cumulative percentages may not total 100%.
17
20.9
72.5
60.5
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Verymanageable
Manageable Notmanageable
Not at allmanageable
The implementation of InCAS
Figure 1: Manageability of this year’s implementation of InCAS
The majority of
principals
(93.4%, n=170)
found the
implementation
of InCAS
across Years 4
to 7 to be
manageable.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Despite such positive feedback, a small number of principals expressed
concerns regarding the manageability of the InCAS assessments. 16
respondents felt their school did not have a sufficient number of computers to
administer the assessments across Years 4 to 7.
Furthermore, 13 respondents stated that schools require funded substitute
cover to release teachers to administer InCAS. 12 principals indicated that
they encountered challenges organising the administration of InCAS.
18
Figure 2: School management decisions to support the
completion of InCAS
Figure 2
provides a
breakdown of
the managerial
decisions
principals made
for the
completion of
InCAS in their
school.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Figure 2 above reveals that the majority of principals (85.2%, n = 155) made
timetabling decisions for the administration of InCAS. A large number of
respondents also made decisions regarding the distribution of resources
(73.1%, n = 133) and supervision of classes (68.7%, n = 125). Over half of
respondents (51.6%, n = 94) made managerial decisions regarding the
deployment of staff.
Principals were asked to record any procedures they implemented to support
the administration of InCAS. A small number of principals indicated that they:
arranged substitute cover for the supervision of classes (20 comments);
organised the availability of the computer suite (13 comments); and
used non contact days to supervise pupils completing InCAS
(6 comments).
51.6
73.1 68.7
85.2
2.2
0
20
40
60
80
100%
of
Res
po
nd
ents
Deployment of staff
Resources
Supervision of classes
Timetabling
Other
19
45.9
54.1
0
20
40
60
80
100%
of
Res
po
nd
ents
Yes No
Figure 3: Challenges administering InCAS
Figure 3 illustrates
that 45.9% of
respondents (n=83)
encountered
challenges
administering the
InCAS
assessments.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Respondents were asked to highlight any particular difficulties they
encountered. Responses are summarised below:
insufficient computer capacity within school (15 comments);
computers / laptops froze during the assessments (8 comments);
challenges logging pupils onto InCAS (7 comments); and
unreliable internet access in school (7 comments).
20
CCEA support
Table 1: Use of CCEA support
Responses Support Number %
Training Manual 173 95.6 InCAS preparation letter 167 91.8 Resources folder 163 90.1 Inky pupil demo 160 87.9 On-line materials on www.nicurriculum.org.uk
144
79.6
CCEA helpdesk 127 70.2
Table 1 above provides a breakdown of the various modes of support
principals and their schools made use of. The majority of principals made use
of the Training Manual (95.6%, n = 173), InCAS preparation letter
(91.8%, n = 167) and Resources folder (90.1%, n = 163).
Principals had an opportunity to comment on their answer. A small number of
principals commended the usefulness of the Inky Pupil Demo (n = 14) and the
InCAS Preparation Letter 09/10 (n = 8).
Figure 4: Support provided by CCEA
Principals were
asked to rate the
support provided
by CCEA. The
response was very
positive with 94.5%
of respondents
(n = 171) rating this
support highly.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
57.5
37
50.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very good Good Poor Very poor
21
28.637.4 34.6
38.5
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Attitudes DevelopedAbility
MentalArithmetic
Spelling
Although principals rated the support provided by CCEA highly, just over a
third of respondents (38.1%, n = 69) felt they would benefit from further
support to assist with the administration of InCAS.
Principals recommended the following:
further funded substitute cover (18 comments);
additional computers (17 comments); and
more examples of InCAS questions (9 comments).
Non statutory assessments
51.6% (n = 94) of respondents administered the non statutory assessments to
their pupils.
Figure 5: Use of non statutory assessments
Figure 5
provides a
breakdown of
the non
statutory
assessments
that principals
administered
within their
schools.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Over a third of principals administered the Spelling assessment
(38.5%, n = 70), the Developed Ability assessment (37.4%, n = 68), and the
Mental Arithmetic assessment (34.6%, n = 63). 28.6% of principals (n = 52)
also made use of the Attitude assessments.
22
57.7
42.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Irish Medium Assessments
Five principals from Irish Medium schools or Units responded to the evaluation.
These respondents indicated that they administered the following non statutory
assessments to their pupils: Spelling (n = 3); Mental Arithmetic (n = 2) and
Attitudes (n = 2).
Sharing InCAS outcomes with parents Figure 6: Do InCAS outcomes reflect pupils’ level of ability?
57.7% of
principals
(n=105) agreed
that the
outcomes from
InCAS reflected
their pupils’ level
of ability.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Of the 42.3% (n = 77) who disagreed, a small number of principals expressed
concerns regarding the reliability of InCAS outcomes. Such concerns are
summarised below:
discrepancies were apparent between InCAS outcomes and pupils’ level of
ability (38 comments);
year 4 InCAS outcomes were lower than pupils’ level of ability
(22 comments);
InCAS outcomes were inconsistent with other standardised assessments
(20 comments);
pupils scored higher than expected (16 comments); and
pupils scored lower than expected (12 comments).
23
71.4
28.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Two examples of such comments are provided below.
“There were occasions when the pupils’ scores did not compare with the
teacher’s own judgement or indeed other standardised assessments.”
“Some [pupils] achieved much higher than expected and some much
lower.”
Figure 7: Do you believe reporting outcomes to parents enhanced the
information shared at the parent meeting?
71.4% of
respondents
(n=130) felt that
reporting InCAS
outcomes
enhanced the
information
shared at the
parent meeting.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
However, 28.6% of Principals (n = 52) disagreed. 25 of these principals
highlighted that parents became anxious whenever InCAS outcomes where
inconsistent with their child’s level of ability. One such respondent commented:
“For some parents the outcomes did not reflect the pupil’s capabilities or
work standards in class and so we felt they were misleading and hence
confusing for all concerned.”
In addition to reporting InCAS outcomes from this academic year, 85.2% of
principals (n = 155) provided parents with their child’s age equivalent
outcomes to InCAS from previous years. Five respondents felt that reporting
24
this information provided parents with a more detailed representation of pupils’
progress across time.
Figure 8: Charts and tables used to share InCAS outcomes with
parents
45.6
20.9 20.9
3.8
17.6 15.9
54.4
13.76.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Standard feedback table
Scores chart
Longitude chart
Difference table
Age comparison chart
Age standardised scores
Pupil progress chart
Pupil scores table
Developed own format
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Figure 8 outlines the charts and tables that principals used to share InCAS
outcomes with parents.
Feedback demonstrates that the Pupil Progress Chart (54.4%, n = 99) and
Standard Feedback Table (45.6%, n = 83) were the most popular graphical
and tabular formats of sharing information with parents.
25
30.8
60.4
5.5 3.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very effective Effective Not effective Not at alleffective
Figure 9: How effective was your chosen format in sharing
information with parents?
91.2% of
respondents
(n=166) felt that
their chosen
format of sharing
information with
parents was
effective.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Principals were asked if they felt any other format of InCAS outcomes would
be useful for parents. Six respondents felt that a standardised score alongside
an age equivalent score would be useful for parents.
Use of InCAS outcomes by school management
The majority of principals took time to decide how to best use feedback from
InCAS within their school. This is evident from the large number of principals
who held staff meetings to:
discuss how InCAS outcomes would be shared with parents
(89.5%, n = 162); and
determine the use of InCAS feedback within their school (89%, n = 161).
A large number of principals indicated that they used the feedback from InCAS
to inform target setting (73.1%, n = 133) and their school’s development plan
(60.4%, n = 110).
26
79.7
20.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Despite a large number of principals indicating that outcomes were used at a
school management level, a small number of respondents (n = 15) questioned
the accuracy of InCAS outcomes. 10 respondents were of the opinion that data
from other standardised assessments is more reliable than outcomes from
InCAS. One such principal stated that:
“Parents and teachers need to feel more confident in the results before
using them for target setting.”
Future training
Over 60% of principals (62.6%, n = 114) indicated that classroom assistants
supported schools with the administration of InCAS.
Figure 10: Should classroom assistants be trained on InCAS?
79.7% of
principals
(n = 145) believed
classroom
assistants should
be trained in the
use of InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Seven respondents commented that training would be especially beneficial for
classroom assistants who support pupils with Special Educational Needs.
27
Additional comments
Principals were given the opportunity to make any additional comments at the
end of the questionnaire.
A small number of respondents took this opportunity to express concerns
regarding the accuracy of InCAS outcomes. Such comments are summarised
below:
query the accuracy of InCAS outcomes (10 comments);
other standardised assessments are more accurate than InCAS
(8 comments); and
InCAS data errors have led to a lack of confidence in the reliability of InCAS
outcomes (8 comments).
“Found feedback difficult to locate and understand and the difficulties with
program errors this year did not help.”
“I believe there to be great value [in] the use of InCAS but the severe
problems encountered this year will have to be overcome if their potential
value is to be realised.”
A small number of respondents (n = 10) commented that schools are
inadequately resourced for the administration of InCAS. Such respondents felt
that schools require more computers (n = 6) and further funded substitute
cover (n = 4).
28
Results collected from Teacher Questionnaires (Years 4-7)
This section presents the results for each question from the teacher
questionnaire. Results are based on completed questionnaires from 469
respondents. Percentages relate to the number of teachers who answered
each question, which may be fewer than the overall total of 469. A number of
teachers reiterated comments throughout their questionnaire; as a result some
comments are repeated throughout this section of the report.
Findings from this year’s teacher evaluation were compared to that collected
from the previous year. Feedback to each question within the teacher
questionnaire have also been analysed by teacher group (Years 4 to 7). Any
noteworthy differences have been highlighted throughout this section of the
report.
29
89.7
10.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Previous experience of InCAS Just over half of teachers (51.4%, n = 241) indicated that they had
administered InCAS to their pupils in the previous academic year (2008/09).
Figure 11: Confidence administering InCAS for the second year
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
The majority of teachers who have previously administered InCAS
(89.7%, n = 227) indicated that they felt more confident administering InCAS
for the second time. One such teacher stated that:
“Having completed the assessment last year I was familiar with the
system and how it operates. Also [teachers] were prepared for any
problems that would arise.”
30
CCEA support
Table 2: Use of CCEA support
Responses Support Number %
Inky pupil demo 444 94.9 Training Manual 393 84.0 On-line materials on www.nicurriculum.org.uk
369
78.8
Resources folder 332 70.9 InCAS preparation letter 306 65.7 CCEA helpdesk 182 39.0
Table 2 above outlines the various forms of support respondents used in
preparation for InCAS. The majority of teachers used the Inky Pupil Demo
(94.9%, n = 444), Training Manual (84.0%, n = 393) and online materials
(78.8%, n = 369).
31
User issues (challenges and / or difficulties)
Over a third of respondents (37.1%, n = 174) indicated that they experienced
challenges administering the InCAS assessments.
Figure 12: Challenges experienced by teachers
Figure 12
illustrates the types
of challenges
teachers
experienced
administering
InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Figure 12 indicates that the largest percentage of respondents
(24.3%, n = 114) encountered technical difficulties administering InCAS. 78
teachers commented on the technical difficulties they experienced:
encountered problems logging pupils into InCAS (17 comments);
computers / laptops froze (17 comments);
sound failed to work (14 comments);
computers / laptops crashed (11 comments); and
unreliable internet access (10 comments).
The other main challenge highlighted by respondents was regarding
insufficient computer capacity within schools (19 comments).
24.3
5.512.4
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Technical failure Unfamiliarity withthe InCAS software
Other
32
27.9
64.6
7.10.4
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very confident Confident Not confident Not at allconfident
Figure 13: How was the problem resolved?
9.2 4.9 4.1
23
5.3 1.5 2.8
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Contacting CCEA helpdesk
Contacting C2K
Using the Training Manual
Yourself and/or colleagues
Using the online resources
Not yet resolved
Other
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Respondents who experienced problems administering InCAS were asked to
indicate how their problem was resolved. The highest number of teachers
(23%, n = 108) indicated that any problems experienced were resolved either
by themselves or with the assistance of colleagues.
Figure 14: Confidence using the InCAS technology
92.5% of
respondents
(n = 209) who
experienced
challenges
indicated that they
feel confident
about using
InCAS in the
future.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
33
29
64.4
5.80.9
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very user-friendly
User-friendly Not user-friendly
Not at all user-friendly
Completing the InCAS assessments
Figure 15: How user-friendly was InCAS for pupils?
The majority of
teachers
(93.4%, n = 438)
believed their
pupils found the
InCAS
assessments
user-friendly.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
A small number of teachers (n = 29) stated that pupils did not encounter any
difficulties completing InCAS.
“[Pupils] coped well with the whole process.”
Despite this positive feedback, 14 teachers felt that pupils with limited
experience of computers had difficulty with InCAS.
34
Nearly half of respondents (49.3%, n = 231) administered the non statutory
InCAS assessments to their pupils.
Figure 16: Use of non statutory assessments
Figure 16
provides a
breakdown of the
non statutory
assessments
teachers
administered to
pupils.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
35.6% of teachers (n = 167) administered the Spelling assessment to their
pupils, 34.8% of respondents (n = 163) administered the Mental Arithmetic
assessment, 29.6% of respondents (n = 139) administered Developed Ability
and 21.5% of teachers (n = 101) administered the Attitudes assessment.
Six teachers from Irish Medium schools or Units responded to the evaluation.
These respondents administered the Spelling (n = 2) and Mental Arithmetic
assessments (n = 2) to their pupils.
21.529.6
34.8 35.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Attitudes DevelopedAbility
MentalArithmetic
Spelling
35
31.2
58.1
91.7
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very useful Useful Not useful Not at all useful
Figure 17: Usefulness of outcomes obtained from non statutory
assessments
89.3% of teachers
(n = 209) found the
outcomes obtained
from non statutory
assessments to be
useful.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
A small number of teachers specified how they used the outcomes from the
non statutory assessments. 45 teachers used the outcomes to inform target
setting. 33 respondents indicated that they reported the outcomes from the
non statutory assessments to parents. Teachers also used the outcomes for
comparative purposes (n = 23). Specifically, respondents compared the
outcomes from the Developed Ability assessment with Reading and General
Maths outcomes (n = 4) and with pupils’ IQ scores (n = 3).
However 11 respondents expressed concerns regarding the reliability of
outcomes from the non statutory assessments.
“There was some discrepancy between expected performance and
achieved performance.”
36
18.6
62.7
16
2.8
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very confident Confident Not confident Not at allconfident
Interpreting InCAS outcomes
Figure 18: Teachers confidence accessing InCAS outcomes
81.3% of teachers
(n = 381) indicated
that they were
confident accessing
InCAS outcomes.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Respondents were asked to provide reasons for their answer. 23 teachers
commented that they were confident accessing InCAS outcomes as they had
the support of their colleagues.
17 respondents felt that accessing InCAS outcomes was straightforward, and
14 teachers stated that they were familiar accessing outcomes from previous
years.
Closer analysis of results by year group indicates that Year 4 (76.7%, n = 89)
and Year 5 (70.9%, n = 56) teachers expressed the lowest levels of confidence
accessing InCAS outcomes. This academic year (2009/10) was the first year
that Year 4 teachers were required to administer the InCAS assessments and
may explain their lower levels of confidence in accessing outcomes. However,
this is the third year that Year 5 teachers have administered InCAS and
accessed outcomes from this assessment. Year 5 teachers did not provide
specific reasons for their lower levels of confidence.
37
19.8
67.6
11.1
1.5
0
20
40
60
80
100%
of
Res
po
nd
ents
Very confident Confident Not confident Not at allconfident
7.9
66
21.6
4.5
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Veryconsistent
Consistent Not consistent Not at allconsistent
Figure 19: Teachers confidence interpreting InCAS outcomes?
87.4% of teachers
(n = 410) indicated
that they were
confident
interpreting InCAS
outcomes.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
A small number of respondents (n = 25) felt that the graphs and tables were
easy to interpret.
Nevertheless, 19 teachers commented that they had difficulties interpreting
outcomes whenever pupils received a * or # symbol. Teachers felt they
required clarification on how to interpret these scores.
Figure 20: Consistency of InCAS outcomes with teachers’ professional
judgement
73.9% of
teachers
(n = 345) felt
that InCAS
outcomes were
consistent with
their
professional
judgement.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
38
Such findings represent nearly a 20% drop from the previous year, as 93.5%
of teachers (n = 314) deemed InCAS outcomes to be consistent with their
professional judgement in the 2008/09 autumn term.
This year a large number of respondents (n = 66) noted discrepancies
between pupils’ outcomes and the teacher’s judgement of their level of ability.
Two such teachers commented:
“[Pupils’] results contradicted their performance in class. Very
demoralising for teachers, parents and pupils; and
“Whilst some feedback was consistent, there were more inconsistencies
than not”.
40 respondents felt that pupils received higher outcomes than expected. In
contrast, 34 teachers commented that pupils obtained lower outcomes than
expected. A small number of respondents (n = 34) also believed that InCAS
outcomes were inconsistent with feedback obtained from other standardised
assessments.
Use of InCAS outcomes
80% of respondents felt that InCAS outcomes were helpful in informing the
learning and teaching of individual pupils (80.6%, n = 378) as well as overall
classes (79.3%, n = 372). 36 respondents commented that outcomes
informed target setting, particularly highlighting areas for improvement for
individual pupils (n = 24).
However, 15 teachers expressed lack of confidence in the reliability of InCAS
outcomes. Consequently teachers noted that they did not find the feedback
helpful in informing learning and teaching.
39
Figure 21: Charts and tables used by teachers to share InCAS
outcomes with parents
49.7
23.519.8
8.3
31.823.9
45
20
5.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Standard feedback table
Scores chart
Longitude chart
Difference table
Age comparison chart
Age standardised scores
Pupil progress chart
Pupil scores table
Developed own form
at
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Respondents were asked to indicate which charts and tables they used to
share InCAS outcomes with parents. It should be noted that respondents were
given the option of selecting more than one chart or table.
Figure 21 demonstrates that the Standard Feedback Table (49.7%, n = 233)
and Pupil Progress Chart (45%, n = 211) were the most popular formats for
sharing InCAS outcomes with parents.
Just over 63% of respondents indicated that they drew year on year
comparisons on the InCAS outcomes at an individual pupil level (63.2%, n =
295). 15 respondents commented that they used the outcomes to track pupil
progress. A large number of Year 4 and 5 teachers (n = 58) were unable to
draw comparisons at an individual pupil level as this was the first year that their
pupils completed InCAS.
A smaller number of teachers drew comparisons on the InCAS outcomes at
class level (43.4%, n = 202).
40
16.2
63.3
16.2
4.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very confident Confident Not confident Not at allconfident
Figure 22: Teachers confidence in reporting outcomes to parents
Figure 22 illustrates
that 79.5% of teachers
(n = 373) were
confident reporting
assessment outcomes
to parents. This
represents a 12% dip
in confidence levels
from the previous year
(91.6%, n = 308).
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Data analysis of this year’s teacher feedback indicates that Year 5 teachers
had the lowest levels of confidence. Just over two thirds of Year 5 teachers
(69.6%, n = 55) indicated that they were confident reporting assessment
outcomes to parents.
A sizeable number of teachers (n = 56) across all year groups stated that they
were not confident reporting InCAS outcomes as they felt they were
inconsistent with pupils’ level of ability.
16 teachers questioned the reliability of InCAS outcomes and consequently
were not comfortable reporting outcomes to parents. One such teacher stated
that:
“As we felt the scores were inaccurate we were not confident
reporting them”; and
“The InCAS data was inaccurate. We informed parents that we have
to report the data but we have no faith in it”.
Ten respondents also stated that they were less confident reporting outcomes
due to the InCAS data errors.
41
67.8
32.2
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Figure 23: Did the InCAS outcomes enhance the information shared at
the parent meeting?
67.8% of
respondents
(n = 318) felt the
InCAS outcomes
enhanced the
information
shared at the
parent meeting.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
23 teachers commented that the InCAS outcomes provided hard evidence that
supported pupil progress.
However, a small number of teachers (n = 26) reiterated the concern that there
were disparities in InCAS outcomes and pupils’ level of ability. 15 respondents
noted that such inconsistencies caused concerns for parents at the parent
meeting:
“[Inconsistencies] seemed to confuse parents and led them to worry
about their child.”
Over 60% of respondents (61.9%, n = 289) agreed that the InCAS outcomes
helped teachers inform parents of how to support their child’s learning at
home.
Whilst 43 teachers believed that the InCAS outcomes raised parents’
awareness of areas to be supported at home, 11 respondents felt it was not
specific enough to inform parents of how to support their child’s learning.
42
Figure 24: Are age equivalent outcomes meaningful to parents?
81.6% of
teachers
(n = 381) felt that
the age
equivalent
outcomes from
InCAS were
meaningful to
parents.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Despite this positive feedback, a small number of respondents (n = 36)
commented that age equivalent outcomes are not meaningful for parents as
they were inconsistent with their child’s level of ability.
19 respondents reiterated the anxiety caused to parents whose child obtained
a low age equivalent outcome. One such teacher commented:
“It is meaningful but it can be demoralising for some parents / pupils
especially where [pupils] have difficulties but are making good
progress.”
Two thirds of teachers (67%, n = 313) felt that the graphical representation of
the outcomes were meaningful to parents. Closer analysis of teacher feedback
revealed that slightly less Year 4 (56%, n = 65) and Year 5 (58.4%, n = 45)
teachers felt the graphical representation was meaningful.
81.6
18.4
0
20
40
60
80
100%
of
Res
po
nd
ents
Yes No
43
78.7
21.3
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Across all year groups, teacher comments were contradictory. While 21
respondents felt the graphical representation provided a clear depiction of
pupil progress, other respondents (n = 21) felt that graphs required a teacher
explanation. A further 20 teachers believed graphical illustrations of InCAS
data were difficult for parents to understand.
Using InCAS with pupils with Special Educational Needs (SEN)
Figure 25: Did you administer InCAS to any pupils on the SEN register?
78.7% of
respondents
(n = 369)
administered
InCAS to pupils
on the SEN
register.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
68% of such teachers (n = 247) also indicated that they administered InCAS to
pupils with a statement of special educational need.
44
10.9
58.5
23
7.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Verymanageable
Manageable Notmanageable
Not at allmanageable
Table 3: Category(s) of special educational need
Responses Support Number %
Cognitive and Learning 184 49.9 Social, Emotional and Behavioural 107 29.0 Medical conditions / syndromes 71 19.2 Communication and Interaction 58 15.7 Physical 22 6.0 Sensory 16 4.3 Other 6 1.6
Table 3 provides a breakdown of the categories of special educational need
teachers administered InCAS to. Nearly 50% of teachers (49.9%, n = 184)
indicated that they administered InCAS to pupils with cognitive and learning
difficulties.
Figure 26: How manageable did your pupils with a special educational
need find the assessments?
69.4% of teachers
(n = 248) believed
that pupils with
special
educational needs
found the InCAS
assessments
manageable.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
45
However, almost a third of teachers (30.6%, n = 109) believed that the InCAS
assessments were not manageable for pupils with special educational needs.
Teachers commented that such pupils:
randomly clicked on answers (25 comments);
had poor concentration (18 comments);
found it difficult to understand the task involved (14 comments); and
sought assistance while completing InCAS (14 comments).
Table 4: Number of teachers who deem InCAS manageable for pupils
with special educational needs
Manageability Please indicate which year group you teach: Number % Year 4 50 58.1 Year 5 45 67.2 Year 6 39 69.6 Year 7 64 92.8 Composite classes 50 63.3
Table 4 above suggests that familiarity with InCAS may have had an impact on
the manageability of the assessments for pupils with special educational
needs. Feedback from teachers reveals that pupils find InCAS more
manageable as they progress up the school.
Table 4 demonstrates that Year 4 pupils found InCAS to be less manageable
(58.1%, n = 50) than pupils in other year groups. A much higher percentage of
Year 7 teachers (92.8%, n = 64) indicated that their pupils found InCAS
manageable. Three Year 7 teachers commented that their pupils were familiar
with InCAS from previous years, which made the assessments more
manageable for these pupils.
Over a third of respondents (34.1%, n = 124) agreed that pupils encountered
difficulties with particular InCAS modules. A large number of teachers
commented that pupils had particular difficulties with the Reading Module
46
(50 comments). 20 of such teachers stated the Comprehension module was
particularly difficult for pupils with special educational needs.
57 teachers who have previously administered InCAS highlighted that they
recognised developments made to the assessment from last year. Such
teachers believed the changes have made InCAS more manageable for pupils
with special educational needs.
Just under half of the respondents (46.7%, n = 168) indicated that pupils with
special educational needs who completed InCAS have a classroom assistant
specifically assigned to support them. 71.7% of teachers (n = 170) felt that
classroom assistants should be trained in the use of InCAS.
Figure 27: Do you require any additional support in order to assist pupils
with a special educational need in the use of InCAS?
A small
percentage of
respondents
(24%, n = 81)
agreed that they
require further
support to assist
pupils with a SEN
in the use of
InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Eight teachers felt that further guidance is required on administering InCAS to
pupils with special educational needs.
24
76
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
47
Additional Comments
Teachers provided additional comments at the end of the questionnaire.
A number of respondents expressed concerns regarding the accuracy of
InCAS outcomes. 26 teachers commented that the data errors that occurred
this year have reduced their confidence with the outcomes from this
assessment. Respondents (n = 10) also commented that data errors caused
extra work for schools to rectify. One teacher responded:
“The fact that there were errors in the calculation of the Maths
results was most inconvenient and time consuming.”
Teachers (n = 23) were concerned that InCAS outcomes are inaccurate and
that feedback is inconsistent with pupils’ ability levels (n = 18).
Teachers (n = 10) also expressed confusion regarding the number of pupils
who received a # or * symbol. Teachers stated that clarification is required on
such symbols.
12 teachers took the opportunity at the end of the questionnaire to highlight
their support of InCAS. One such respondent commented:
“InCAS is a useful diagnostic tool for confirming professional
judgement [and] arranging groups within class.”
Eight respondents also stated that InCAS is a user-friendly assessment, while
seven teachers commented on their pupils’ enjoyment of InCAS.
48
Results collected from mainstream Parent Questionnaires
This section presents the results for each question from the mainstream parent
questionnaire. Results are based on completed questionnaires from 15,838
respondents. Percentages relate to the number of parents who answered each
question, which may be fewer than the overall total of 15,838. A number of
parents reiterated comments throughout their questionnaire; as a result some
comments are repeated throughout this section of the report.
Findings from this year’s parent evaluation were compared to that collected
from the previous year. Feedback to each question within the parent
questionnaire has also been analysed by the year group in which the parents’
child is in (Years 4 to 7). Any noteworthy differences have been highlighted
throughout this section of the report.
49
Background Information
Figure 28: Breakdown of respondents by year group
Figure 28 illustrates
that parental
feedback was
collected evenly
across the four year
groups.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
The majority of parents indicated that they have access to a computer
(94.8%, n = 14,962) and the internet (95.8%, n = 14,592).
98% of respondents stated that English was their child’s first language
(98%, n = 15,491). Of the 2% who indicated that English was not their child’s
first language, Irish (63 comments) and Polish (52 comments) were the most
common additional languages recorded.
Just over 7% of parents indicated that their child has a special educational
need (7.7%, n = 1,209). The most common special educational needs
identified by parents included, Dyslexia, (n = 338) Attention-Deficit
Hyperactivity Disorder (n = 92), Autism (n = 78) and Asperger’s Syndrome
(n = 74).
50
Awareness of InCAS
Figure 29: Did your child sit the InCAS assessments during the autumn
term?
Figure 29 shows the
percentage of parents
who were aware that their
child sat the InCAS
assessment in the autumn
term.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Over 96% of parents indicated that their child sat the InCAS assessment in the
autumn term (96.3%, n = 15,202). 459 parents commented that they were
made aware of this at a parents meeting, while 426 parents indicated that their
child told them this information. However, 332 respondents stated that they
were not made aware until after their child had completed InCAS.
Figure 30: Are you aware of the purpose of InCAS?
Figure 30 illustrates that the
majority of parents were
aware of the purpose of the
InCAS assessments
(87.6%, n = 13,763).
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
51
Such parents stated that:
their child’s teacher/school informed them of the purpose of InCAS
(534 comments);
they were already aware of the purpose of InCAS (273 comments); and
they received an information leaflet on the purpose of InCAS
(162 comments).
Despite such positive feedback, a large number of parents stated they were
unaware of the purpose of InCAS (364 comments). 238 respondents felt that
there was not enough information given to parents. One such parent
commented that there was, “little to no communication from the school as to its
contents, its purpose etc.”
Figure 31: Did you receive a CCEA "Information for Parents" leaflet
about InCAS?
Figure 31 illustrates the
percentage of parents who
received a CCEA
“Information for Parents”
leaflet about InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Figure 31 indicates that the majority of respondents received a CCEA
“Information for Parents”, leaflet about InCAS (87.7%, n = 13,366). Of those
respondents who received the leaflet, 323 commented that it was informative
and clearly explained the InCAS assessment.
52
InCAS Feedback
Figure 32: Did you attend a parent meeting about your child in the
autumn term?
Figure 32 displays the
percentage of
respondents who
attended a parent
meeting about their
child in the autumn
term.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Over 97% of respondents attended a parent meeting during the autumn term
(97.2%, n = 15,330).
Figure 33: Did you receive your child's Reading and General Maths age
equivalent outcomes in writing?
Figure 33 illustrates
that the majority of
parents received their
child’s Reading and
General Maths age
equivalent outcomes in
writing
(86.5%, n = 13,596)
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
82.4% of respondents also indicated that they received a chart presenting
these outcomes (n=12,891).
53
Of those parents who received the outcomes in writing or a chart, 91.5% found
the information easy to understand (n = 13,749). 281 parents commented that
the information was explained well to them and 269 stated that the information
was straightforward to understand.
Despite such positive feedback, 205 parents indicated that they found the
information difficult to understand and 213 respondents felt the information
was too basic and that they required a further breakdown. One such parent
commented that there was, “very little information – not very helpful”.
Figure 34: Did the InCAS outcomes reflect your child’s
level of ability?
Figure 34 displays the
percentage of
respondents who felt
their child’s InCAS
outcomes reflected their
level of ability.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Over 74% of respondents believed their child’s InCAS outcomes reflected their
level of ability (74.2%, n = 11,146). 177 parents stated that the outcomes
highlighted their child’s strengths and weaknesses accurately.
25.8% of respondents (n = 3,882) did not believe their child’s InCAS outcomes
reflected their level of ability. Closer analysis of such findings indicates that a
higher proportion of parents with a child in Year 4 (37.3%) did not feel InCAS
was reflective of their child’s level of ability. Parents of Year 4 pupils
commented that they felt the InCAS outcomes were too low and consequently
did not reflect the ability of their child (380 comments). 146 respondents felt
54
the InCAS outcomes did not reflect the teacher’s judgement. Two parents
stated:
“As discussed and agreed with by our child’s teacher, the test results
did not reflect his reading ability”, and
“My child’s teacher and I were appalled at these results, we both
agreed that these did not give an accurate account of my child’s
ability at all. Very disappointed”.
Over 82% of respondents felt the InCAS outcomes helped to inform them of
their child’s: strengths (85.3%, n = 13133); and areas for development
(82.3%, n = 12,556). 235 parents commented that the InCAS outcomes
informed them of what to concentrate on at home.
Respondents who did not feel the InCAS outcomes informed them of their
child’s strengths and areas for development stated that:
the outcomes were inaccurate (435 comments);
the outcomes were not specific enough (318 comments);
outcomes did not inform them of anything new (314 comments); and
they preferred the teacher’s assessment of class work (216 comments).
Figure 35: Based on the outcomes from the InCAS assessments and
discussion with the teacher, have you undertaken any activities
to support your child’s learning?
Figure 35 illustrates the
percentage of parents
who undertook activities
to support their child’s
learning at home.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
55
Having received their child’s outcomes from the InCAS assessments, just over
half of the respondents indicated that they have undertaken activities to
support their child’s learning (55%, n = 8,354). Types of activities that parents
have undertaken include:
additional reading (2,382 comments);
extra maths (1,567 comments);
completing more work at home (363 comments); and
arranging for their child to work with a tutor (320 comments).
45% of respondents (n = 6,845) did not undertake any activities to support
their child’s learning. A large number of such parents indicated that they have
not undertaken any additional activities with their child having received the
InCAS outcomes as:
they were already supporting learning at home (739 comments);
no areas for development were highlighted (299 comments);
they deem teacher advice and/or other assessments more reliable than
InCAS (223 comments); and
they did not receive guidance on how to support their child’s learning
(130 comments).
56
Figure 36: Have the outcomes from the InCAS assessments enhanced
the information shared at the parent meeting?
Figure 36 shows the
percentage of
respondents who felt
the outcomes from the
InCAS assessments
enhanced the
information shared at
the parent meeting.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
The majority of respondents felt the InCAS outcomes enhanced the
information shared at the parent meeting (71.5%, n = 10,804). Such parents
commented that the InCAS assessment provided:
a clear way to inform parents of their child’s ability (247 comments);
an objective source of information (165 comments);
a focused discussion at the parent teacher meeting (150 comments); and
highlighted information on the child’s strengths and weaknesses (139
comments).
28.5% of respondents (n = 4,297) did not feel the outcomes from the InCAS
assessments enhanced the information shared at the parent meeting. A closer
study of the findings revealed that a higher proportion of parents with a child in
Year 4 (34.7%, n = 1,323) did not feel the results from the InCAS assessments
enhanced the information shared at the parent meeting. Such respondents felt
that the InCAS assessments provided no additional information (86 comments)
and that they preferred the teacher’s assessment of their child (85 comments).
65 respondents commented that they felt the InCAS outcomes were inaccurate
or unreliable. One such respondent commented that they “don’t think it gives a
true reflection on the child’s ability.”
57
Figure 37: Are you aware if your child encountered any difficulties
completing InCAS?
Figure 37 displays the
percentage of parents
who were aware their
child encountered
difficulties completing
InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Nearly 24% of respondents stated that their child encountered difficulties
completing InCAS (23.8%, n = 3667). Parents indicated that their child
encountered the following problems:
ran out of time (227 comments);
left out / skipped questions (159 comment);
did not understand what was expected of them as they had no practice
(120 comments); and
found the General Maths assessment difficult (119 comments).
58
Additional Comments
Parents were presented with the opportunity at the end of the questionnaire to
provide any additional comments they had on the InCAS assessments.
A large number of respondents had concerns regarding the InCAS
assessment and the outcomes it produces. Such parents highlighted that they
felt:
the outcomes were inaccurate/unreliable (963 comments);
parents were not told enough about the InCAS assessment process
(441 comments);
InCAS is unnecessary – it is a waste of money/time (244 comments);
teacher’s professional judgement and their ongoing assessment of class
work is more valuable (212 comments);
InCAS relies too heavily on ICT skills (206 comments); and
the outcomes from InCAS are not detailed enough to aid parents in
supporting their child’s learning (188 comments).
A sizeable number of parents also used this opportunity to voice their support
on InCAS. Such respondents commented that the InCAS assessment was a
useful tool (174 comments). One parent stated InCAS is “a good tool to
identify a child’s development and more importantly to identify any deficits that
need to be worked on.”
Other comments from parents are summarised below.
the results highlight strengths and weaknesses (117 comments);
InCAS should replace the transfer test (110 comments);
InCAS is a good indicator of ability (109 comments); and
it clearly shows the child’s progress (64 comments).
59
Results collected from mainstream Pupil Questionnaires
This section presents the results for each question from the mainstream pupil
questionnaire. Results are based on completed questionnaires from 11,117
respondents. Percentages relate to the number of pupils who answered each
question, which may be fewer than the overall total of 11,117. A number of
pupils reiterated comments throughout their questionnaire; as a result some
comments are repeated throughout this section of the report.
Findings from this year’s pupil evaluation were compared to that collected from
the previous year. Feedback to each question within the pupil questionnaire
have also been analysed by the year group pupils were in (Years 4 to 7). Any
noteworthy differences have been highlighted throughout this section of the
report.
60
18.1 23.2 26.232.5
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Year 4 Year 5 Year 6 Year 7
48.4 51.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Background Information
Figure 38: Breakdown of respondents by the year group
Figure 38 illustrates
that pupils who
responded to the
questionnaire were
split fairly evenly
across Years 4 to 7.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
The majority of pupils indicated that they have access to a computer at home
(92.5%, n = 10,281).
Awareness of InCAS
Figure 39: Do you know why you did InCAS?
Figure 39 shows
the percentage
of respondents
who were aware
of why they
completed
InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
61
20.9
55.5
21
2.6
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Very Easy Easy Not Easy Not at all Easy
Just over half of respondents (51.6%, n = 5,731) indicated that they were not
sure why they completed InCAS. Of those pupils who were aware of why they
completed InCAS (48.4%, n = 5,386), the majority felt it was for teachers to
see what level their work was at (n = 2244) and to assist with their learning (n
= 947). Other respondents felt that InCAS feedback identifies areas in which
pupils require assistance (n = 252) and provides parents with information on
how their child is performing (n = 66). A smaller number of pupils believed the
assessments provided information to post-primary schools (n = 66).
Figure 40: Ease of completing InCAS Reading and General Maths?
Figure 40 shows
that the majority of
pupils
(76.4%, n = 8,484)
found the InCAS
Reading and
General Maths
assessments easy
to use.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Pupils were asked to comment on their response. A large number of
respondents (n = 502) indicated that topics explored in both assessments were
covered in class. Pupils commented that they found the General Maths
(n = 211) easy and a large proportion of these respondents (n = 160) stated
that Maths is one of their strongest subjects. Other respondents (n = 204)
found the Reading assessments easy and explained that they are skilled in
this subject (n = 133). Pupils indicated that they found Inky helpful for
preparing them for the assessments (n = 234) and found the instructions very
clear (n = 172). A sizeable number of pupils (n=203) said they possess good
ICT skills and have a personal computer at home (n = 37).
62
18.8
81.2
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Despite such positive feedback, a large number of respondents commented
that they found the General Maths (n = 360) and Reading (n = 240)
assessments difficult. A small number of these pupils felt that Maths (n = 36)
and English/Reading (n = 24) were not their strongest subjects. A sizeable
number of respondents (n = 183) indicated that some of the questions in the
assessments had not yet been covered in class. 124 pupils believed that they
did not have enough time to complete the assessments.
Figure 41: Did you have any problems completing InCAS?
Figure 41 displays
the percentage of
pupils who
encountered
problems
completing
InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
The majority of respondents (81.2%, n = 9,024) indicated that they did not
encounter any problems completing InCAS. However, 18.8% (n = 2,091) of
respondents experienced problems completing InCAS. A sizeable number of
such pupils stated that they found the General Maths (n = 375) and Reading
(n = 307) assessments difficult. Pupils also complained that they had
insufficient time (n = 268) to complete both assessments. A smaller number of
respondents commented that the assessments covered areas not yet taught in
class (n = 27), disliked the voice used throughout InCAS (n = 20) and
complained that their test froze (n = 15).
63
31.2
68.8
0
20
40
60
80
100%
of
Res
po
nd
ents
Yes No
Figure 42: Do you think the InCAS assessments could be better?
Figure 42 illustrates
the percentage of
pupils who felt the
InCAS assessments
could be better.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
Over two thirds of pupils (68.8%, n = 7,645) did not think the InCAS
assessments could be better. Nevertheless, 31.2% (n = 3,465) of respondents
disagreed.
A large number of such pupils (n = 451) requested additional time to complete
both assessments. Respondents also recommended that a greater number of
questions should be included within assessments (n = 127). 87 respondents
requested a different audio voice to be used. 59 pupils recommended a more
difficult General Maths assessment whilst a similar number of respondents
took a different viewpoint and sought less difficult General Maths (n = 53) and
Reading (n = 50) questions.
A number of pupils took this opportunity to comment on the layout of InCAS
and recommended brighter colours (n = 79), bigger text throughout the
assessments (n = 52) and more pictures (n = 49). Other pupils (n = 40)
requested more subjects to be introduced to the InCAS software. 21
respondents would have liked to complete more practice questions before
each assessment and a further 21 requested a facility to change the volume
during the test.
64
84.5
15.5
0
20
40
60
80
100
% o
f R
esp
on
den
ts
Yes No
Figure 43: Did Inky show you how to do InCAS?
Figure 43 illustrates
that the majority of
respondents
(84.5%, n = 9,390)
agreed that Inky
showed them how
to do InCAS.
Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.
The majority of pupils also deemed Inky to be helpful (88.5%, n = 8,272). A
sizeable number of pupils commented that Inky helped them learn (n = 195),
had a clear voice (n = 102) and said Inky was fun (n = 41).
Despite such positive feedback, a number of respondents indicated that they
did not like Inky (n = 182) with a small number stating they disliked the voice
(n = 10). Close analysis of pupil feedback across the year groups revealed
that the highest proportion of respondents who felt that Inky did not show them
how to complete InCAS were Year 6 (15.6%, n = 453) and Year 7 pupils
(21.8%, n = 786).
65
Additional Comments
Pupils had an opportunity at the end of the questionnaire to provide any
additional comments they had on the InCAS assessments.
A large number of respondents commented that they found completing InCAS
fun (n = 1,048), believed it assisted their learning (n = 306) and would like the
opportunity to complete the assessments again (n = 142). A large number of
pupils also indicated that they found Inky helpful (n = 236).
Despite such positive feedback pupils also highlighted a number of concerns.
Some pupils found the Reading (n = 97) and General Maths (n = 58)
assessments difficult and recommended making them easier. 72 respondents
took an opposing view and recommended that the assessments are made
more difficult. 24 respondents felt that certain parts of the assessments
covered areas they had not yet been taught in school.
Pupils also took the opportunity to make a number of recommendations.
Respondents requested more time to complete the assessments (n = 213) and
indicated they would like brighter colours (n = 96) and more pictures (n = 34).
66
5.0 Conclusion
This evaluation reveals that the errors in the InCAS outcomes during the
2009/10 autumn term have reduced stakeholders’ confidence with the
reliability of this assessment. Feedback from respondents indicated that
respondents particularly queried the reliability of outcomes for Year 4 pupils,
with many commenting that scores were lower than pupils’ actual level of
ability. It is imperative that all educational bodies (The Department of
Education, CCEA, and CEM) provide reassurance to schools by outlining the
procedures taken to ensure the reliability of InCAS in the future.
Feedback from this evaluation also indicates that schools encountered
challenges administrating InCAS. However, only a small number of
respondents specified challenges they encountered. The manner in which
InCAS is administered varies greatly between schools, depending upon their
individual setting. Nevertheless, schools may benefit from guidance outlining
the ways in which schools can best manage the administration of InCAS.
Furthermore, a review of ICT provision should be carried out to identify schools
in need of additional ICT resources. Feedback from principals and teachers
also suggests that training classroom assistants on the administration of
InCAS would further support schools.
Over 50% of pupils (51.6%, n = 5,731) indicated that they were not sure why
they completed InCAS. In keeping with the aim of the “Big Picture” of the
Northern Ireland Curriculum at Key Stages 1 & 23 which “aims to empower
young people to achieve their potential and to make informed decisions
throughout their lives” pupils should be aware of why they are completing this
assessment. Furthermore, if pupils’ outcomes from InCAS are explained in a
way which is meaningful to them, they should have an understanding of their
strengths and weaknesses. Such information should enable them, in
collaboration with their teacher or parent, to formulate achievable targets for
themselves for the year ahead.
3 http://www.nicurriculum.org.uk/docs/key_stages_1_and_2/Big-PicturePrimary-KS12.pdf
67
Despite the areas for improvement highlighted above, this evaluation has
indicated that schools have generally found InCAS manageable to deliver;
pupils have found this assessment user-friendly and InCAS outcomes are
useful to schools and parents. Moreover, principals and teachers rated the
support provided by CCEA very highly. CCEA should continue to offer such
support to ensure schools are adequately supported and empowered to
manage these new assessment arrangements.
68
6.0 Recommendations
Prior to the administration of InCAS in the 2010/11 autumn term there are a
number of issues documented within this report that should be addressed with
a matter of urgency.
This evaluation suggests that CCEA should:
in partnership with other educational bodies put a communication strategy
in place at the start of the autumn term to rebuild confidence throughout the
primary sector on the reliability of InCAS;
continue to provide schools with guidance on the ways in which schools
can best manage the administration of InCAS;
continue to empower schools to embrace InCAS and develop expertise and
confidence in administering, analysing and making use of the outcomes
produced from InCAS;
continue to work in partnership with C2K and CEM to prevent technical
difficulties occurring during the administration of InCAS;
provide classroom assistants with training on the administration of InCAS;
liaise with schools to ensure that the purpose of the InCAS assessments is
clear to pupils;
encourage teachers to share outcomes from the InCAS assessments with
pupils and set targets with them for the year ahead;
advise the Department of Education to carry out a review of ICT resources
to identity schools with inadequate ICT provision for the administration of
InCAS;
in partnership with the Department of Education, continue to communicate
with parents on the InCAS assessments; and
provide further advice to teachers on the meaning of the * or # symbols
allocated to some pupils’ outcomes.