quality data, sound judgements, powerful learning. sara glover – executive director, portfolio...
TRANSCRIPT
Quality data, sound judgements,powerful learning.
Sara Glover – Executive Director, Portfolio CoordinationStephen Fraser – Data, Outcomes and EvaluationOffice for Children and Portfolio Coordination
2010 NMR ConferenceMantra Erskine Beach Resort, Lorne,May 31st – June 1st 2010
The essence of accountability is looking forward, using data to inform judgments about current performance and formulate plans for reasonable actions.
(Earl and Katz, 2006)
Outline of the institute
• Working with data• Building a data story
• Break
• Building a data story contd.
• Leading change – a process to construct change theories• Engaging stakeholders• Monitoring impact
A framework for thinking and action
Engagement
AnalysisPlanning
and action
Monitoring progress
Knowledge sharing and
capacity building
School Level Report
Attitudes to SchoolStaff Opinion Parent Opinion
VCE Data Service (online by VCAA)
NAPLAN Data Service (online by VCAA)
School Performance SummaryPercentile/SFO Report
• Build data literacy across your school – data managers, data teams
• You are not your data.
• Data is half the story; context is the other half.
• Look for the main stories. Don’t get bogged down in the detail and miss the main stories.
• Don’t react to any one particular data set, for any one particular year. Always look at a range of data sets over time. Look for a consistent story.
• Beware of small numbers (e.g. in small schools). Use the raw data (number of students) or aggregate statistics (e.g. over several years).
Working with data
• Absolute scores (usually shown as vertical blue columns, e.g., mean, % “C”) are useful for;– knowing how you compare to some standard (e.g. the VELS),– monitoring over time (i.e. are the scores increasing, declining or showing
no clear pattern) and– specifying targets.
• Percentiles (usually shown as horizontal orange bars) are useful for;– comparing your own performance to other schools, and – identifying strengths and weaknesses.
Interpretation Advice (… continued)
Always remember how the data are constructed
1
1.5
2
2.5
3
3.5
4
4.5
52
00
7
20
08
20
09
20
07
20
08
20
09
20
07
20
08
20
09
20
07
20
08
20
09
20
07
20
08
20
09
20
07
20
08
20
09
20
07
20
08
20
09
20
07
20
08
20
09
Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10 Yr 11 Yr 12
Teacher Effectiveness
1=
stro
nlg
y d
isa
gre
e &
5=
stro
ng
ly a
gre
e
0th percentile
100th percentile
10th percentile
90th percentile
50th percentile
Targeting low-SES students vs Targeting low performing students Source: Masters (2009) using PISA data
A comparison of NAPLAN scores for low SES students and all students
0%
2%
4%
6%
3
127
181
219
250
277
302
324
346
367
387
408
429
452
477
506
544
608
NAPLAN scaled score
pe
r ce
nt
All
Low SES*
* "low SES" defined as occupation of parent is … machine operator, hospitality staff, assistant, labourer or related worker, or not in paid work in last 12 months
Leading change: using data to engage people in the change process
• Build a data story• Develop a focus• Dig into the detail• Examine practice – what are we doing now – how do we know if it’s
working?• Develop a change theory – what would happen if we did things differently?• Engage stakeholders• Do the work• Monitor impact
Building a data story along a developmental continuum
Engagement
AnalysisPlanning
and action
Monitoring progress
Kindergarten Participation in our community
Early Childhood Community Profile
Kindergarten Participation rate for 4 year olds across the LGA: 91%
• Less than the metropolitan average• Less than the rural average
Children’s development across our community (AEDI)
Developmental Vulnerability• Physical Health and
Wellbeing: 9%• Social Competence: 10%• Emotional Maturity: 8%• Language and Cognitive:
13%• Communication and General
Knowledge: 24%
Early Years Reading
English Online Interview• High proportions of
students in the lower bands
Assessment of Reading Prep – Year 2
• - declining Prep outcomes (improving towards Year 2)
Reading Prep to Year 6 - VELS
• Mismatch with EOI and with AoR• Greater distribution of outcomes towards Year 6 – more at the top,
more at the tail
Reading Prep to Year 6 - NAPLAN
• Improvement over the last two years• Greater distribution of outcomes towards Year 6 – more at the top, more
at the tail (consistent with VELS)
Reading Year 7 to 10 - VELS
• Decline over three year period
• Greater distribution of outcomes towards Year 10 – more at the top, more at the tail (consistent with Prep to Year 6)
Numeracy Year 7 to 10 - VELS
• Mixed over three year period
• Improvement at Year 10
Reading Year 7 to 10 - NAPLAN
• Improvement over last two years• Greater proportions of students in the lowest two bands by Year 10
(somewhat consistent with VELS)
Year 7 Reading Year 9 Reading
Numeracy Year 7 to 10 - NAPLAN
• Improvement over last two years• Stronger than Literacy• Lower proportions of students in the lowest two bands by Year 10
(somewhat consistent with VELS)
Year 7 Numeracy Year 9 Numeracy
Post-Compulsory Outcomes - VCE
• Decline in English• Increase in Maths (very strong)
• Overall (allstudy) steady improvement
Post-Compulsory Outcomes - VET
• High participation• High completion
Post-Compulsory Outcomes - VCAL
• Low participation• High completion
Transitions and Pathways
• Excellent and improving retention across the board
7-10
10-11
11-12
7-12
Transitions and Pathways
• Overall – excellent proportions of students continuing with their education or training
• Declining University enrolments.
Engagement and Wellbeing
Strengths• Motivated students• Positive teacher-learner
relationships
Opportunities• Stimulating learning – high
demand content – lacking• Safety an issue link with
distress
• Attendance excellent.
Climate
Strengths• High Expectations• High motivation and
goal alignment
Opportunities• Collegiality and decision
making
Building a data story
• We have lower kinder participation rates• We have vulnerability in our community – language/general
knowledge/communications• Literacy a challenge from the outset• We are seeing a decline in literacy outcomes at most transition points• Strong mathematics performance• We have very strong retention and pathways through to further education
and training, with declining University enrolment.
• We have engaged and motivated students – is the level of task challenge lacking?
• We have motivated teachers with high expectations• Student and staff alike have a lack of say
Out
com
esE
xper
ienc
es
Dig into the detail
• What can we learn about mathematics instruction in the Secondary years – about the students undertaking further maths in VCE
• NAPLAN Data Service – Item Analysis Report (SREAMS SPA).
Developing the plan…examining practice
Engagement
AnalysisPlanning
and action
Monitoring progress
Examine practice – what are we doing now
• What are our relationships with kindergartens like?• What are our transition arrangements/plans like? Are teachers getting
the right information about literacy at the start of each year?• What level of involvement do teachers and students have in decisions
that affect them?
Developing a change theory
• Deciding what to do.
• The most important, but often overlooked aspect of these theories is that they should be falsifiable – that is, they can be proven false.
If we do …x , then we expect to see …y.
Monitoring progress…communicating successes and challenges against what we set out to do
Engagement
AnalysisPlanning
and action
Monitoring progress
Monitoring Impact
• If the data doesn’t really capture what you’re trying to achieve in your school, target on the behaviours that will.
Behaviour Impact Data (if any)
If …
then …
If …
then …
If …
then …
If … x
then … y
Engaging stakeholders – building ownership
• Thinking about the traction achieved by engaging stakeholders vs going it alone
• A question of how to harness the enthusiasm/effort/ownership of stakeholders at all levels
• And what information/communication tools can be used to do so
Engaging stakeholders
Key Messages Communication Tool
School Leaders
Classroom teachers
Parents
Students