Achievement Exam Analysis

ED3604E/FFebruary 5/9, 2010

adapted from materials created byGerry Varty, Director of Instruction

Wolf Creek Public Schools

Achievement Exams Administered by Alberta Learning yearly to all

students in Grades 3, 6, and 9

Purpose: Serves as a ‘dipstick’ measurement Gauges general performance of a large population Measures effectiveness of curriculum and

initiatives designed to effect learning Should be used to improve teaching and learning

What they are not: An effective measure of individual student ability

An effective measure of school performance in the short-term

A ‘gateway’ or filter to direct individual student promotion.

“High-Stakes” tests. Nobody fails, nobody gets fired, no school loses funding or autonomy over student performance on these exams.

What they are: Data, through which we can see trends indicating

the effectiveness of initiatives (such as AISI) designed to improve student learning.

General measures of Program content and delivery over time

“Large-Scale” assessments – enough students write them to create a basis for consistent appraisals.

Only a piece of the larger assessment puzzle

Those distinctions are important… By using PATs for purposes they were

designed for: Government can use them to determine the

general level of attainment over time Schools can use them to gauge the

effectiveness of program enhancement, or to serve as a springboard for discussion and professional development

Parents can use them to get an idea of how their kids perform (relative to the large group) on that specific sample of outcomes.

Those distinctions are important… Using PATs for purposes they were not

designed creates issues such as: Schoolworks, Fraser Institute, etc

‘ranking’ schools (or teachers) based on aggregate PAT data

PAT data being used to determine grade or program placement for individual kids

This data is only one part of the story.

What to look for: Performance over time that indicates

strengths or weaknesses in Programs. Where there are strengths, look at those, to

seek out Best Practices which might be possible to replicate in other areas

Where there are weaknesses, they can be addressed those through collaboration and Professional Development initiatives.

Strengths of Achievement Testing

Reliable, valid data. Every student in the province writes the same exam. If others are achieving where we are not, we know that we can improve on that area.

Good fit to many curricular objectives. While they do not tell the whole story, they do tell a useful part of it.

The trick is to find the useful parts; sometimes, the very act of searching for those is an opening for great collaboration and sharing of ‘best practices’ among teachers.

PATs aren’t everything… But they’re not nothing, either.

Perhaps the most important function of PATs is to help us figure out which questions we should be asking…

‘Cause any way you cut it, if you teach the curriculum well, the PATs should take care of themselves. When they don’t, we have a problem.

Weaknesses of Achievement Testing You can’t measure everything that’s

important with these. Don’t try. Uber-Security. Not being able to sit

down with the test and look at (and discuss) how the specific outcomes are measured.

Multiple-choice. Questionable statistical practices; cut-

scores, equating, over-analysis…

This data is from one school Did the school show any changes over

this time? What conclusions can you draw, or what

possibilities can you suggest? Does anything need deeper analysis, or

ask for more information?

What else do you need to know about this school?

What do the numbers mean? In their raw form, not very much. Consider these 2 schools:

Sunshine School: 80% acceptable, 20% excellent

Gloomy Academy: 80% acceptable, 20% excellent

Which one is better?

Analyze the Data Distribution for Sunshine School:

Sunshine School

05

1015202530354045

10 20 30 40 50 60 70 80 90

Analyze the Data Distribution for Gloomy Academy

Hmm… they don’t look the same anymore…

Sunshine School

05

1015202530354045

10 20 30 40 50 60 70 80 90

Gloomy Academy

05

1015202530354045

1 2 3 4 5 6 7 8 9

They both have the same statistics…

80% Acceptable20% Excellent

But they’re so different

The problem with ‘cut scores’ Is that the range is too big. Cut scores tell us that a student

with a 54% is the same as one who scored 86%… acceptable.

Cut scores don’t tell you what kids are good at, or help you improve your instruction.

There 3 types of Assessment: Assessment of Learning Assessment for Learning Assessment for Teaching

PATs are two of these. Trying to make them fit the second purpose is a mistake; neglecting them for the third is an even greater error. If they can’t inform our teaching, then they are of no value to us.

Grading/Reporting

AssessmentEvaluation

Assessment of Learning Is what we used to call Summative

Assessment. The process of measuring the Learner’s progress

against the pre-determined standard for knowledge or performance

Reflects a judgment regarding the quality or adequacy of student achievement as determined by curriculum standards

Intent is to describe the student’s standing relative to course outcomes, at this point in time.

Grading/Reporting

AssessmentEvaluation

Proficiency & Outcomes ‘Power Outcomes’ are those critical

outcomes that demand mastery. Assessing for that takes more time and is

more difficult.

You can’t do a good job of that on a PAT … but good assessment during the year should yield concurrent results on the PAT.

Clearing the bar.. in order to ‘get’ credit for an outcome on the

PAT, you only have to answer at the difficulty level of the question.

From that, we can infer ONLY that you can also answer easier questions … for all we know, you may also be able to answer harder ones.

That fact makes ‘equating’ unsupportable; it is based on the premise that the student could not ALSO get the harder question.

Assessment for Learning Is what we used to call Formative

Assessment.

PAT’s don’t do that. One shot. End of year. No do-overs, no learning from mistakes.

But…

Grading/Reporting

AssessmentEvaluation

For kids, this isn’t necessarily a bad thing. Sometimes, our system of using tons of

accumulated marks throughout the whole year works against kids.

Purpose is everything. What are we trying to assess, what outcomes are we watching, what do we look for as evidence?

What poor assessments ‘hang around’ to mess up the final totals for the year?

How would you grade Jared? You all remember

Jared … he’s the Subway guy.

Jared lost a lot of weight by eating Subway sandwiches and walking every day.

Jared … Before and After

How would you grade Jared?

What grade would Jared get in your ‘lifestyle, nutrition, and DPA class?

(assume the ‘before’ and ‘after’ pictures coincide with the timelines for your class)

In most of our schools, Jared … would not get a great mark.

Probably, no more than 50% Our pre-occupation with calculated

‘Averages’ would ensure that at least some of his marks were low. After all, he only got to be the way he is now in the last month.

We would have to ‘average in’ all of his earlier overweight scores.

Assessment for Teaching Is another story…

This is formative assessment for us. Professional Development Program and curriculum development What can we do that impacts the

performance of our students? If you do a good job of teaching, the

achievement tests take care of themselves.