Using the ISA Interactive Diagnostic Report The ISA Interactive Diagnostic Report helps school leaders and teachers to interpret and use the ISA data to inform improvements in teaching and learning. The Interactive Diagnostic report shows your school’s ISA results in a range of graphic displays which makes it easier to identify trends and patterns for your school in comparison with all other ISA schools and to interpret diagnostic information. This report is part of the ISA standard reporting package and is provided to schools with more than 10 participating students at any ISA grade level. It is a Microsoft Excel 2007 (PC) document, delivered online via your school’s ISA Online Results secure web page. This report does not provide an analysis of student performance over time. Please refer to the ISA Interactive Tracking Report to monitor the performance over time of individual students and of different groups of students within your school. The ISA Interactive Diagnostic Report helps school leaders The ISA Interactive Diagnostic Report provides summary statistics for your ISA results. Pull-down menus allow the graphs to be instantly customised, by grade level, for reading, mathematical literacy and writing. You can visually compare your school’s mean scale scores and scale score distributions by grade with all other schools and with other ‘like’ schools for the current year of the report and over time. You can also compare mean scale scores for subgroups of gender and English language background. The graphic displays of summary statistics are ideal for presenting to school boards or councils to report on your school’s ISA performance. Example of an Interactive Diagnostic Report summary statistics graph

The example of a summary statistics graph above shows the distribution of scale scores for your school (Mount ISA in this demo) and all other ISA schools by grade level for reading. A pull-down menu allows you to instantly customise the graph to show reading, mathematical literacy, Writing A (narrative writing) or Writing B (expository writing) scores.

2

Summary statistics graphs in the ISA Interactive Diagnostic Report Compare your school with all other ISA schools for the current year using: • A box and whiskers plot showing the scale score spread (minimum, lower quartile, median, upper quartile,

and maximum) by grade – all grades on one graph with the average score of a grade plotted as a dot on the box plot (see example above);

• A box and whiskers plot showing the scale score distribution by subgroups (gender, or English language

background) – one grade per graph;

• A bar chart showing average percentage correct by the framework classifications of the item (Aspect, Text Format, or Text Type for reading, and Process or Content for mathematical literacy) – one group of items, one grade per graph; and

• A bar chart showing item percentage correct for each grade – one grade one graph.

Compare your school with all other ISA schools for the last three years using: • A box and whiskers plot showing the scale score spread (minimum, lower quartile, median, upper quartile,

and maximum) for the last three years by grade and domain – one grade per graph. Perform drill-down performance analysis for your school using: • A dot chart showing the minimum, mean, the maximum scale scores for each class within a grade - one

grade per graph; • A dot chart showing the students’ scale scores for each class within a grade - 25 students per graph

(classes are shown with more than 2 graphs if needed);

• A bar chart showing the average percentage correct for different kinds of items according to their framework classifications for all classes in a grade – one group of items, one grade per graph;

• A bar chart showing item percentage correct for each class in a grade, and ordering the items by item

difficulty within a group of items – one class, one group of items per graph; and

• A response matrix chart showing the scored responses for each student by item classification – one class one group of items per graph.

The ISA Interactive Diagnostic Report helps classroom teachers The ISA Interactive Diagnostic Report provides diagnostic information for classroom teachers in graphic formats with pull-down menus that allow instant customisation of the class reports to show selected groups of items. These graphs help teachers to quickly identify overall weaknesses and strengths in the performance of a class in particular aspects of reading, mathematical literacy and writing and to also identify which students might require extra support or extension work. Example of an ISA Interactive Diagnostic Report diagnostic graph

In this example of a diagnostic graph, you can view the percentage of correct answers for all the Grade 3 reading items that address interpreting in a class, and for all other ISA Grade 3 students. You can immediately see which

3

interpreting items were harder or easier for this class compared with everyone else. The pull-down menu allows this report to be instantly customised to view each of the reading aspects, text types and text formats. A similar graph is available for mathematical literacy which can be customised by content aspects and each of the processes. Example of an ISA Interactive Diagnostic Report diagnostic table

This example of a diagnostic table is included on the display with the diagnostic graph shown above. The table shows how students in a class performed on a group of items in comparison with all other ISA students. The items in this table are ordered by item difficulty, with the easiest items located to the left side. The ordered item table allows teachers to pin-point items in which their students demonstrated strength or weakness in comparison to other ISA participating students. The table shows which items displayed in the graph that each student answered correctly (shown in green). This allows teachers to identify which students have contributed to mean class scores of note, such as those that are particularly high or low relative to all other ISA students. In this example, the students are ordered by their proficiency level. This can be easily done by clicking the “sort by band” button. With students arranged by their proficiency level and items ordered by item difficulty, this diagnostic report allows teachers to observe overall students’ response pattern which shows mostly correct responses (in green cells) towards top left of this table and mostly incorrect responses (in brown cells) around the bottom right. The box at the bottom of the example is a pull-down menu for the descriptors. When teachers identify an item that is of concern, they can use this menu to see the core skills that this item addresses. Two buttons at the bottom right of this example allow teachers to sort the students by either proficiency level or by student name. Diagnostic graphs in the ISA Interactive Diagnostic Report Compare your class with all other ISA students at this grade level using: • a dot graph showing the scale scores for each student in your class and the mean scale score for this grade

for your school and all others for reading, mathematical literacy, narrative writing and expository writing; • a bar graph showing the percentage of correct answers for all other ISA schools and your class for each

item by mathematical literacy content area and process and by reading aspect, text type and text format, a table showing correct, incorrect and missing responses for each student and a list of descriptors; and

• a bar graph showing the overall writing score for each student and the component scores for each of the

writing criteria.

4

How to use this Report The report is provided in the form of an interactive spreadsheet. By clicking on the tabs at the bottom of the spreadsheet it is possible to view the eleven different displays available. The second tab gives a summary of the different displays. This summary is replicated here:

Gr_Math Analyses the Mathematical Literacy domain, with the item (question) as the focus. Displays each item ranked either by item ID, by ISA percentage correct or by item group for the grade at your school. Highlights individual items, and groups of items. The percentage correct for All Other Schools for each grade is included for comparison.

Gr_Read Analyses the Reading domain, with the item (question) as the focus. Displays each item ranked either by item ID, by ISA percentage correct or by item group for the grade at your school. Highlights individual items, and groups of items by Aspect, Text Format and Text Type. The percentage correct for All Other Schools for each grade is included for comparison. The remaining pages provide detailed descriptions of how to interpret the information presented in each of the worksheets.

Glossary

Lists abbreviations and explains some terms and statistical concepts used in the spreadsheet.

ISA_Scale

Displays the ISA and school’s mean scale scores for all domains and in all grades. Also shows the spread (minimum, maximum, median) of scores for each domain.

Long_Scale

Displays the spread (minimum, maximum, median) of scores in your school and schools with the similar percent-age of English speaking background students (known as ‘Like’ schools) over time.

Like_Scale

Looks at subgroup performance: in all other ISA schools, in all other like schools and at your school.

Classes

Displays the means scale scores in all other ISA schools, all other like schools and in your school. Displayed too are mean scale scores and its spread (minimum, maximum) for each class in your school.

Students

Displays the mean scale score for each class in your school and all students’ scale scores for each class, by do-main and grade.

Snapshot

Analyses the Reading and Mathematical Literacy domains, with the item group as the focus. Display the average percentage correct for items grouped (e.g. by Aspect, Text Format and Text Type for Reading). All ISA schools, your school and all the classes in your school for each grade are included for comparison.

Cl_Math

One sheet per grade level. Shows class result for each Mathematical Literacy question as a percentage correct, compared with percentage correct for all other students at the same grade level. Questions are grouped by Proc-ess and Content. Question descriptions and students' proficiency level are displayed. The class result can be ranked either by students' proficiency level or students' name.

Cl_Read

One sheet per grade level. Shows class result for each Reading question as a percentage correct, compared with percentage correct for all other students at the same grade level. Questions are grouped by Aspect, Text Format and Text Type. Question descriptions and students' proficiency level are displayed. The class result can be ranked either by students' proficiency level or students' name.

Cl_Write

Displays each student's component raw scores and proficiency level for Writing A and Writing B by class, as well as the class mean (for classes with more than 10 students). The class result can be ranked either by students' pro-ficiency level or students' name.

5

This shows how the performance of students in this school is compared to the performance of students in all other schools at the same grade levels. In this example, the domain is reading, and the table shows the number of students (N) that had valid scores in the ISA testing programme, the mean (M) and standard deviation (SD) of the scale scores at each grade level in this school and in all other schools. The information for this school is displayed on the right hand side. This information is being compared to data from all other schools. The graphs in this example show the spread of scale scores in each grade of this school, side-by-side with the performance distribution from all other ISA schools. The graphs show that the average scale scores in this school have become progressively higher as the grade level increases (as one would expect). The graphs also show that the students in this school were performing at a similar level to that of ‘All Other schools’.

ISA Scale Displays scale score spread (minimum, lower quartile, median, mean, upper quartile, and maximum) for each do-main and in all grades. Also shows score spread of all other schools for comparison.

6

This shows how the performance of students in this school compares to the performance of students in ‘All Other Schools’ and to the performance of students in ‘Other Like Schools’ at the same grade level in the last three years. In this example, the selected grade is Grade 5, and the domain is reading. The information from the current year is displayed on the right hand side. This information is being compared to data from last year and from two years ago. The data from two years ago appears on the left hand side. The graphs in this example the most recent results suggest that Mount ISA Grade 5 students are performing better than the students two years ago at the same grade in this school. The graphs show that Grade 5 students in this school were performing at a relatively similar level to ‘All Other Schools’ and ‘Other Like Schools’ in the last three years.

Long Scale Displays the score spread (minimum, lower quartile, median, upper quartile, and maximum) and comparison group scores over time.

7

This shows the subgroup performances (i.e. boys vs. girls, and English speaking background vs. non-English speaking background) in this school, and how they compare to the performance of students in ‘All Other Schools’ and ‘Other Like Schools’. In this example, the selected grade is Grade 7, and the domain is reading. This example shows that in this school, the average score of girls is higher than the average score of boys at Grade 7 in the reading test. The girls in this school are performing at a similar level with the girls at ‘All Other Schools’ and ‘Other Like Schools’. This example also shows the average score of the students from non-English speaking backgrounds is higher than those from an English speaking background in this school.

Like_Scale Looks at subgroup performance: in All Other Schools, in Other Like Schools; and at your school.

8

This shows the difference in scale scores among classes for a particular grade. In this example, the selected grade is Grade 8, and the domain is Reading. This example shows that among three classes in grade 8 in this school, class “8A” performed better than the other two classes. It also shows that the average scale score for Grade 8 reading in this school is at a level above the average performance in ‘All Other Schools’ and ‘Other Like Schools’.

Classes Displays mean scale scores and spread (minimum and maximum) for each class at your school. Displayed too, are the means for all classes in the same grade in All Other Schools, Other Like Schools, and in your school.

9

This shows the difference in scale scores among students for a class. The dots in the chart represent individual student scale scores. In this example, the domain is reading, the grade is Grade 8, and the first 25 students in the class “8A” are selected (8 8A Page 1). The language background (‘N’ for non-English speaking background and ‘Y’ for English speaking background) and gender (‘M’ for male and ‘F’ for female) of each student are displayed in the table to the right of the graph. This example shows that the ‘Student 585’ had the highest reading scale score among the 25 students, and that ‘Student 601’ had the lowest scale score. The majority of students in this example had reading scores in the range of 400 to 650.

Students Displays all of the student scale scores for each class, by domain and grade.

10

This compares the students’ performance by item type among classes for a particular grade. The performance is measured in terms of average percentage correct among the items of the same framework classification. In this example, the domain is reading, the grade is Grade 3, and the items are grouped by Text Type. In this example this school has four classes in Grade 3, i.e. ‘3A’, ‘3B’, ‘3C’ and ‘3D’. The items in grade 3 are classified into one of seven text types: Argument, Diagram, Exposition, Instruction, Narrative, Poem, and Table. This example shows that Grade 3 students in this school performed at a similar level with the ISA benchmark in the items about Narratives, Poems, and Tables. On the other hand, the Grade 3 students in this school performed below the ISA benchmark for the text types Argument, Diagram, Exposition, and Instruction. This example also shows that class ‘3C’ performed better than the other three classes for all text types except Diagram, and that class ‘3D’ performed below average for nearly all text types.

Snapshot Displays average percentage correct for each class, by item type (e.g. Reading items may be grouped by Aspect, Text Format or Text Type). The chart shows the average percentage correct for all classes in a grade for each item type. Displayed too, are the percentage correct for all classes in the same grade in All Schools and in your school. This report allows teachers to understand the strengths and weaknesses of each class by sub-areas of Reading and Mathematical Literacy.

11

This compares the students’ performance on all items in group of items for each class. In this example, the domain is reading, the class is the ‘7A’ in Grade 7, and the items shown are those classified as "interpreting" items (16 items in total). (Note the partial credit item 26 is displayed once for each score point, i.e. 26.1 and 26.2). This example shows the items ordered from easiest to most difficult on the chart above. For each item, the percentage correct for class ‘7A’ is displayed on the right, and it is compared to the percentage correct of all students on the left. Among these items, class ‘7A’ students performed better than the benchmark on items 1, 8, 9, 20, 30, and 32. On the other hand, this class performed below the benchmark on items 5, 14 and 17. The response matrix chart shows the students’ responses on each item in a group of items. A correct response is coloured in green, an incorrect response is coloured in brown, and a missing response is coloured in white. In this example, the students are ordered from high ability to low ability. (This can be changed by clicking the ‘Sort by band’ button). This example suggests that the students with proficiency bands below 4 may need support to master the skills represented by the relatively difficult items 17, 32 and 26.2.

Cl_Read (Cl_Math and Cl_Write are also provided in your ISA Diagnostic Report) One sheet per grade level. Shows the class result for each Reading question as a percentage correct, compared with percentage correct for all ISA students at the same grade level. Questions can be grouped by aspect, text type, or text format. Question descriptions and students' proficiency level are displayed. The class result can be ranked either by students' proficiency level or by students' name.

12

This shows how the students performed on each item at a particular grade. It compares the performance at item level between this school and all students at the same Grade level. In this example, the domain is reading, the grade is Grade 9, and the items have been organised by Aspect. Reading items are classified into one of three Aspects: Interpreting, Reflecting, and Retrieving Information. In the chart above, the Reflecting items, which require students to make links between what they read in a text, and outside knowledge, are highlighted in yellow. The percentage correct of the Grade 9 students in this school is shown by the bar in the chart. The dotted line on the chart represents the percentage correct of all students. In this example, the Reading items in Grade 9 are ordered by item difficulty. This can be achieved by clicking the second button located at the right hand side of the chart. In this example, the chart shows that the Grade 9 students performed above the benchmark on items 8, 13, 18-19, and 25-28. On the other hand, the chart also shows that the Grade 9 students performed below the benchmark on items 5, 11, 20-21 and 31. The descriptions of these items are displayed in the table below the graph. These descriptions can be used to identify skills for which students in the class may need further teaching.

Gr_Read (Gr_Math is also provided in your ISA Diagnostic Report) This graph shows analyses of the Reading domain, with the item (question) as the focus. It displays each item ranked either by item ID, by item difficulty, or by item difficulty within each item classification. You can select and highlight individual items and groups of items by Aspect, Text Format and Text Type using the pull-down menus. The percentage correct for ‘All Other Schools’ for each grade is included for comparison.

13

The second chart shows the same set of items ordered by item difficulty within each Reading Aspect: Interpreting, Reflecting, and Retrieving Information. This is done by clicking the last button located at the right of the chart. This chart allows teachers to understand the strengths or weakness of students’ performance at predefined sub-areas of a construct, and to obtain detailed information at the item level. In this example, the Reflecting items are highlighted. This example shows that the Grade 9 students performed better than the benchmark on Reflecting items 16 and 25, but they performed below the benchmark on Reflecting item 31. This example also shows that the Grade 9 students did not exhibit any systematic weakness on any one Aspect.

Further information

Email isa@acer.edu.au Phone +613 9277 5555 or

Website www.acer.edu.au/isa