Developing a comprehensive, empirically based research framework for classroom-based assessment

Language Testing 2011Kathryn Hill and Tim McNamara

University of Melbourne, Australia

Presented by:Amirhamid Forough Ameri (ahfameri@gmail.com)and Saeed Shiri

January 2016

Abstract

This paper presents a comprehensive framework for researching classroom-based assessment (CBA) processes, and is based on a detailed empirical study of two Australian school classrooms where students aged 11 to 13 were studying Indonesian as a foreign language. The framework can be considered innovative in several respects. It goes beyond the scope of earlier models in addressing a number of gaps in previous research, including consideration of the epistemological bases for observed assessment practices and a specific learner and learning focus. Moreover, by adopting the broadest possible definition of CBA, the framework allows for the inclusion of a diverse range of data, including the more intuitive forms of teacher decision-making found in CBA (Torrance & Pryor, 1998). Finally, in contrast to previous studies the research motivating the development of the framework took place in a school-based foreign language setting. We anticipate that the framework will be of interest to both researchers and classroom practitioners.

Introduction

Previous studies have focused on issues such as validity and reliability (e.g. Gipps, 1994; Heurta-Macias, 1995), criteria and standards (e.g. Leung, 2007; Leung & Teasdale, 1997b) and the influence of external assessment and reporting regimes on classroom

practices (e.g. Brindley, 1998, 2001; Clarke & Gipps, 2000; Davison, 2004).

Relatively fewer studies, however, have focused on the actual processes of classroom-based assessment (research on which this framework is intended to guide).

Introduction

The framework we propose is based on a detailed empirical study of two Australian school classrooms where students aged 11 to 13 were studying Indonesian as a foreign language.

Although essentially a bottom-up study, an initial orientation to the research reported on here was developed from the existing literature.

In this case, two themes from the literature guided the investigation: the dimensions and scope of CBA, as well as the way these two aspects intersected with each other.

Introduction

DimensionsMcNamara (2001) sets out three critical dimensions of assessment:

evidence

use

interpretation

Introduction

According to McNamara, CBA is ‘[a]ny deliberate, sustained and explicit reflection by teachers (and by learners) on the qualities of a learner’s work’ and the use of this information, for example, ‘as an aid to the formulation of learning goals…’ (2001, p. 343).

However, the literature reveals significant diversity in how each of these dimensions is understood (Table 1).

Introduction

Introduction

The definition adopted for the study is designed to reflect this diversity and, in line with ethnographic principles, to admit all possible evidence. We thus propose the following amended definition of CBA:

Any reflection by teachers (and/or learners) on the qualities of a learner’s (or group of learners’) work and the use of that information by teachers (and/or learners) for teaching, learning (feedback), reporting, management or socialization purposes.

Note this definition of CBA incorporates both formative (or assessment for/as learning) and summative assessment (assessment of learning).

Introduction

Scope In terms of scope, the following focal research questions, reflecting

issues identified by Leung (2005) and Rea-Dickins (2006) respectively, were used to guide the empirical study:

1. What do language teachers do when they carry out classroom-based assessment?

2. What do they look for when they are assessing learners? 3. What theory or ‘standards’ do they use? 4. Do learners share the same understandings?o The relationship between scope and dimensions is set out in Table 2.

Introduction

Design of the study

As the focus is on understanding processes rather than describing outcomes, existing CBA research is essentially qualitative in nature, typically drawing on discourse-oriented social-constructivist approaches to cognition and language use.

Torrance and Pryor (1998) drew on theories of classroom interaction, constructivist theories of learning and theories of motivation and attribution to interpret their data (classroom observation and interviews).

Our study used ethnographic methods, in particular, participant observation and case studies, and a ‘grounded’ approach to analysis (Rea-Dickins, 2001).

Design of the study

Research context The study took place in two Indonesian language classrooms in Victoria,

Australia, and coincided with the introduction of a new curriculum and standards framework, VELS (VCAA, 2008).

Reasons for choosing Indonesian: Firstly, as this was an ethnographic study involving participant

observation, it was important to choose a language that the primary researcher (Hill) speaks and understands.

A second consideration was the availability of suitable programs.

Design of the study

Participants and programs One high school (Years 7 to 12) and one primary school (Preparatory to Year 6), both

government schools, were recruited for the study. The selected programs were both well established and could be considered ‘typical’

in terms of contact hours, resources, curriculum and student demographic. One Indonesian class was recruited from each year level. The Year 6 class received one hour of Indonesian instruction per week as compared to five 55-minute classes per fortnight (or 125 minutes a week) in Year 7.

One Year 6 (primary) teacher and two Year 7 (high school) teachers volunteered for the study. Each had a high level of competence in language teaching (Duff & Uchida, 1997) and a genuine interest in the research. The Year 6 teacher taught Indonesian P-6 and had taught the participating students for four consecutive years.

Design of the study

Data collection and data analysis Data collection took place in the final 10 weeks of Year 6 (end of

primary school) and the first 10 weeks of Year 72 (beginning of secondary school) (Figure 1).

Data comprised approximately 80 hours of audio-recordings (including classroom interactions and teacher and focus group interviews) as well as participant observation, field notes and documents (reports, worksheets, etc.) see Table 3.

A summary of the data set from a single unit of instruction in Year 7 has been provided as an example (Table 4).

Design of the study

Design of the study

Design of the study

Findings

RQ1 What do teachers do? Four key assessment processes were identified in the data:

planning

framing

conducting using

Findings

Planning assessment The first category, ‘Planning’, arose from an analysis of internal syllabus

documents as well as from discussions about teaching and assessment with, and between, the participating teachers.

This category captures information about the type and nature of planned assessment tasks and the relationship of assessment to instruction as well as to the relevant external frameworks

Findings

Framing assessment for students This category investigates how (or whether) learners in the respective

classrooms came to know that a given activity was ‘for assessment’, which Rea-Dickins (2006) argues is important if learners are to adopt an appropriate orientation to the task.

This category became significant precisely because assessment activities in Year 6 were embedded in teaching in a way that made the assessment dimension essentially ‘covert’ from the learners’ perspective.

Findings

Conducting assessment The processes captured by this category range from explicit, planned,

formal assessment activities to less visible, unplanned, instruction-embedded assessment activities. Table 5 sets out the terms used to distinguish between these types.

Findings Using assessment data This category classifies assessment as for ‘teaching’, ‘learning’, ‘reporting’, ‘management’ and

‘socialization’ purposes respectively. Again, while these are presented as distinct categories, analytically it is not always possible to separate one purpose from another.

Teaching. How the teachers’ informal observations of how the Year 7 class was progressing (‘incidental’, ‘group-level’ assessment) was used to inform the pace of teaching.

Learning (feedback). Assessment-related information was also used to promote immediate learning, through the provision of feedback on performance.

Reporting. the Year 7 teachers use assessment-related information to inform decisions about students’ end-of-year reports.

Management. The use of assessment for classroom management, such as controlling or reinforcing behaviour, for encouragement or for creating a positive atmosphere.

Socialization. The ‘socialization’ of learners into the local conventions of teaching and assessment.

Findings

RQ2 What do teachers look for? Sources of information about the valued enterprises and qualities and

standards operating in the respective classrooms included syllabus documents, written or verbal instructions, assessment rubrics, written or verbal feedback, report-writing meetings and written reports.

This category has been divided into: information provided ‘in advance’ of task performance, information provided in written or spoken ‘feedback’ and information provided through ‘reporting’ respectively.

Findings

In advance Figure 5 provides a segment of the instructions to students for a Year 6

speaking and writing task. It provides advance information about the obligatory components of the task (Parts A, B and C), the response format (writing and speaking), performance conditions (‘with a partner’) and weighting (15 from a total of 60 points).

However, the specified criteria (‘organization’, ‘persistence’, ‘getting along’ and ‘confidence’) refer to personal qualities rather than any feature of written and spoken language.

Findings

Findings In feedback The importance of feedback in communicating criteria and standards as well as the

strategies for achieving them (e.g. Sadler, 1989; Torrance & Pryor, 1998; Tunstall & Gipps, 1996).

In the following example the Year 7 class had been given an aural discrimination task, where they had to note how many times they heard a specific vocabulary item. Here the teacher provides information about the acceptable standard (or range of performance) for this task, that is, exactly seven times (the correct answer) or close to that number (e.g. five times).

Ok, whether you heard it five times or seven times as long as you’re in the vicinity you should be pretty happy with yourselves that you’ve heard it that many times. If you’ve written down ‘once’ or if you’ve written down ‘24’ then there’s a bit of a problem. (T2,

Year 7)

Findings

In reporting Another source of evidence about what teachers look for was provided

during the reporting process. Tunstall and Gipps (1996) suggest teachers have a ‘notion of excellence’

which they characterize as part of the teachers’ ‘guild knowledge’. This ‘guild knowledge’ informs what Wiliam (2001) has termed ‘construct-referenced’ assessment, which ‘relies on the existence of a construct (of what it means to be competent in a particular domain) being shared by a community of practitioners’ (pp. 172–173).

Findings

RQ3 What theory or standards do teachers use?A number of researchers have postulated a close relationship between teachers’ representations of the subject or content area, their pedagogic principles, and their assessment practices (e.g. James, 2006; Leung, 2005, 2007; Thomas

& Oldfather, 1997; van Lier, 2004; Wiliam, 2001). Hence this category explores articulated views regarding the subject

(Indonesian), language and language learning, and assessment, which may underlie classroom practice.

Findings

Views of the subject or content area (Indonesian) Syllabus documents provided an important source of evidence regarding how the

discipline (i.e. Indonesian) was constructed in the respective classrooms. For example, the Year 7 syllabus document (Figure 6) comprises a series of generic (culturally neutral) topics.

Beliefs about language and language learning This category investigates the theories of language and language learning that appear

to underpin teachers’ beliefs and practice. Beliefs about assessment The belief in the importance of intuition and knowledge of the students over time

rather than a reliance on formal assessment.

Findings

RQ4 Learner understandings Rea-Dickins (2006) and others have highlighted the need for further

research on assessment from a learner perspective.

Hence this category explored learner understandings regarding the nature of the subject (Indonesian, and language and foreign language

learning more broadly), as well as their notions of assessment (criteria and standards).

Findings Understandings of language learning As with their teacher, the Year 7 students attribute their friend’s competence

in Indonesian to ‘innate’ variables (ethnicity and general intelligence), rather than effort.

Understandings of assessment Previous research has found that learners often draw on their own, possibly

incongruent, understandings of task, criteria and standards (Coughlan & Duff, 1994; Moni, 1999; Torrance & Pryor, 1998).

The Year 7 teacher anticipates and expressly discourages a known propensity for students to focus on presentation at the expense of content in their written work.

Discussion: A framework for CBA research

The findings from the study can now be summarized as a framework for CBA research:

Discussion: A framework for CBA research

Conclusion

The aim of this empirical study was to understand rather than evaluate CBA practices in the respective classrooms with the aim of expanding, rather than answering, the questions that should be asked in CBA research.

There is already a volume of research evidence regarding the effects of different CBA practices on learning, not least of all that found in Black and Wiliam’s (1998) influential meta-analysis. However, there is clearly a place for experimental studies of how the different CBA processes outlined in this paper might impact on learning outcomes.

In conclusion, we anticipate that the framework will be useful for researchers interested in understanding classroom-based assessment and for teachers wishing to gain greater insight into the integration of assessment in their everyday teaching practices and the impact of their assessment practices on learning.