Xiaming XiPresented by Chaewon Lim

Introduction To provide a means to fully integrate fairness

investigations and set priorities

- treats fairness as an aspect of validity and conceptualizes it as comparable validity for all relevant groups

- enhances the meaning of validity by integrating fairness in a principled way

Background Standard for Educational and Psychological Testing

(the 1999 Standards)

- Absence of bias, equitable treatment of all test takers in the testing process, and equity in opportunity to learn the material in an achievement test

The development of conceptual frameworks expanded the scope of fairness (Kunnan, 2000, 2004)

Empirical research that is motivated by these frameworks has been slow to catch up

Research Questions 1. How do we define fairness in a way that is

meaningful and not too abstract for practitioners?

2. What are the concrete steps to follow in investigating fairness?

Different Conceptualizations of Fairness

View1 : Fairness as an independent test quality

View2: Fairness as an all-encompassing test quality

View3: Fairness linked directly directly to validity

1. Fairness as an independent test quality The code of Fair Testing Practices in Education (the 1988

code)

- Developing and selecting appropriate test, administering and scoring tests, reporting and interpreting test results, or informing test takers

- A test quality that permeates the whole assessment process

- The primary focus is the division of responsibilities between test developers and users

The Standards for Fairness and Quality by Educational Testing Service (ETS, 2002)

2. Fairness as an all-encompassing test quality A test has to be valid to be fair

Kunnan (2000) expands on the 1988 Code and describes fairness as a three-faceted concept: validity, access and justice

The social, ethical, legal and philosophical aspects of fairness

Kunnan’s more recent work (Kunnan, 2004) – access , administration, validity, absence of bias, and social consequences

McNamara and Roever (2006) – Interrelatedness of the psychometric and social approaches

3. Fairness linked directly to validity The 1999 Standards

- Fairness as lack of bias, fairness as equitable treatment of all examinees in the testing process, and fairness as equity in opportunity to learn the materials covered in an achievement test

- providing a few types of validity evidence

Willingham and Cole (Willingham & Cole, 1997; Willingham, 1999) – comparable validity

A New Approach to Fairness Definition of fairness – fairness is defined here as

comparable validity for identifiable and relevantgroups across all stages of assessment, from assessment conceptualization to the use of assessment results.

A fairness argument in a validity argument An examination of the intended interpretations and

uses of the TOEFL iBT test scores

The validity argument for the TOEFL iBT test

1. the construction of an interpretive argument

2.the interpretive argument should be evaluated in the context of a validity argument

Chappelle et al. (2008) – typical inferential bridges

Some argument structures may also include an explicit representation of rebuttals that would invalidate or reduce the force of the claim or conclusion. This is where the fairness argument can be articulated

Inappropriate decision made about the assessment process may have effects on fairness issues pertinent to multiple inferences

Setting priorities for fairness investigations Focusing on the weakest areas in the argument would

help maximize the use of resources.

Willingham and Cole – different fairness issues are interconnected and should be considered globally.

DIF/ Fairness reviews -> finding assumption underlying a specific warrant that is susceptible to rebuttals that are specific to the fairness argument

Conclusion Fairness is characterized as comparable validity for

relevant groups that can be identified.

The argument-based structure allows us to have a coherent framework for fairness argument.

Inferential steps, rebuttals, score-based decisions, actions, and consequences