A MULTIDIMENSIONAL APPROACH TO THE IDENTIFICATION OF TEST FAIRNESS

EXPLORATION OF THREE MULTIPLE-CHOICE SSC PAPERS IN PAKISTAN

Syed Muhammad Fahad Latifi fahad.latifi@aku.edu

Dr. Thomas Christiethomas.christie@aku.edu

- Perspective of Test Fairness- Substantive / judgmental analysis- Statistical

- Dimensionality of content.Primary dimensionSecondary dimension ( impact or bias)

- Test item measuring dimension other than primary dimension is produce differential Item functioning (DIF).

- Bundle of Test items measuring dimension other than primary dimension produces differential bundle functioning (DBF).

Multidimensionality of test item / bundle of items.

DIF/DBF can be uniform and non-uniform.

DIF / DBF can produce interesting explanations . e.g., it may be due to item format characteristics, subject matter related factors and cognitive skills measured on the test.

Males are considered as reference group (majority group)

Females are considered as focal group (minority group)

RESULTS

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0 1 2 3 4

English Mathematics Physics

Favors Males

FavorsFemales

Beta-Uni

Phase –one DIF

One item in each subject were found with severe DIF. i.e. only 3.7% of the total item pool.

Phase –one DIF (cont.):

Three Severe DIF Items

Phase One-DIF (cont.):

Item Beta-Uni Subject Content Area Required proficiency Favors14 0.100* English Reading Skills Explore implicit

meaning of a text.Male

11 0.105* Mathamatics Equations and Graphs

Plot the pairs of points to obtain the graph of a given linear expression.

Male

4 -0.150* Physics Wave, Sound and Optics

Describe the uses of ultrasound.

Female

Content and proficiency expectation for C-level DIF.

* is Significant at p < 0.05.

Subject Content Area

English

Reading Skills

Mathematics

Linear & Quadratic Equations, Inequalities and Graphs

Physics

Waves, Sound and Optics

4. Ultrasound is used for different purposes, which of the following is NOT a current use of ultrasound? A. Detection of fault in engine B. Measure the depth of an ocean C. Diagnosis of different diseases D. Ranging and detection of aeroplanes

14. According to the passage, the construction of the Pisa tower began in A. the middle of the Field of Miracles. B. white marble. C. the twelfth century. D. the period of Mussolini.

Question

11. The pair of points which lie on the straight line (AB)

is (0,0) and

A. (2,0) B. (3,-2) C. (-3,-3) D. (-2,2)

Appendix A-2

Three Severe DIF actual item text

Analogous to DIF, DBF is conceptualized as several DIF items acting in concert to produce an item bundle favoring matched examinees from one group over another, as judge by bundle score.

The term bundle indicate a set of items organized/ grouped together because they share a common content dimension, cognitive similarity or share a common item structure.

Four organizing principles are suggested in literature. 1) Test Specification, 2) Content Analysis, 3)

Psychological Analysis, and 4) Empirical Analysis.

Test Specification as organizing principle is used in this study.

Phase Two-DBF

Phase Two-DBF (cont.) :

Phase Two-DBF (cont.) :

Subject Suspected bundle No. of items Beta-Uni Favors

EnglishContent area1: Reading Skills. 12 -0.033 FemalesContent area 2: Listening Skills. 12 -0.181* Females

MathematicsContent area 5: Linear & Quadratics Equation, Inequalities and Graphs. 9 -0.141 Females

Content area 3 & 4: Coordinate Geometry, Trignometry & Theorem; Fraction, Functions and Algebraic manipulation.

20 0.301 Males

Content area 4: Fraction, Functions and Algebraic manipulation. 4 0.086* Males

PhysicsContent area 6 & 7: Electronics, ICT and Radioactivity;Electrostatics, Current and Magnetism.

17 -0.249 Females

Content area 8: Wave, Sound and Optics 7 0.132* Males

* is Significant at p < 0.05.

For each bundle analysis matching subtest was created by combining remaining items in the subject; items displaying C-level or larger DIF is ignored in all DBF analysis.For the ease of reporting, each subtest within each subject is sequenced order in which they are presented in Figure 2. For example Listening Skills is called content area 1, Reading skills is called content area 2 and so on.

Results from Differential Bundle Functioning for SSC English, Mathematics and Physic, SSC May-2011 examinations.

However, DBF is controversial due to amplification and cancellation effect.

The small item-level differences, which may go unnoticed, can be magnified when the same difference is evaluated with a bundle, also called DIF-amplification .

DIF cancellation is caused when the bundle of items exhibiting DIF against one group while another bundle of items exhibits DIF against the alternate group and therefore each is canceled out. Authors of the 1999 Standards for Educational and Psychological Testing state: “Although DIF procedures may hold some promise for improving test quality, there has been little progress in identifying the causes or substantive themes that characterize items exhibiting DIF. That is, once items on a test have been statistically identified as functioning differently from one examinee group to another, it has been difficult to specify the reasons for the differential performance…" (p. 78).

Phase Two-DBF (cont.) :

CONCLUSION

The results of this study indicates that there were only three items with Level‑C DIF in the AKU-EB`s SSC May 2011 English, Mathematics and Physics examinations.

DBF is controversial and has limited significance from practitioners’ perspective. Further, to-date , no guidelines exist to interpret the effect size measure for DBF and thus, research is needed to identify and evaluate effect size guidelines for interpreting differential bundle functioning.

Taken together, the present study suggests that the small amount of DIF found does not confound the validity of the interpretation of the examinees’ test scores on SSC examination and likewise test development practices are fair for both male and females.