Fitz-Albert R. Russell, PhD JPEstablishing Standards for Students’ Assessment

INTRODUCTION

• Psychometric assessments used for decisions about work, jobs, and careers.

• In education, students assessment used for four types of decisions.

• Administrative & policy formulation, instructional, certification, selection decisions.

• Testing & measurement applied to other persons & for other reasons-not dealt with here

BACKGROUND

• > 1,000 incidences of between Grs. 1-11.• Each is important to the child life chances• Some are high stakes & summative in nature• Examples of high stakes students’ assessment • Grade Six Achievement Test (GSAT)• CXC Secondary Education Certificate (CSEC)

VALIDITY AS QUALITY INDICATOR• Are decisions made about: possibilities, abilities,

achievements, prospects, & predisposition based on assessments appropriate for purpose?

• Three considerations:• Are testing conditions are fair to all test takers• Are scores reliable: consistent across time,

space, thought, culture variations, instrument form and test takers’ ability

• Is test valid: uses made of results are appropriate & consistent with intended use.

MESSICK’S VIEW• Validity: the pre-eminent consideration in

students’ assessment• Incorporates concerns of fairness and reliability• Drives need for the establishment of standards.• Messick (1989a, 1989b) viewed validity as

“evaluative judgment of a unitary concept that is based on evidence and consequences of appropriate test use and score interpretation”

• Professionals must recognize issues of bias & drift in items, be able to obviate them

STANDARDS IN ASSESSMENT• Classical concepts of validity do not represent

all issues of quality.• All elements in arrangements & administrative

conditions impact validity judgment, & require quality considerations.

• Quality is not necessarily JIT delivery of material• Rather, a good regime and protocol consistently

applied (Meeter 1997) is required• Standards established & rigorously applied

serve to improve the validation judgment.

TEST DEVELOPMENT PROCESS• Curriculum Analysis• Assessment planning• Item development, improvement, and validation• Compilation of assessment instruments• Preparation of first copies• Enroll test takers & prepare registers• Materials Management & Presentation• Response digitization• Mark, score, scale, equate, bias assessment• Score utilization & reporting• Process documentation.

NCME STANDARDS BY APA• The “standards” organized into three parts• Part 1: Test Construction, Evaluation, and

Documentation• Part 2: Fairness in Testing• Part 3: Testing Applications.

JAMAICAN STANDARDS: SET 1 • Relates to instrument construction & validation• Explicit curriculum referencing the assessment• Learning outcomes with assessment specification• A set of exclusive sample items;• Preliminary testing of item in final form of a test ;• Items reliability (pbis) >=.75• At least one parallel form of a test available• No item re-used within a three year period.

JAMAICAN STANDARDS: SET 2• Relates to materials management and student

processing• First copy of instrument lodged & used as

authentic paper. • Security features built into the printing,

processes and packaging;• Packages opened in assessment room, with

certification by principal; • Documentation of the process, at each step;

JAMAICAN STANDARDS: SET 3• Relates to scoring, scaling equating & analysis• Turnaround on all scores within 30 days• Appropriate scoring procedures & scaling

procedures pre-designated for open-ended/self response items

• Pre-designated procedures for dealing with missing responses

• Comparative scores reported as equated scores• Item & test analysis results documentation;• Procedures carried out by professionals

JAMAICAN STANDARDS: SET 4• Relates to appropriate documentation, reporting

& use of data• Interpretation: Appropriate pre-designated

procedure for producing standardized scores;• Reporting: documentation to inform

interpretation & judgment of validity• Documentation: assessment documented &

presented to the NCE• Utilization: Response available for individual test

takers, facilitating “value added” analysis.

JAMAICAN STANDARDS: SET 5• Relates to issues of fairness• Administration: conditions for test takers to write

tests under fair and equal conditions• Presentation of assessments: Tests may be

presented as paper based or tablet form. • Rate of work: empirically determined providing

time allowing 90 % of test takers to complete• Right of appeal: An appeal panel established

CONCLUSION I• Standards for governance & management,

accountability, quality assurance measures, & expected service delivery

• Standards proposed a) assessment instruments construction & validation b) materials management & student processing c) scoring, scaling equating & analysis d) appropriate documentation, reporting & use of data e) issues of fairness.

• Standards for high stakes assessments

CONCLUSION II• Standards represent important advance

governance in Jamaica. • Process of consultation & agreement important

as complex & technical issues are discussed, possibilities explored & expectations accepted

• Standards codified & compiled into a document along with standards for other technical and related areas of service delivery.

• Standards are about governance, management, and service delivery.

• Accountability of the Permanent Secretary.

•

• Thank you!

Fitz - A. Russell, PhD JP for Edumetrics HPT Consulting 18