Overreach in High-Stakes Testing: The case of Chile’s National University

Admission Test (PSU)

9th ITC Conference, San Sebastian, SpainJuly 5, 2014

Mladen KoljaticMonica SilvaRichard P. Phelps

2

Topics of the presentation

1.Types of admission tests

2.The PSU context and problems

3.Lessons to be learned

4.The role of international testing associations

1. Types of admission tests

Aptitude Tests or Achievement Tests in the Context of Admission?

“ Where school inputs vary widely…academic achievement tests are likely to measure the opportunity to learn more than the ability to learn. [Developing] nations concerned about picking their future talent must consider the possibility that an aptitude test…may be more able to overcome the local differences in school quality” (Heyneman, 1987, p. 253)

SOURCE: Phelps, Standardized Testing Primer, 2007

Aptitude Tests or Achievement Tests in the Context of Admission?

“ Achievement tests are fairer to students because they measure accomplishment rather than ill-defined notions of aptitude; they can be used to improve performance; they are less vulnerable to charges of socioeconomic bias; they are more appropriate for schools, because they set clear curricular guidelines and clarify what is important for students to learn…the movement away from aptitude tests is an appropriate step for U.S. students, schools and universities” (Atkinson, 2001)

Purpose of Admission Tests

To predict college successwithout providing an unfair or unjustified advantage to any particular group of test-

takers

SOURCE: Phelps, Standardized Testing Primer, 2007

2. The PSU context and problems

School Context: 2 curriculums

Curricular Branch

Type of School

ScientificHumanistic (SH)

Technical-Professional (TP)

Municipal

Private-Subsidized

Private- Paid —

The context of Chile´s PSU

PSU focused on measuring the SH curriculum…

Students who attend the TP tracks the poorest in the nation over 90% wish to pursue tertiary education

social mobility highly related to university education in Chile

* Source: Mineduc, 2009

The context of Chile´s PSU

PSU test scores used Admission Financial aid Government subsidies to universities

very high-stakes associated to test scores

11

The context of Chile´s PSU Centralized admission system to all public funded

universities (+ some private universities) PSU is run by a department within the largest public

university (U. de Chile) Admission system requires: PSU test scores High-school GPA (HSGP) Class ranking No other criterion is considered for application to

higher education institutions

History of Chile´s PSU

From 1966-2003 : aptitude-type test (similar to SAT) From 2004- the present: PSU

Achievement tests: measure 100% of SH curriculum Mathematics Language Science (Chemistry, Physics and Biology) Social Sciences (History, Geography and Economics)

Change of tests aimed at evaluating outcomes of an educational reform

13

Project to Develop the PSU TESTS

Supported and promoted by the Ministry of Education and council of rectors of public universities (CRUCh)

Developed using public funds

Project led by an economist and a social psychologist with no training or expertise in the development of high-stakes tests

Multiplicity of Purposes for the PSU

Measure the implementation of a new curriculum Fairly measure mastery of one of the two national cu

rricula (SH) Incentivize high schools to implement the new curric

ulum Incentivize high school students to study more Predict overall success of students in universities Predict success across very different types of univers

ity programs Reduce socio-economic disparities in performance

15

“ Evidence of the test´s technical quality should be provided for each purpose“ (St.13.2)

as well as “adequate monitoring of potential negative consequences” (St.13.1)

Multiplicity of Purposes for the PSU

How the PSU Runs

Source: adapted from the Pearson Report (2013)

Council of Rectors of Chilean Universities "owners" of the PSU

designated by CRUCh as supervisors and official evaluators of the PSU PRESIDED BY CREATORS OF THE TEST

agency responsible for developing test items, test assembly, tests administration, test scoring, for CRUCh and associated universities

Main funder of the PSU since 2007.It holds the purse for CRUCh universities, since they are funded through public monies

CRUCH

CRUCH´S COMMITTEE OF

TECHNICALADVISORS FOR PSU

U. de Chile

Ministry of Education

Official Chilean Evaluations of the PSU

Eight evaluative reports from CRUCh between 2004-2010:

Pseudo-evaluations: overly optimistic reports aimed to construct a positive image of the test

Do not report any analyses for the TP group When reporting gaps, omit the TP data from the analyses

(without saying so) Deny the existence of a growing score gap between private

paid and municipal school groups, especially those attending the TP track

CRUCh denies access to data bases to independent researchers

19

International Evaluations of the PSU

ETS (2005) Pearson (2013)

20

Efforts to suppress information

2005: ETS Report not released, according to U. Chile because of secrecy clause imposed by ETS

2012: New audit. CRUCh intended: Auditors to report to expert team chosen by CRUCh …to curtail analyses for TP students... to control and edit final report

21

2007: U. de Chile sued to release ETS report

A group of faculty and students represented by a pro-transparency nonprofit organization go to court judge rules in favor of U. de Chile

Efforts to reveal information

22

Efforts to reveal information

ETS report is finally released in 2012 thanks to TRANSPARENCY LAWS

passed in the country

From 2007: 5 years of pressure by university student leaders, faculty and leaders of NGOs for new audit

2012: Funded by the Ministry of Education (maximum amount suggested by CRUCh´s CTA: US$400,000)

Pearson’s Final Report of the PSU (2013): Recommendations

PSU needs to be revised to advance equity for TP groups

reduction in contents or study alternatives for the equitable

assessment of CH and TP students PSU is below predictive standards of other

international tests belying claims from CRUCh’s experts as to the

predictive capacity of the PSU

24

Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS

Inadequate use of PSU cut scores to assign financial help and scholarships

Inadequate use of scores to pass judgment on the quality of schools

25

Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS

Faulty piloting of test items DIF analyses does not include factors as

socioeconomic status, high school curricular branch (S&H and TP)

Inadequate difficulty level of the PSU Math Test (detected in ETS’s 2005 audit)

Actual reporting of PSU Science score deemed “untenable”: Science PSU requires not one, but three scores

26

Pearson’s Final Report of the PSU (2013): SERIOUS DEFICITS

Inadequate level of analysis expected when equating a high-stakes assessment: misleading information regarding year-to-year equating and pilot calibrations. “THESE ACTIVITIES ARE NOT TAKING PLACE EVEN THOUGH THE DOCUMENTATION PROVIDED SEEMS TO INDICATE THEY ARE.” (p.43) PSU equating is below international standards

ETC…

27

Pearson’s Evaluation of the PSU (2013)…

ComprehensivePearson met with stakeholders other than CRUChIncorporated some analyses for the TP group (at no extra cost)Specified new studies that needed (e.g.,, to guide reduction of contents)

However…Recommendations are not prioritized Highly technical report, not aimed at policy makers Did not look at effects on enrollment

Pearson’s Final Report of the PSU (2013):

Revealed how easy it is to mislead the public when it comes to technical matters

CRUCh deceived the public as to the quality of the tests, in terms of prediction and fairness

CRUCH does NOT reduce contents in the PSU, against Pearson's reccomendation

CRUCh's new commitee of technical advisors (CTA) is controlled by an associate of the PSU test-makers

29

Aftermath of Pearson’s Report...

Past Problems = Present ProblemsVested interestsNo real changes in the governance systemLack of transparencyNo accountability

Main problems remain unsolved…

30

Aftermath of Pearson’s ReportStudents demand an end to the PSU elimination of the test

Only HSGPA and class ranking as selection criteriaMoratorium not only to the PSU but to other educational tests as well …

31

3. Lessons to be learned

32

Changes in tests…

Some nations that are considering moving towards achievement tests might benefit from looking at the Chilean experience to avoid a proliferation of purposes, test corruption and abuse…

Changes entail risks…

-Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test

-There is a need for a governance scheme that guarantees independent evaluations in order to protect applicants’ rights to be tested by good and fair tests

34

Data access is not only a problem in developing nations…

“The SAT item-level information is not frequently available to independent researchers who are not affiliated with ETS or the College Board…”

Source: Santelices & Wilson, 2010.

Governance Issues in Order to Prevent Test Abuse

• Test-developing agency should be professional and independent

• Account for funds received and quality of work• Report to external boards of experts that can dem

onstrate independence from the agency • Periodically audit tests for quality and fairness• Audit reports & data bases should be public

--> TRANSPARENCY AND ACCOUNTABILITY

4. The role of international testing associations

Changes in tests…

Question to ask:

How can international organizations such as ITC play a more active role in promoting good consulting practices by test experts in developing nations?

Is there a larger role for testing associations such as ITC in the promotion

of good practices worldwide?

•The ITC Guidelines on Adapting Tests•The ITC Guidelines on Test Use•The ITC Guidelines on Computer-Based and Internet-delivered Testing•The ITC Guidelines on Quality Control in Scoring, Test Analysis and Reporting of Test Scores…•The ITC Guidelines for Test Review?•The ITC Guidelines for International Consulting?

Overreach in High-Stakes Testing: The case of Chile’s National

University Admission Test (PSU)

Mladen KoljaticMonica Silva

Richard P. Phelps

9th ITC Conference San Sebastian, Spain

July 5, 2014

41

Predictive validities of the PSU(CRUCh vs Pearson estimates)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013; CTA

Predictive validities: ACT and PSU

SOURCE: ACT, Research Summary Services, 1997_1998; Pearson, Final Report Evaluation of the Chile PSU, January 2013

Predictive validities: SAT and PSU

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

44

Predictive validities: SAT and PSU(BUSINESS MAJORS)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

Predictive validities: SAT and PSU(Education)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

46

Percentage-point drop in enrollment of municipal school graduates attending two top universities

UC U. de Chile PAA (adm. 2003) 17 32 __________________________________ PSU (adm.2004) 17 29PSU (adm.2005) 16 25 PSU (adm.2006) 14 20 PSU (adm.2007) 13 --PSU (adm.2008) 11 --PSU (adm.2009) 11 --PSU (adm.2010) 12 --

Source: official institutional info

Drop in enrolled students from Municipal Schools, other CRUCh Universities

porcentaje matrïculados municipal

0

10

20

30

40

50

2003 2004 2005 2006 2007 2008

PUC

U CONCE

USACH

PUCV

Fuente: E. Simonsen, 2009

Estimated average 10% drop in enrollment from students from municipal schools at top CRUCh Universities in three years NO

measures taken…