“you can’t always get what you want: accountability and limits in score reports to boards”...
TRANSCRIPT
“You Can’t Always Get What You Want:
Accountability and Limits in Score Reports
to Boards”Troy Elliott
Association of Social Work Boards
2006 Annual Conference
Alexandria, Virginia
Council on Licensure, Enforcement and Regulation
Expect the Unexpected: Are We Clearly Prepared?
Censored image of psychometrician in Mick Jagger style tight leather pants
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Who We Are
Association of Social Work Boards (ASWB)
49 states, DC, USVI, 7 provincesPublic protection missionSocial work licensure examinations—about 25,000
administrations annuallyOther services: Disciplinary databank, continuing
education provider approval, credentials registry, communication, etc.
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
The ASWB Examinations
• Five categories (Associate, Bachelors, Masters, Advanced Generalist, Clinical)
• 4 option multiple choice• 150 scored, 20 pretest• CBT• 4 hour time limit
Psychometric and administration support through ACT, Inc.
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
The Score
IRT process – no single raw score applicable to all forms
Slight variations to maintain consistent overall difficulty – anchor exam
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
The score, continued, unfortunately
So what everyone gets is the equated score, right?
If only it were that easy…
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
The score, continued, unfortunately
The Scaled Score Zone, or Ketchup vs. Catsup—the debate rages on
“70” versus “75”Where does it exist? Why does it exist?
How many English majors does it take to develop an apt simile?
70 75
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Now that the basics are covered…
Who gets what right away: Passing Candidates
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Now that the basics are covered…
Who gets what right away: Failing Candidates
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Now that the basics are covered…
Who gets what right away: Regulatory Boards
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Summary reportsEvery year, to boards
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Choose your cliché The greatest thing since sliced bread….or
something else entirely?
Our great idea: reports to schools
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
School dataWhat the schools get
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
How could anything go wrong?
• Misuse – program outcome, marketing• Requests for more data• Fingerpointing
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What does this have to do with my topic?
Reporting service highlighted relationship between regulatory boards and schools
Some boards interested in school data
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Data salad, hold the context
Requests and questions from ASWB member boards
• Subscores?• Jurisdictional pass rates?• School data shared with everyone?
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Data salad, hold the context
ASWB’s initial answers:
• No• No• No
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What’s our problem anyway?
The three Ps: philosophy, psychometrics, potential litigation
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What’s our problem anyway?
Philosophy
Whose data is it?
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What’s our problem anyway?
Psychometrics
150-item test with content areas ranging from 3 – 22 percent of the examination.
Schools—and sometimes, entire jurisdictions—with small testing populations, even smaller first-timer numbers.
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What’s our problem anyway?
Potential litigation
• Misuse/misrepresentation of data• Hiring/firing/program changes based on
poor data
Would we prevail? Probably. But at what cost?
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Coming soon to an association near you
Protection of the exam meets service to members
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
Under consideration…
Compilation of jurisdictional pass rates, with identification where allowed
All school data reported to boards (in addition to existing program)
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What did we learn?
A victim of our own success?Licensing (and the exam)more
important than ever• Fewer exemptions• More importance among employers• Recognition by educational
programs
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What did we learn?
Looking for outcomes in all the wrong places
• Test data can get caught up in political and economic pressures faced by schools
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What did we learn?Member education never stops
• Turnover not just limited to board members
• Multidisciplinary boards increase the need for education (“If they can do it why can’t you?”)
• Culture of profession regulated impacts expectations, available options
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What do we still need to learn?
Information culture – has the data train left the station?
• Expectations• “Good” data vs. “transparency”• Whose data is it anyway?
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
What do we still need to learn?
Association culture – when is service a disservice?
• Long range planning, goal identification: don’t forget overall profile
• Tough decisions: sometimes you have to make them
Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia
And you thought I’d never shut up
Troy Elliott, Communications Director
Association of Social Work Boards
400 South Ridge Parkway, Suite BCulpeper, VA 22701
800-225-6880
Email: [email protected]