1 examination accreditation program put to the test: exam development made easy april 26, 2008...
TRANSCRIPT
![Page 1: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/1.jpg)
1
Examination Accreditation Program
Put to the Test: Exam Development Made Easy
April 26, 2008Pasadena – California -USA
![Page 2: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/2.jpg)
2
Put to the Test: Introducing the Presenters
Critical Issues
Lawrence J. Fabrey, Ph.D.
Content DevelopmentTadas Dabsys
Writing Exam QuestionsJames F. Fryer, Ed.D., CPCU
Performance Analysis Nikki Eatchel M.A.
![Page 3: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/3.jpg)
3
Critical Issues in Examination Development
• Validity • Reliability • Fairness • Legal Defensibility
• Standards
![Page 4: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/4.jpg)
4
Critical Issues: Validity
Chapter 1 in the Standards– Standards for Educational and Psychological
Testing (AERA/APA/NCME, 1999)– 24 separate standards
Also in Chapter 14 – Testing in Employment and Credentialing– Especially 14.8 through 14.10: Explicit definition,
link between job and test content, rationale basis
![Page 5: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/5.jpg)
5
Critical Issues: Validity
We cannot claim an examination is “valid” “Links in the chain of evidence used to support
the validity of the examination results” Links include:
– Job Analysis, Test Specifications, Item Writing, Standard Setting, and Examination Construction, Administration and Scoring.
![Page 6: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/6.jpg)
6
Critical Issues: Validity
Traditional Sources of Validity Evidence• Content• Criterion Related (Predictive or Concurrent)• Construct
Now -- Validity is a unitary concept: The degree of accumulated evidence is key
![Page 7: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/7.jpg)
7
Critical Issues: Validity
Current (1999 Standards) –
Validity Evidence based on:• Test Content• Response Processes• Internal Structure• Relation to Other Variables• Consequences of Testing
![Page 8: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/8.jpg)
8
Critical Issues: Validity
For Licensing, Validity Evidence Based on Test Content is Critical
• Job analysis provides the foundation• Identifies job-related activities/tasks (and sometimes KSAs)
that will be used to define test content
![Page 9: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/9.jpg)
9
Critical Issues: Reliability
Chapter 2 in the Standards– 20 distinct standards (some not
relevant for RE licensing exams)Reliability refers to the consistency
of measurement (reproducibility)
![Page 10: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/10.jpg)
10
Critical Issues: Reliability
All scores (all measurements) have a component of error
Scales, equivalence
![Page 11: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/11.jpg)
11
Critical Issues: Reliability
In general, four factors increase reliability:
1. Measurement of homogeneous content
2. Heterogeneous candidate groups
3. Longer tests
4. High quality items
![Page 12: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/12.jpg)
12
Critical Issues: Reliability
Estimates of reliability to be derived depend on measurement model:– Classical Test Theory (CTT)– Item Response Theory (IRT)
![Page 13: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/13.jpg)
13
Critical Issues: Reliability
Classical Test Theory– Unit of measurement: based on # correct– Attempt to estimate “true score”– Estimates of reliability focus on
consistency of scores
![Page 14: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/14.jpg)
14
Critical Issues: Reliability
Item Response Theory– Based on probabilities of correct responses – Attempt to estimate probability based on
underlying trait (or ability) parameter– Estimates of reliability focus on information
obtained
![Page 15: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/15.jpg)
15
Critical Issues: Fairness
Chapter 7 in the Standards Four principle aspects of fairness:
1. Lack of bias
2. Equitable treatment in testing process
3. Equality in outcomes
4. Opportunity to learn
![Page 16: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/16.jpg)
16
Critical Issues: Fairness
Lack of bias• Technical definition:
Bias is present if different meanings of scores are present for identifiable subgroups
• Differential item functioning (DIF)• Judgmental and statistical processes
• Content-related sources• Response-related sources
![Page 17: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/17.jpg)
17
Critical Issues: Fairness
Equitable treatment in testing process• Equal opportunity to demonstrate
knowledge• Standardized testing conditions
• Security (before, during, after)
• Equal opportunity to prepare• Candidate materials
![Page 18: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/18.jpg)
18
Critical Issues: Fairness
Equality in outcomes• Are equal passing rates for identifiable
subgroups required?• No, what is required is that all outcomes
must have the same meaning• Equality of the passing point
![Page 19: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/19.jpg)
19
Critical Issues: Fairness
Opportunity to learn• Primarily applicable to educational
achievement testing
![Page 20: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/20.jpg)
20
Critical Issues: Fairness
Highlights of Chapters 8, 9, and 10
8: Rights and Responsibilities of Test Takers – focus on candidates’ rights (e.g., to information)
9: Testing Individuals of Diverse Linguistic Backgrounds – issues of translation, adaptation, modification and potential impact on validity
10: Testing Individuals with Disabilities – ADA requires “reasonable accommodations”
![Page 21: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/21.jpg)
21
Critical Issues: Legal Defensibility
• All Standards could be considered• Standards are not legal
requirements• New Standards under development
![Page 22: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/22.jpg)
22
Critical Issues: Legal Defensibility
Other considerations• ARELLO Guidelines for Accreditation • EEOC Guidelines (1978)
![Page 23: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/23.jpg)
23
Put to the Test:
Content DevelopmentTadas Dabsys
![Page 24: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/24.jpg)
24
Exam Content Development
Job Analysis Survey Sampling Methodology Analysis of the Survey Results Content Specifications
![Page 25: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/25.jpg)
25
Job Analysis Survey
Job Analysis Procedure
– Content Development– Format Development
![Page 26: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/26.jpg)
26
Sampling Methodology
Sampling – Representative Sample Target Groups Sources for Addresses/Contact Information
![Page 27: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/27.jpg)
27
Analysis of the Survey Results
Demographic Information Professional Activities Knowledge
– Rating and Respective Response Scales– Summary Statistics– Identification of Qualifying Professional
Activities/Knowledge
![Page 28: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/28.jpg)
28
Content Specifications
Development of Content Specifications Establishing Content Base for Inclusion in
Exams Linking Knowledge Areas to Professional
Activities Development of Operational Definitions Test Outline and Content Weighting
![Page 29: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/29.jpg)
29
Put to the Test:
Writing Exam QuestionsJames F. Fryer, Ed.D., CPCU
![Page 30: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/30.jpg)
30
Developing the Exam Question(Item Writing)
How?National Subject Matter Experts
– Qualifications– Experience– Background in Job Analysis
![Page 31: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/31.jpg)
31
Developing the Exam Question(Item Writing)
Linking Back to the Job Analysis: Activities and Tasks Subject/Content Area of the Task The Knowledge Statement (KSAs)
![Page 32: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/32.jpg)
32
Developing the Exam Question(Item Writing)
Linking to the Knowledge Statement:
Content Topic Area – Leasehold EstatesContent Sub-Topic Area – Basic Elements of a LeaseJob Task – Negotiate a LeaseKSA – Knowledge of the financial ramifications of the common lease provisions referred to as Net Lease, Gross Lease, Triple-Net Lease, Percentage Lease, Base Rent, Effective Rent
![Page 33: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/33.jpg)
33
Developing the Exam Question(Item Writing)
Linking to the Knowledge Statement:
Knowledge Statement – The entry level practitioner must be able to compare and contrast the interrelationships of the commonly negotiated lease provisions referred to as Net Lease, Gross Lease, Triple-Net Lease, Percentage Lease, Base Rent, Effective Rent and equate the comparison in terms of financial impact on cost to the landlord or tenant.
![Page 34: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/34.jpg)
34
Developing the Exam QuestionBalancing Cognitive Levels:
Recall: Able to state the definition of a “latent defect.”Application: Applies knowledge of the definition of “latent defect” by recognizing and classifying certain described property conditions as meeting the definition of a latent defect.Analysis: Applies knowledge of the definition by recognizing certain described property conditions as meeting the definition and interpreting the information to determine what action should be made.
![Page 35: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/35.jpg)
35
Developing the Exam Question(Item Writing)
Basic Item Writing Principles: One Correct Answer Relevant and Appropriate Realistic Important to Related Task Straightforward Clear and Concise No Clues that Give Away the Answer Entry-Level Reading Level
![Page 36: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/36.jpg)
36
Developing the Exam Question(Item Writing)
Item Writing Principles for Multiple-Choice: Define the Task Express a Complete Thought Reduce the Reading Load Use of Negative Words Response Options need to:
– Fits the Stem Logic– Equal in Length– Unique Meaning– Plausible– Avoid “All the Above” and “None of the Above”
![Page 37: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/37.jpg)
37
Put to the Test:
Performance Analysis Nikki Shepherd Eatchel, M.A.
![Page 38: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/38.jpg)
38
Exam Question (Item) Performance Analysis
There are various types of analyses used for evaluation of exam and item performance:
Item Response Theory (1 parameter, 3 parameter)
Classical Test Theory
Classical Test Theory is the most common analysis type used for evaluation of state-based licensure exams.
![Page 39: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/39.jpg)
39
Exam Question (Item) Performance Analysis
Classical Test Theory – Item Statistics
p-value: Percentage of candidates who answer an item correctly
rpbis: Correlation between performance on an item and
performance on the overall exam
![Page 40: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/40.jpg)
40
Exam Question (Item) Performance Analysis
Classical Test Theory – Item Statistics
option analysis: Percentage of candidates who answer each option
option rpbis: Correlation between performance on the options and the overall exam
![Page 41: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/41.jpg)
41
Exam Question (Item) Performance Analysis
Classical Test Theory – Issues to Consider
p-value: Too high, too lowrpbis: Too lowoptions: Distractors with no responses
Distractors with more responses than key
O rbpis: Distractors higher than keyOmits Above expectations
![Page 42: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/42.jpg)
42
Exam Question (Item) Performance Analysis
Classical Test Theory – Sample Statistics
![Page 43: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/43.jpg)
43
Exam Question (Item) Performance Analysis
Classical Test Theory – Second sample
![Page 44: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/44.jpg)
44
Exam Question (Item) Performance Analysis
Forensic Data Analysis – Item Level
With the increasing security issues within the testing industry (e.g., brain dump sites, black market content, etc.) forensic data analysis is critical for the evaluation of a testing program.
![Page 45: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/45.jpg)
45
Exam Question (Item) Performance Analysis
Forensic Data Analysis – Item Level
What does Forensic Data tell you?
![Page 46: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/46.jpg)
46
Exam Question (Item) Performance Analysis
Forensic Data Analysis Suspicious Candidate Activity
Candidates who did better on harder questions than they did on easier questions
Candidates who achieved a high score (in the top 20%) on the exam while only viewing questions for a very short period of time (over half of the items 10 seconds or less each)
![Page 47: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/47.jpg)
47
Exam Question (Item) Performance Analysis
Forensic Data Analysis Suspicious Candidate Activity
Test takers who viewed some items for extended periods of time while cycling quickly (less than 8 seconds each) through the remaining items
Test takers with scores around chance-level while spending short amounts of time on each item (pre-viewing).
![Page 48: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/48.jpg)
48
Exam Question (Item) Performance Analysis
Forensic Data Analysis – Item Level
While there are a number of item and exam level
statistics available for forensic data analysis, many
of them revolve around classical test theory and
item level performance. These types of analyses
can be valuable in both test development
forecasting and in evaluating security issues.
![Page 49: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/49.jpg)
49
The ARELLO Accreditation Program
Overview of the function and purpose
![Page 50: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/50.jpg)
50
The ARELLO Accreditation Program
Independent Evaluation Confidence and Protection Defensible Standards Enhanced Quality Facilitates License Recognition
![Page 51: 1 Examination Accreditation Program Put to the Test: Exam Development Made Easy April 26, 2008 Pasadena – California -USA](https://reader035.vdocuments.us/reader035/viewer/2022070323/56649e175503460f94b02089/html5/thumbnails/51.jpg)
51
QUESTIONS?COMMENTS?
Fabulous Prize Giveaway