cas examination process julie stenberg, fcas cane meeting march 20, 2007
Post on 27-Dec-2015
217 Views
Preview:
TRANSCRIPT
CAS EXAMINATION PROCESS
Julie Stenberg, FCAS
CANE MeetingMarch 20, 2007
2
CAS Admissions Process Audit
• Fall 2000 – CAS Issues RFP for External Review of Admissions Processes
• The Chauncey Group (Subsidiary of ETS) Selected
• Spring 2001 – Chauncey Group Conducts Audit of CAS Admissions Processes
3
Audit Findings
The CAS Does Many Things Well:
• Good Communication with Candidates
• Sound Procedures for Maintaining Confidential Information
• Exams are Administered with Appropriate Controls and Standardized Procedures
4
Audit Findings
Several Areas for Improvement:• Need Better Link Between Learning
Objectives and Exams/Readings• Learning Objectives and Exam
Blueprints Should be Published• Need Better Training of Item Writers• Need to Consider Alternative Processes
for Selecting Pass Marks
5
Major Objectives
The Chauncey Group Engaged to Help CAS with Three Issues:
• Write Better Learning Objectives and Establish Links to Readings/Exams
• Develop a Process for Training Item Writers
• Pilot an Alternative Process for Selecting Pass Marks
6
Major Milestones• August 2001 – Chauncey Began Facilitating
Meetings to Write Learning Objectives• October 2001 – Piloted Pass Mark Panel
Process for Exams 6 & 9• March 2002 – Piloted Item Writer Training
Classes for Exams 6 & 9• April 2002 – Pass Mark Panels Meet for Exams
5, 7 & 8• June 2002 – Item Writer Training for Exams 5,
7 & 8
7
Major Milestones
• October 2002 – Pass Mark Panels Meet for Exams 6 & 9
• February 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes
• April 2003 – Executive Council Approves New Learning Objectives for Exams 3, 5-9
8
Recent Improvements
• Evolution of CBT
• Improvement to Pass Mark Panel Process
• Expanded Sample Answer Sets
• Increased Communication
• CAS Board White Paper
9
Learning Objectives
The way things were
– What topics should successful candidates understand
– What readings should they know?
The way things are now
– What should successful candidates be able to DO?
10
Learning Objectives
The way things were
– Individual topics and readings were the basis for assigning the writing of exam questions
The way things are now
– Learning Objectives are the basis for assigning the writing of exam questions
11
Learning Objectives
The way things were
– Syllabus “blueprints” were the documents governing the review of the Syllabus and the construction of Exams
The way things are now
– Learning Objective Documents are the basis for the review of the Syllabus and the construction of Exams
12
Learning Objectives
The Syllabus Committee has developed Learning Objective Documents for CAS Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9
13
Learning Objective Documents
Five Elements
• Overview Statement for a Group of Learning Objectives
• Learning Objectives
• Knowledge Statements
• Syllabus Readings
• Weights
14
Learning Objective Documents
Overview Statements
• Certain Syllabus Sections Can Have Multiple Learning Objectives (e.g., Ratemaking)
15
Learning Objective Documents
Learning Objectives• What successful candidates should be able to
do• Learning Objectives Should:
Clearly state a main intentReflect a measurable outcomeSupport an attainable behaviorRelate to the learner’s needs or job
functionHave a definitive time frame
16
Learning Objective Documents
Knowledge Statements
• Support Learning Objectives
• In order to accomplish the objective, what does the candidate need to know?
17
Learning Objective Documents
Readings
• An individual reading may be listed under more than one learning objective
• Readings listed under multiple objectives may facilitate more synthesis/reasoning/cross-topic Exam questions
18
Learning Objective Documents
Weights (by Learning Objective)
• Shown as ranges
• The ranges are guidelines and are not intended to be absolute
• Ended practice of candidates calculating de facto weights by reading or topic from past Exams
19
IV. Ratemaking This section contains objectives covering ratemaking in broad, general principles, as well as specific detail. Candidates should have a thorough understanding of the basic principles of ratemaking, so that they can analyze data, select an appropriate technique, and develop a solution to a numerical problem. In addition, the candidate should be able to compare specific ratemaking techniques in terms of advantages and disadvantages as they are applied to specific situations and for different lines of business. Weight: xx-xx% 1. Explain the role of exposure bases in the
ratemaking process. Weight: xx-xx%
Definition of exposure base Characteristics of exposure base Impact of exposure change Coverage provisions
Reading(s): McClenahan, Finger, Principles, Bouska, Feldblum (WC), Graves & Castillo, Jones 2. Use appropriate premium data to
estimate premium input into the overall rate level indication, adjusting for the following:
Coverage and benefit level changes
Rate level changes Premium trend
Weight: xx-xx%
Compilations of experience (CY/PY/AY) Written versus earned premium Rate changes Policy terms Distributional shifts/changes in volumes (tend over time) Parallelogram method Extension of exposures Definition of exposures Impact of law changes
Reading(s): McClenahan, Finger, Principles, Feldblum (Asset Share), Feldblum (WC), Jones, ASOP #13
20
Learning Objectives and the Syllabus
• Learning Objective Documents Provide High Level Guidance
– Review of Current Syllabus Material
– Identification of Topics Requiring New Syllabus Material
• Weights help Syllabus Committee Target Specific Objectives
21
Future Changes to Learning Objective Documents
• These are Living Documents– Never Perfect– Subject to Change
• Updates – When and How Often?– Once a Year Per Exam Seems
Reasonable– At Least Disruptive Time for Candidates
22
Future Changes to Learning Objective Documents
• CAS Executive Council (VP-Admissions) Performs Oversight and Final Approval of Any Changes
– Just as it does with changes to the Syllabus
– Just as it has with the current Learning Objective Documents
23
Learning Objective Summary• Transition to Published Learning
Objectives Should Help the CAS Achieve:– Better Syllabus Content and Exam
Questions– More Transparent Basic Education
Process– Better Model for Evaluating Future
Changes to the Syllabus– Better Model for Evaluating Future
Changes to the Desired Education of Casualty Actuaries
24
Writing Exam QuestionsQuestion 1 – According to Miller, “Writing Exam Questions”, which of the following is true?
I. Writing exam questions is the same now as it was 6 years ago
II. Writing exam questions is easier now than it was 6 years ago
III. Writing exam questions is harder now than it was 6 years ago
A. II onlyB. I and II onlyC. I and III onlyD. II and III onlyE. I, II and III
25
Writing Exam QuestionsI. Writing exam questions is the same now as
it was 6 years ago
True – Question writers have always wanted to write good, fair, high quality questions.
1. Still takes the same time commitment
2. Still requires studying assigned readings
3. Still involves choosing the areas you want to test
4. Still involves peer review by others
26
Writing Exam QuestionsII. Writing exam questions is easier now than
it was 6 years ago
True – We have more tools to work with
1. We have identified objectives
2. They identify the readings tied to those objectives
3. Question writing skills are taught – what to do and what to avoid
4. We have a common language with which to make constructive criticisms
27
Writing Exam Questions III. Writing exam questions is harder now
than it was 6 years ago
True – We have been conditioned by years of studying old questions
1. Triple True-False are often easier to write than short answers for Multiple Choice
2. The easiest questions to write may not always fit the objectives
3. The bar is higher and we don’t like to fail
28
Writing Exam Questions
Question 1 – Solution: E
A. Some candidates will think nothing has changed
B. Some candidates will think the new process could not possibly make it harder to write questions
C. Some candidates always guess C when they don’t know the answer
D. I used to guess D
E. Correct answer
29
What Have We Learned From The Chauncey Initiatives?
• Questions should be focused on learning objectives, rather than individual papers
• Triple True/False is not the only kind of multiple choice question
• Art of selecting good “wrong” multiple choice answers
30
What Changes Should The Candidates See On The Exams?
• Better questions
• Questions with many possible full-credit answers
• Less “according to” and “based on” questions
• Heavy “list” papers have become open-book
31
Setting the Pass Mark
• Identify Purpose of the Pass Mark
• Convene Pass Mark Panel
• Analyze Exam Statistics
• Prepare Recommendation
• Proceed through Approval Process
32
Purpose of the Pass Mark
• Pass Minimally Qualified (or better) Candidates– Those who have demonstrated a
sufficient grasp of the syllabus material
• Fail Others
• There is no predetermined pass ratio
33
Purpose of the Pass Mark
Failers Passers
Minimally Qualified Candidate
34
Pass Mark Panel
• Panel includes:New Fellows (1-3 years)Fellows experienced in practice areaOfficers of exam committee
• Recommends a pass mark independent of the normal exam committee procedures
35
Pass Mark Panel
• Defines Minimally Qualified CandidateWhat he or she should will knowWhat he or she will not knowWhat he or she will be able to
demonstrate on the exam
• Relates Criteria to Learning Objectives for defining the minimally qualified candidate.
36
Pass Mark Panel
• Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question).
• Scores are assembled and shared in a group format.
• Group discusses ratings and may change estimates
• Facilitator compiles ratings and shares results with exam committee officers
37
Analyze Exam Statistics (back at the Grading Session)
• Collect Initial Scores for All Candidates• Review/Discuss Key Measures
High, Low, MeanPercentiles, Percentile RelationshipsPass Mark Panel RecommendationPrior statistics from previous examsCAS Board goal, “…that 40% or more of the candidates
should get a score of 70% or more on any given exam; and all candidates that get such a score should pass.”
• Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)
38
Prepare Recommendation
• Recollect scores if any have changed and review all relevant statistics again.
• Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark.
• Justify Recommended Pass Score
39
Approval Process
• Part Chair
• General Officer (Spring / Fall)
• Exam Committee Chair (Arlie Proctor)
• VP-Admissions (Jim Christie) – The final decision on the pass mark is the responsibility of this position.
40
Appeal Process
• In the event of a candidate appeal, a grader may
be called upon to review the appeal and
reconcile the score with the grading key.
41
Join the Exam Committee
Fill out the annual CAS participation survey
or
Contact the exam committee recruiter directly
Rhonda Walker
rpwalkerbhnj@comcast.net
42
Questions and Comments
Contact
Julie Stenberg at
julia.stenberg@travelers.com
Or
Arlie Proctor at
aproctor@munichreamerica.com
top related