casualty actuarial and statistical (c) task force · the motion passed unanimously. 2. adopted the...

© 2019 National Association of Insurance Commissioners 1

CASUALTY ACTUARIAL AND STATISTICAL (C) TASK FORCE Casualty Actuarial and Statistical (C) Task Force Apr. 6, 2019, Meeting Minutes Casualty Actuarial and Statistical (C) Task Force Mar. 22, 2019, Conference Call Minutes (Attachment One) Casualty Actuarial and Statistical (C) Task Force Mar. 12, 2019, Conference Call Minutes (Attachment Two) Casualty Actuarial and Statistical (C) Task Force Feb. 12, 2019, Conference Call Minutes (Attachment Three) Comment Letter on the Statement of Actuarial Opinion Instructions (Attachment Three-A) Comments on Best Practices for Regulatory Review of Predictive Analytics White Paper (Attachment Three-B) Casualty Actuarial and Statistical (C) Task Force Jan. 29, 2019, Conference Call Minutes (Attachment Four) Casualty Actuarial and Statistical (C) Task Force Jan. 8, 2019, Conference Call Minutes (Attachment Five) Casualty Actuarial and Statistical (C) Task Force Dec. 18, 2018, Conference Call Minutes (Attachment Six) W:\National Meetings\2019\Spring\TF\CasAct\contents.docx

Draft Pending Adoption


Draft: 4/18/19

Casualty Actuarial and Statistical (C) Task Force Orlando, Florida

April 6, 2019 The Casualty Actuarial and Statistical (C) Task Force met in Orlando, FL, April 6, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phil Vigliaturo (MN); James J. Donelon, Vice Chair, represented by Rich Piazza (LA); Lori K. Wing-Heier represented by Michael Ricker (AK); Keith Schraad represented by Erin Klug (AZ); Ricardo Lara represented by Lynne Wehmueller (CA); Andrew N. Mais represented by Wanchin Chou (CT); Stephen C. Taylor represented by David Christhilf (DC); David Altmaier represented by Sandra Starnes (FL); Colin M. Hayashida represented by Gerald Hew (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Robert H. Muriel represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Cynthia Amann (MO); Mike Causey represented by Kevin Conley (NC); Marlene Caride represented by Mark McGill (NJ); Barbara D. Richardson represented by Stephanie McGee (NV); Jillian Froment represented by Thomas Botsko (OH); Glen Mulready represented by Andy Schallhorn (OK); Andrew Stolfi represented by TK Keen (OR); Jessica Altman represented by Michael McKenney (PA); Raymond G. Farmer represented by Michael Wise (SC); Kent Sullivan represented by J’ne Byckovski (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Adopted its March 22, 2019; March 12, 2019; Feb. 12, 2019; Jan. 29, 2019; Jan. 8, 2019; Dec. 18, 2018; and 2018 Fall

National Meeting Minutes Mr. Vigliaturo said the Task Force met March 22, 2019; March 12, 2019; Feb. 12, 2019; Jan. 29, 2019; Jan. 8, 2019; and Dec. 18, 2018. During these meetings, the Task Force took the following action: 1) adopted statistical reports; and 2) adopted a comment letter to the Executive (EX) Committee’s ad hoc group regarding the Statement of Actuarial Opinion instructions related to the definition of Qualified Actuary. The Task Force also met March 19, 2019, in regulator-to-regulator session, pursuant to paragraph 3 (specific companies, entities or individuals) of the NAIC Policy Statement on Open Meetings, to discuss rate filing issues. The Task Force viewed a Casualty Actuarial Society (CAS) livestreaming event March 26, 2019, in lieu of its typical Predictive Analytics Book Club conference call. The CAS provided access to its livestreaming event from its Ratemaking, Product and Modeling (RPM) Seminar, which included the following sessions: Machine Learning and Artificial Intelligence; The Predictive Modeling Cooking Show; The Changing Role of the Actuary in the Face of Disruption; Insurance Claims Prevention Using New Predictive Analytics; and Auto Insurance: What Happened and What Happens Next? Mr. Dyke made a motion, seconded by Mr. Piazza, to adopt the Task Force’s March 12, 2019 (Attachment One); Feb. 12, 2019 (Attachment Two); Jan. 29, 2019 (Attachment Three); Jan. 8, 2019 (Attachment Four); Dec. 18, 2018 (Attachment Five); and Nov. 15, 2018 (see NAIC Proceedings – Fall 2018, Casualty Actuarial and Statistical (C) Task Force) minutes. The motion passed unanimously. 2. Adopted the Report of the Actuarial Opinion (C) Working Group Mr. Vigliaturo said Julie Lederer (MO) provided a written report for the Actuarial Opinion (C) Working Group. The Working Group has not met in 2019. In mid-March, the Financial Examiners Handbook (E) Technical Group asked the Working Group to review the property/casualty (P/C) reserves and claims handling exam repository and provide feedback by May 31. Mr. Vigliaturo appointed Anna Krylova (NM) as vice chair of the Working Group. Ms. Mottar made a motion, seconded by Mr. Botsko, to adopt the report of the Actuarial Opinion (C) Working Group. The motion passed unanimously.



3. Adopted the Report of the Statistical Data (C) Working Group Mr. McGill said the Statistical Data (C) Working Group is working on the formulas in the Report on Profitability by Line by State, primarily regarding allocation of investment gain. Mr. Piazza made a motion, seconded by Mr. Botsko, to adopt the report of the Statistical Data (C) Working Group. The motion passed unanimously. 4. Discussed the Appointed Actuary Attestation of Qualification and the Three-Year Experience Requirement Proposals Mr. Vigliaturo said there was discussion during the March 22 hearing with the Executive (EX) Committee’s ad hoc group about the Statement of Actuarial Opinion instructions, written comments received and distributed edits. He said the changes the Task Force needs to consider for adoption relate to the attestation and experience charges. He said the other changes will be decided by the ad hoc group. The Task Force discussed the need to review its proposed changes within a document showing all proposed changes, whether the qualification documentation should include specific reference to companies’ insurance lines and scope of business, additional information to be included in instructions and/or in the annual regulatory guidance, and a concern about potentially rushing the project with lower quality instructions. The Task Force decided to meet via conference call in regulator-to-regulator session to consult with NAIC staff in order to gain a better understanding of how the Task Force’s proposed instructions fit with the ad hoc group’s proposed instructions. 5. Discussed the Predictive Analytics White Paper Mr. Piazza said the white paper was exposed for a 60-day public comment period ending Feb. 12. That exposure period was then extended another week until Feb. 19. The Task Force held two discussions during its Feb. 12 and March 12 conference calls. Following those calls, volunteer drafters mapped the comments to paragraphs of the paper to aid evaluation. Mr. Piazza asked for any comments on the 16 best practices. He said he would like to consolidate and reword the best practices. He suggested that the best practices and knowledge statements should be focused on the regulatory practice and not at the industry. Birny Birnbaum (Center for Economic Justice—CEJ) said the charge to the Task Force is limited to rate filings, but state insurance regulators should consider the review of pricing more generally. After marketing and underwriting, the data is already refined. If the company uses factors of concern to state insurance regulators—such as credit scoring, employment, occupation, etc.—then state insurance regulators should be concerned that the rate filing does not fully capture the pricing factors being used through underwriting tiers, etc. He said one of the fundamental areas of regulatory review is the data. State insurance regulators should ask about the data used to develop the model and judge whether the data is incomplete, biased or faulty in some way. He said that would be a best practice in review of the ratemaking model. Mr. Piazza said state insurance regulators will be looking at the data and evaluating whether there are biases in the data. He said they will not be evaluating how the companies may have targeted their products. Mr. Birnbaum clarified he does not suggest the review of models for marketing or underwriting in the scope of the charge; he said the scope to review ratemaking predictive models should include understanding the data used to develop, test and produce ratemaking results. Mr. Piazza agreed that bias is an issue identified in the knowledges. Ralph Blanchard (Travelers) said the data described by Mr. Birnbaum is not biased. He said data has a characteristic. He said life insurance companies do not use mortality trends of the U.S. population because that data does not match to its underwriting. He said the data for ratemaking must have the same characteristic underwriters are using to underwrite. Mr. Birnbaum said if the data describes the book of business and all attributes of those customers, then that would be fine. But if pricing factors have been applied in the underwriting process, then it is important to understand the situation, and companies should be disclosing that information. Mr. Keen asked if Mr. Birnbaum’s suggestion includes advising state insurance regulators to look at variables in the model, but also evaluating variables excluded from the model and the reason why. Mr. Birnbaum said to look for unfair discrimination state insurance regulators need to know if the ratemaking data has been refined, especially if underwriting or rating tiers were used. He said the underwriting tiers might include risk variables that cause regulatory concern.



David Kodama (American Property Casualty Insurance Association—APCIA) said the APCIA’s member companies suggest that the regulatory review of models be consistent in approach with other requirements in the rate filing process. He said state insurance regulators should not impose different standards and requirements. He said state insurance regulators should give proper deference to existing Actuarial Standards of Practice (ASOPs). He said the APCIA supports consistency but not the imposition of new standards onto the process. Mr. Piazza said the charge is to provide guidance to the states. He said the aim is to use standards that the states already have. He said many states need help to better understand and improve reviews. He said one potential result of the work might be for companies to provide information upfront in the filing rather than the states having to request information, which could eliminate numerous additional weeks in the review process. Mr. McGill said state insurance regulators would like companies to provide support of models similarly to how the companies meet standards to support traditional rate filings. Mr. Birnbaum added that state insurance regulators are not creating new standards. He said state insurance regulators are responding to companies’ new data usage and techniques by developing new regulatory tools and techniques in response. Mr. Piazza said the next steps will be for the drafting group to review the comments and provide suggested changes. Using the suggested changes, the white paper will be redrafted and exposed for comment. He said the process will likely result in the identification of some policy or other issues that are not actuarial. He said those issues may be brought to the attention of other NAIC groups. He said he expects the drafting and review processes might need to be repeated a couple of times, with an aim to submit the white paper to its parent committee and the Big Data (EX) Working Group by the Fall National Meeting. He said he would also like to be able to propose revisions to the Product Filing Review Handbook and state guidance at the same time. 6. Discussed NAIC Activities Regarding Casualty Actuarial Issues Ms. DeFrain said predictive analytics sessions will be held June 6–7 at the NAIC/NIPR Insurance Summit in Kansas City, MO. She said the training will include case studies. Mr. Grassel said Iowa’s Global Insurance Symposium (GIS) is being held April 23–25, with predictive analytics training for state insurance regulators being conducted in the days before and after the GIS. Mr. Chou said the NAIC is providing training to enhance the Own Risk and Solvency Assessment (ORSA) reviews on Section 3 of the ORSA, which includes economic capital models. 7. Heard Reports from Actuarial Organizations Kathleen C. Odomirok (American Academy of Actuaries—Academy) said the Academy’s Committee on Property and Liability Financial Reporting (COPLFR) educates practicing actuaries on regulatory requirements around the Statement of Actuarial Opinion. She said around 80 actuaries attended the annual seminar for opinion writers. Around 150 registrants participated in a follow-up webinar on selected topics, which was held Feb. 1 and focused on effective report writing. She said the COPLFR will be contacting state insurance regulators to help with updating the annual Loss Reserve Law Manual, updating the P/C practice note on Statements of Actuarial Opinion, and updating the practice note on risk transfer with respect to reinsurance contracts. Lisa Slotznick (Academy) provided a list of activities from other Academy’s Casualty Practice Council groups. She said the Property and Casualty Risk-Based Capital Committee works with the NAIC. The P/C Extreme Events and Property Lines Committee will soon be publishing a paper on wildfires, is working on a catastrophe bond paper and is working on flood insurance issues. Other groups are providing predictive analytics training at the NAIC/NIPR Insurance Summit in Kansas City, MO, and are working on cybersecurity, cyber insurance and the federal Terrorism Risk Insurance Act (TRIA) program. Ms. Slotznick said there is a lot of research being conducted regarding cyber, wildfires and a climate risk index. Godfrey Perrott (Actuarial Board for Counseling and Discipline—ABCD) discussed the ABCD’s investigations of complaints and requests for counseling. Kathleen A. Riley (Actuarial Standards Board—ASB) discussed exposure and adoption actions taken on various ASOPs. Information can be found on the ASB website.



Providing information on P/C actuarial research, R. Dale Hall (Society of Actuaries—SOA) presented the SOA’s general insurance actuarial research and education, and Mr. Blanchard presented the CAS’ P/C actuarial research. Information about this research can be found on the SOA and CAS websites. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\4-6 CASTF Min.docx

Attachment One Casualty Actuarial and Statistical (C) Task Force

4/6/19


Draft: 4/1/19

Casualty Actuarial and Statistical (C) Task Force Conference Call March 22, 2019

The Casualty Actuarial and Statistical (C) Task Force met via conference call March 22, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo (MN); James J. Donelon, Vice Chair, and Rich Piazza (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling and Daniel J. Davis (AL); Michael Conway represented by Mitchell Bronson and Sydney Sloan (CO); Andrew N. Mais represented by Wanchin Chou (CT); Stephen C. Taylor represented by David Christhilf (DC); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Robert H. Muriel represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa and Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Julie Lederer (MO); Marlene Caride represented by Mark McGill (NJ); John G. Franchini represented by Mark Hendrick and Anna Krylova (NM); Mike Causey represented by Arthur Schwartz (NC); Barbara D. Richardson and Gennady Stolyarov (NV); Andrew Stolfi represented by David Dahl (OR); Jessica Altman represented by Kimberly Rankin (PA); Raymond G. Farmer represented by Will Davis (SC); Kent Sullivan represented by Nicole Elliott, Miriam Fisk and Walt Richards (TX); Mike Kreidler represented by Eric Slavich (WA); and James A. Dodrill represented by Joylynn Fix (WV). 1. Heard a Report on the Appointed Actuary Project Commissioner Ridling asked NAIC staff to provide some background on the Appointed Actuary Project. Kris DeFrain (NAIC) said, in 2017, the Executive (EX) Committee decided to revise the property/casualty (P/C) Statement of Actuarial Opinion (SAO) instructions to create an objective definition of a “qualified actuary” and make other changes in line with guidance presented by an NAIC consultant. To do this work, the Committee did three things: 1) appointed an ad hoc group of commissioners to oversee the Appointed Actuary Project (the ad hoc group now includes Commissioner Ridling, Superintendent Cioppa and Commissioner Donelon); 2) asked NAIC staff to work with actuarial organizations and volunteers to complete the project; and 3) assigned three charges to the Task Force.

Working with a consultant and numerous volunteers, NAIC staff completed the first phase of the Appointed Actuary Project: the Job Analysis Project. The output was a list of knowledge statements defining what a P/C Appointed Actuary needs to know and do to provide an expert actuarial opinion to accompany the annual statutory financial statement.

NAIC staff then began the second phase of the Appointed Actuary Project: the Educational Standards and Assessment Project. This second phase used the knowledge statements from the Job Analysis Project to develop minimum actuarial educational standards. These standards determine the syllabi and educational content of an actuarial education program to provide minimum basic education for a P/C Appointed Actuary. The project also documented numerous areas where the P/C actuary is expected to need additional experience and continuing education (CE) to do the Appointed Actuary job.

With those minimum standards in place, the NAIC is now assessing the P/C actuarial education offered by the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA). With a few months of the assessment remaining, the result due in mid-May will determine which actuarial credentials, under specific terms, will be accepted as the new NAIC accepted actuarial designations. These designations will form the foundation of the new “qualified actuary” definition in the SAO instructions. At the same time NAIC staff worked on these projects, the Task Force worked on its three charges related to the attestation, experience and continued competence. The result of the first two charges are included in the SAO instructions being discussed today. The work on competence will continue into 2019, as it does not require an immediate change to the SAO instructions.

Ms. DeFrain said the purpose of the hearing today is to invite comments on the revised SAO instructions. These revised instructions combine the proposals of the Committee’s ad hoc group and the Task Force. These instructions were released for a 60-day public comment period beginning Dec. 15, 2018, and ending Feb. 15, 2019.

The Committee’s ad hoc group discussed the written comments and agreed to make numerous changes in the SAO instructions to address a majority of the written comments. Those changes were made to the SAO instructions last week. Ms. DeFrain discussed that document briefly, as it is believed that it resolves many comments and clears the way for the agenda today.


4/6/19


Ms. DeFrain said the document provided in the materials for the conference call includes the insertion of comments to the side in order to explain where changes were made and, sometimes, where changes were considered and alternative action is to be taken. The change with the biggest impact is that one of the components on the definition of “qualified actuary” was removed. That former subpart read “has sufficient experience and knowledge obtained through basic education, continuing education, or experience to understand reserving for the company’s lines of business and business activities.” It was agreed that this would be removed from the definition, given that it would remain in the documentation for the attestation.

The top four remaining issues were identified and placed on the agenda. Ms. DeFrain said additional changes to the instructions might be made after the hearing. Ms. DeFrain said the agenda is also split between the ad hoc group’s proposals and the Task Force’s proposals. She referred to the initially proposed and exposed instructions for the identification of which group proposed which changes. Commissioner Ridling agreed that the purpose of the hearing is to hear oral comments to decide whether to make additional changes to the SAO instructions. 2. Heard Comments on the Proposal to Re-expose the SAO Instructions After Completion of the Educational Standards and

Assessment Project Ms. Mottar, Ms. Lederer and Mr. Vigliaturo said they and the Task Force support the re-exposure of the SAO instructions after the NAIC completes the CAS and SOA’s actuarial examination assessments and determines any restrictions on the actuarial designations. 3. Heard Comments on the Proposal to Require Academy Membership in the Qualified Actuary Definition Mary Miller (American Academy of Actuaries—Academy) said it is simpler to require “a member of the Academy” than the current professionalism wording used. She said that while the CAS and SOA are international bodies, the Academy is the U.S. actuarial association and home of professionalism in the U.S. She said almost every other country would require membership in their countries’ associations to do statutory work in that country. Mr. Dyke said the Task Force outlined arguments in favor and against including Academy membership as a requirement. He said he supports Academy membership as a requirement in the definition. He said the Academy does a lot of work for the NAIC and to support the overall profession, particularly in respect to the qualification of actuaries. He said the Academy is the U.S. national association and it would be simpler to refer to membership in the Academy than the way it is written now. He referred to the Task Force’s letter. Ralph Blanchard (Travelers) expressed concern regarding the Casualty Practice Council reviewing qualifications of non-members. He said he is not comfortable with the Academy playing a quasi-regulator role. Ms. Miller said the Academy would not review qualifications of anyone who is not a member of the Academy. Mr. Stolyarov said the Task Force agreed on a proposal to add a qualifier to add Academy membership in order for the Casualty Practice Council to review. Mr. Schwartz said an actuary does not have to be a member of the Academy to be held to the Actuarial Standards of Practice (ASOPs) and the Actuarial Board of Counseling and Discipline (ABCD). Mr. Stolyarov said the Task Force identified reasons why Academy membership should not be a requirement. He said the professionalism framework applies to actuaries regardless of Academy membership. Academy membership is not a current requirement in the SAO instructions. Academy membership has many benefits, but making it a requirement has a cost of $675 in membership dues. Mr. Stolyarov said adding Academy membership is beyond the scope of the current project to create an objective definition. He said the vast majority of appointed actuaries are U.S. residents, so state insurance regulators should not require Academy membership for all actuaries when there might be an issue with a non-U.S. resident. He referred to additional points in the comment letter.


4/6/19


Commissioner Ridling said the definition of “qualified actuary” must be harmonized before the states are harmonized. 4. Heard Comments on the Proposal to Modify the Grandfathering Clause Mr. Stolyarov said the Task Force agreed to propose revised wording for the grandfathering clause to note that the restrictions that would apply to designations would also apply in grandfathering. He said the grandfathering clause would be clearer without changing the spirit of what is intended. Ann Weber (SOA) said a note about the restrictions seems fair and makes sense. She said any changes to the way grandfathering is written needs careful consideration for fairness. She said alternative language proposed during Task Force discussions would be unfair because CAS actuaries that came through before the end date would be deemed qualified, whereas SOA actuaries would not. She said the current process is intended to be fair to all actuaries. Mr. Blanchard said his concern with grandfathering seems premature to comment before the assessment is complete. Commissioner Ridling said that is in line with those who have requested a re-exposure. Mr. Stolyarov said it is important not to deprive an actuary solely because of the date the designation is inferred. He said the syllabi are updated continuously and that will continue. Ms. Lederer expressed concern regarding whether the assessment will find the designation deficient in certain areas. She said she does not believe it appropriate to grandfather the designation when it is shown to not meet minimum standards. Ms. Miller said the qualification standards do provide a method for people to become qualified if there are gaps in their education. Mr. Stolyarov said that if there are some deficiencies in the syllabus, there is no way to say an individual did not obtain the education through another method. He said a preemptive bar, because of the timing of completion, seems punitive. 5. Heard Comments on the Commissioners’ Power to Accept Non-Qualified Actuaries as Appointed Actuaries

Ms. Miller said the term “Qualified Actuary” in the SAO instructions has been changed to “Appointed Actuary” because of the definition change in order to state that a commissioner can approve someone to be an Appointed Actuary even if that person is not a Qualified Actuary. She said the instructions cannot give or take away a commissioner’s authority, but past drafters did not want to emphasize this option because it seems to weaken the work to define “Qualified Actuary.” Mr. Stolyarov said the issue did not receive much discussion, but he favors keeping the wording as-is. He said this is a rare situation. He said it is helpful to have this explained for commissioners to understand and to not have it hidden in other parts of the instructions. Ms. Miller said commissioners have the power without it being encouraged in the instructions. Commissioner Ridling questioned whether it would be better to not address that particular issue. Mr. Schwartz suggested changing the description from “insurance regulatory official” to “commissioner.” He also expressed concern that there would be a verbal approval. Mr. Piazza pointed out that the restrictions require written approval. He said that if the sentence is retained in the instructions, then the term “insurance regulatory official” should be modified to explain who that is. Commissioner Ridling agreed.

6. Heard Comments on the Attestation and Experience Proposals Mr. Schwartz expressed concern that the documentation of qualifications related to the lines of business and company structure is too restrictive. He said actuaries must do the “look in the mirror” test. Ms. Miller said there is little help to a board of directors determine whether the actuary is qualified. She said an earlier version of the SAO instructions referenced the Academy’s attestation document. Otherwise, there is quite a burden placed on the board of directors and state insurance regulators. She agreed that discussing the lines of business and company structure will be


4/6/19


confusing to boards. The qualification standards already require the actuary to have the experience and qualifications to do the work. She added that the paragraph should use the word “attestation.” Mr. Stolyarov said he has no opinion on what the qualification documentation should be called. Regarding the documentation requirement, he said it does not need to be annual because once experience is attained, it cannot be unattained. He said he reviews qualifications of actuaries and has clear instructions. State insurance regulators need to be able to review the documentation as they see fit. Ms. Miller said she believes the documentation should be in the board of directors’ documentation and not in the actuarial report. The ad hoc group will meet to determine next steps before the CAS and SOA assessments are completed. The Task Force will discuss its proposed instructions at the Spring National Meeting. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\3-22 CASTF min.docx

Attachment Two Casualty Actuarial and Statistical (C) Task Force

4/6/19


Draft: 3/29/19

Casualty Actuarial and Statistical (C) Task Force Conference Call March 12, 2019

The Casualty Actuarial and Statistical (C) Task Force met via conference call March 12, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo (MN); James J. Donelon, Vice Chair, represented by Rich Piazza (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Ricardo Lara represented by Mitra Sanandajifar and Lynne Wehmueller (CA); Michael Conway represented by Sydney Sloan (CO); Andrew N. Mais represented by Susan Andrews, Qing He and Wanchin Chou (CT); David Altmaier represented by Howard Eagelfeld and Sandra Starnes (FL); Doug Ommen represented by Travis Grassel (IA); Robert H. Muriel represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Gina Clark, Brent Kabler and Julie Lederer (MO); Marlene Caride represented by Mark McGill (NJ); John G. Franchini represented by Anna Krylova (NM); Barbara D. Richardson represented by Gennady Stolyarov (NV); Jillian Froment represented by Tom Botsko (OH); Glen Mulready represented by Nicolas Lopez (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Raymond G. Farmer represented by Will Davis (SC); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka and Elizabeth Howland (TX); Mike Kreidler represented by Eric Slavich (WA); and James A. Dodrill represented by Joylynn Fix (WV). Also participating was: Gordon Hay (NE). 1. Discussed Comments Received on the Best Practices for Regulatory Review of Predictive Analytics White Paper Mr. Vigliaturo said comments were received on the draft white paper and attached to the Task Force’s Feb. 12 minutes (see NAIC Proceedings – Spring 2019, Casualty Actuarial and Statistical (C) Task Force, Attachment Three). He said anyone who submitted written comments would be allowed to present those during this conference call. Adam Pichon (LexisNexis) said Lexis Nexis supports some standardization, such as checklists, to provide filing information for state insurance regulators to review models. One area of concern is that the white paper is focused on generalized-linear models (GLMs), yet there are other models that are starting to be accepted for use. He said there should not be so much rigor that the “art” side of modeling, such as the evaluation that attributes have plausible causality or rationally make sense, is hindered. He said another example occurs when running a GLM with small amounts of data and evaluating p values. He said some variables are known to be predictive even if the p value is greater than 5%, which is a generally accepted threshold for p values. He said confidentiality to protect models and techniques used to develop the model is important. Mr. Vigliaturo said the Task Force’s intent is to limit the scope by focusing on only GLMs for private passenger auto and homeowners. He said this was decided in order to make progress on the work. Mr. Hay said his comments concern the definition of “unfairly discriminatory,” and the states needing their own definition. Richard Gibson (American Academy of Actuaries––Academy) said the Academy would like to see more explicit recognition of Actuarial Standards of Practice (ASOPs). Michael Woods (Allstate) said Allstate supports a more consistent framework. He said Allstate believes in transparency and attempts to pre-empt as many comments as possible. He suggested removing the request to provide intuitive arguments for why a rating variable is related to an outcome because it conflicts with ASOP No. 12 (Rating Classification). Alternatively, state insurance regulators could request that the Actuarial Standards Board (ASB) change ASOP No. 12. He said state insurance regulators should not try to reproduce the model, but the white paper should focus on the review of procedures and outcomes, rather than on rebuilding the model. He said the raw data is considered proprietary, noting that there are security risks to sending raw data. The white paper places a large emphasis on state-specific results, yet most rating plan models are built using multistate datasets for credibility purposes. The white paper seems to emphasize univariate indications, but multivariate indications are more accurate. He said some sections ask for earned premium, but that would require bringing all premiums to a common basis. He suggested that the Task Force focus on the goal of the white paper to help states with limited resources. He said the current framework creates a large scope and would make the review more difficult for the states with limited resources.


4/6/19


David Kodama (American Property Casualty Insurance Association—APCIA) said the APCIA supports the effort. He said the APCIA wants a product to clearly outline and describe priority areas for state insurance regulators. The APCIA believes that best practices could foster comprehensive upfront dialogue between the filing company and state insurance regulators that support an efficient and effective review appropriately focused on ensuring compliance with the applicable statutory or regulatory rating standards. He said the regulatory reviews need to allow insurers to innovate and make improvements in risk-assessment practices. Mr. Kodama said there are five key priority areas for the AICPA: 1) use practices developed and employed by state insurance regulators and leverage ASOPs; 2) do not extend the statutory scope of the filing review process beyond rates being “not excessive, inadequate nor unfairly discriminatory” (i.e., do not create a requirement for rating variables to adhere to some subjective intuitive relationship standard); 3) do not create a “one-size-fits-all” restrictive checklist; 4) identify essential information to be included in the filing to reflect the priority areas to the state insurance regulator in determining how the model impacts the methodology and output (i.e., there will be cases where more information is required, but the APCIA urges that the current number of elements or information required should be significantly reduced); and 5) protect insurers’ financial and intellectual property. Ms. Wehmueller said California provided technical edits and supports development of the white paper. She said it might be worthwhile to emphasize the purpose of the white paper to be a guide and that state insurance regulators will not be taking the 92 items as a requirement in their review of models. She asked if there is a way to pare down the list to the priority items for the states that do not have a large staff to review these models. Patrick Foltyn (The Cincinnati Insurance Companies) said the rules seem prescriptive, noting that a principle-based approach would be more prudent. He said flexibility is needed. Robert Curry (Insurance Services Office—ISO) said the ISO has made rate filings with GLMs for 10 years, and no state insurance department has ever requested all the information included in the white paper; as such, he said some streamlining is recommended. He said there is some concern about the data underlying the analysis required; i.e., the data is proprietary and could have personally identifiable information. He asked whether state insurance regulators need the data, given that traditionally the policy-level information to trace through an analysis has never been requested. The focus should be on the filing and what model is being used. The state insurance regulator should not be looking to see if a better model could have been developed by a different modeler. He said the Task Force appears to be trying to extend federal Fair Credit Reporting Act (FCRA)-type protections to non-FCRA-type data. ASOP No. 12 does not require an actuary to establish a cause-and-effect relationship. He said it is good to attempt to do that, but it should not be a requirement, given that it is not always possible to do so. Mr. Kabler said data-mining and similar techniques differ from the traditional scientific method. With data-mining, there is not necessarily a known causal understanding or a hypothesis to test. The relationships and what they mean are not known. They may go through thousands of variables, resulting in many false positives. The p value will say the relationship is not likely random even when it is random. He said the ability to explain causality is greatly diminished. He said there is a causal relationship between youth and driving; however, no one knows the causal explanation for credit scoring. He said ASOPs raise the issue of causality. He said it is not that actuaries should dispense with causality entirely; only that actuaries are not required to prove causality. He said random correlations are going to make their way into rating systems, and this merits more attention in the white paper. He said the American Statistical Association (ASA) has discussed these issues, and it recommends not using data-mining. He said he does not recommend that, because causality is not the primary concern. Mr. Davis said, in the rush to market, some of these variables may be wrong; therefore, they could easily be unfair. Mr. Kabler said there may be remedial measures that could be incorporated into the review process, such as the adjustment of p values to account for having many variables and using hold-out datasets. Michelle Rogers (National Association of Mutual Insurance Companies—NAMIC) said the white paper is thorough, but it might be getting too prescriptive with the large number of items of requested information. She said there needs to be an emphasis in the white paper on flexibility. She said the cost of rate filing reviews could get too costly, impact speed to market and innovation and, ultimately, impact consumers. She suggested identifying what is truly essential in every rate review, eliminating those that are unnecessary, and identifying other information that could be requested if the need is demonstrated.


4/6/19


Mr. Stolyarov said consistency across the states, rather than uniformity, should be the goal. He said the states have common areas of focus. He said the states are not going to request all the information listed. He said the correlation-only argument is a misunderstanding of ASOP No. 12. While ASOP No. 12 says it is not necessary to establish a cause-and-effect relationship, an actuary should still select risk characteristics that are related to expected outcomes. He said senior modelers recognize a need to consider causality, why a variable might be predictive and/or what behaviors the variables are trying to represent. He said the use of large datasets will result in false positives and treatments that do not make sense. He added that the states protect confidentiality. He said some items do not make sense, such as penalizing for paying bills on time, not having auto loans or not carrying a balance on credit cards. He said the white paper does not prescribe any particular treatment but sets forth questions state insurance regulators could ask. Ryan Purdy (Merlinos & Associates, Inc.) asked how the state insurance regulators are going to review and adopt the guidelines and best practices if they do not have access to a modeler. He questioned whether there might be a reasonable approach where the states without access to experts could have alternative guidelines, such as evaluating controls over the process. He asked whether the controls are in place. He also asked whether the company’s experts are qualified. He said this would be similar to the risk-focused approach used in financial analysis and examination. Mr. Serbinowski said the project will prove useful to state insurance regulators. He has less concern with correlation errors, given that insurers are not interested in spurious correlations because they are detrimental to the insurers’ business, so those would be weeded out over time. He said he is concerned with consumers being able to edit the data. Gender and credit scoring have been used for a long time, and no causality has been explained in a rigorous way. Mr. Vigliaturo said the volunteer drafters will sort through over a hundred pages of comments and propose changes to the white paper. He said the Task Force will discuss a revised version at the Spring National Meeting. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\3-12 CASTF min.docx

Attachment Three Casualty Actuarial and Statistical (C) Task Force

4/6/19


Draft: 3/1/19

Casualty Actuarial and Statistical (C) Task Force Conference Call

February 12, 2019 The Casualty Actuarial and Statistical (C) Task Force met via conference call Feb. 12, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo and Connor Meyer (MN); James J. Donelon, Vice Chair, represented by Rich Piazza and Lawrence Steinart (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Ricardo Lara represented by Giovanni Muzzarelli and Mitra Sanandajifar (CA); Michael Conway represented by Mitchell Bronson (CO); Paul Lombardo represented by Susan Andrews (CT); Stephen C. Taylor represented by David Christhilf (DC); David Altmaier represented by Howard Eagelfeld and Robert Lee (FL); Colin M. Hayashida represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Kevin Fry represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Julie Lederer (MO); Marlene Caride represented by Mark McGill (NJ); John G. Franchini represented by Mark Hendrick (NM); Barbara D. Richardson represented by Gennady Stolyarov (NV); Jillian Froment represented by Thomas Botsko (OH); Glen Mulready represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Raymond G. Farmer represented by Michael Wise (SC); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka, Elizabeth Howland and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). Also participating was: Gordon Hay (NE). 1. Received a Report from the Actuarial Opinion (C) Working Group Mr. Vigliaturo reported that he appointed Ms. Lederer as chair of the Actuarial Opinion (C) Working Group. Ms. Lederer said the Working Group did not propose changes to 2019 Statement of Actuarial Opinion instructions because of the actuarial qualifications project. The Working Group has not met in 2019. 2. Received a Report from the Statistical Data (C) Working Group Mr. Vigliaturo reported that he appointed Carl Sornson (NJ) as chair of the Statistical Data (C) Working Group. Mr. McGill said the Working Group plans to continue to review the formulas in the Report on Profitability by Line by State (Profitability Report). There are no other planned changes to other reports. Mr. Lee said the “dwelling fire” data in the Dwelling Fire, Homeowners Owner-Occupied, and Homeowners Tenant and Condominium/Cooperative Unit Owner’s Insurance Report (Homeowners Report) is only fire, and it does not include allied lines. He said the allied lines include all the weather-related categories, and they are the majority of what Florida would consider to be “dwelling fire.” Mr. McGill said that is a new issue, and he will inform Mr. Sornson. 3. Discussed its 2019 Charges and Work Plan

Mr. Vigliaturo addressed the Task Force’s 2019 charges and top priorities. He said the first three charges include monitoring other groups. He said the Task Force will continue to monitor other work that might be of interest and report back to the Task Force. He said he will ask Wanchin Chou (CT) to continue on as a liaison with the NAIC’s Own Risk and Solvency Assessment (ORSA) work. To facilitate regulatory discussion regarding filing issues, Mr. Vigliaturo said there will be monthly conference calls. He asked regulators to forward any agenda items or topics to be addressed. Mr. Vigliaturo said the Task Force had three 2018 charges to address appointed actuary issues, which were generally referred to as: 1) the attestation; 2) the three-year experience period; and 3) the continued competence charges. He said the currently exposed 2019 Statement of Actuarial Opinion instructions includes the Task Force’s proposals to address the attestation and the three-year experience period charges. He said with adoption of those changes, the charge will be considered completed.


4/6/19


Mr. Vigliaturo said the 2018 continued competence charge was revised for 2019. In 2018, the Task Force adopted a plan presented jointly by the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA). He asked Mr. Dyke to continue to liaise with the CAS and the SOA to work on the 2019 charges. Mr. Dyke agreed. Mr. Vigliaturo said a major focus for the Task Force this year is the project requested by the Big Data (EX) Working Group to draft best practices for the review of predictive models and analytics filed by insurers to justify rates. He said the Task Force decided to begin this work by drafting a white paper.

Mr. Vigliaturo said the predictive analytics “Book Club” conference calls are scheduled for the rest of this year. He said there are currently no selected papers or articles to read nor speakers scheduled, so he asked for Task Force members to send suggestions to NAIC staff. He asked interested parties to volunteer to speak.

4. Adopted a Comment Letter on the Statement of Actuarial Opinion Instructions

Mr. Vigliaturo said the property/casualty (P/C) Statement of Actuarial Opinion instructions were released for a public comment period ending Dec. 15, 2018. The Task Force held calls Jan. 29, 2019; Jan. 8, 2019; and Dec. 18, 2018, to discuss the changes proposed by the Executive (EX) Committee’s ad hoc group. During the Jan. 29 conference call, the Task Force discussed a potential comment letter to send to the ad hoc group. Some volunteers on the Task Force redrafted the comment letter in light of that discussion.

Mr. Vigliaturo said the aim of the Task Force is to present agreed-upon changes and then include a second part of the letter to provide accurate information on both sides of the potential change to require American Academy of Actuaries (Academy) membership. The aim is not to debate the two sides of the issue, but it is only to fairly and accurately represent both sides of the issue. The Task Force discussed the grandfathering clause. The Task Force agreed that the general idea is that actuaries who were qualified prior to the revised definitions should continue to be qualified. There was question as to whether the grandfathering should be broader to include anyone who would have qualified previously if the new definition had been in place. Mary Downs (Academy) said the actuarial credentials were deemed by WorkCred to not be equivalent. She said there will likely be confusion with appointed actuaries regarding what they should do with the current wording. Mr. Stolyarov said the WorkCred study is not considered authoritative by Nevada because the study has not been reviewed. He said the word “grandfathering” could be changed if that is confusing. Mr. Dyke said the WorkCred study cannot be ignored without further understanding or some deliberation by the ad hoc group. He said if the exam requirements are substantially similar to before the WorkCred study was completed, then the current study could be applied retroactively, but he is not sure that is the case. Mr. Davis said the executive summary was distributed, and it seems to support Ms. Downs’ statement. Mr. Stolyarov said he would need to see the complete study and not just the executive summary. Mr. Davis said changes will be made in the NAIC’s Educational Standards and Assessment Project so that both organizations will be able to make changes and meet the minimum educational standards. He said his understanding is the assessment project would have looked at prior years and been able to advise about the syllabi in place each year. Mr. Stolyarov said the current project is extensive, objective and rigorous, and it will be public. Kris DeFrain (NAIC) said the current study is a point-in-time assessment of whether the syllabus meets the standards. Mr. Davis said it seems there might need to be additional information to determine the appropriate grandfathering clause. Mr. Hay said if the bar was not cleared on the first pass, then there would be an evolution. Ms. DeFrain said that is accurate, in that the CAS and the SOA will have two years to make any changes to their syllabus to meet the NAIC’s minimum standards. Ralph Blanchard (Travelers) said an Associate Casualty Actuarial Society (ACAS) was all that used to be needed to be qualified; now, the ACAS needs an additional reserving exam. He said restrictions have been in place in the past.


4/6/19


Mr. Stolyarov said the Task Force should also consider that professionals take continuing education and gain experience after they receive their designation, so professionals are not asked to retake the current exams when the syllabi change. The Task Force noted significant reservations in regard to proposing acceptable wording without knowing the restrictions to be placed on the NAIC accepted actuarial designations and/or any other changes the ad hoc group might make. The Task Force agreed to ask for another consultation with the ad hoc group once those restrictions and all other changes are known. Mr. Stolyarov and Mr. Dyke presented proposed changes to the second part of the letter regarding the potential change to require Academy membership. Mr. Stolyarov made a motion, seconded by Mr. Piazza, to adopt the comment letter as revised (Attachment Three-A). The motion passed unanimously. Mr. Vigliaturo asked NAIC staff to finalize the letter and submit it to the ad hoc group by the Feb. 15 comment deadline. 5. Discussed the Best Practices for Regulatory Review of Predictive Analytics White Paper Mr. Vigliaturo said Mr. Piazza will stay in the lead to complete the white paper this year. He said the Task Force discussed the white paper during an Oct. 9, 2018, conference call, and it exposed the paper for a 60-day public comment period ending Feb. 12, 2019. The comment period was subsequently extended until Feb. 19, 2019. Numerous comments were received (Attachment Three-B). Mr. Piazza said the target completion date for the white paper is the Fall National Meeting. He said the next steps will be to take the hundreds of pages of comments and get them put in order for easier review. The drafting group will then document the comments and propose changes to the white paper. Birny Birnbaum (Center for Economic Justice—CEJ) said other groups have addressed a large number of comments such as these by using tables and grouping comments on the same section together. Mr. Piazza agreed. He said the Task Force can re-expose the paper. 6. Discussed the Global Insurance Symposium Answering Mr. Piazza’s question, Ms. Seip said Dorothy Andrews (Insurance Strategies Consulting LLC) will be providing regulator-only predictive analytics training around the Global Insurance Symposium, which is being held April 23–25 in Des Moines, IA. Ms. Darby said the training is before and after the symposium. Ms. Seip said she would send NAIC staff additional information for distribution. Mr. Stolyarov asked for a more detailed description of the training. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\2-12 CASTF min.docx

Attachment Three-A Casualty Actuarial and Statistical (C) Task

4/6/19

To: Executive (EX) Committee Ad Hoc Group — Commissioner Jim L. Ridling, Commissioner James J. Donelon, Superintendent Eric A. Cioppa

From: Phil Vigliaturo, Chair of the Casualty Actuarial and Statistical (C) Task Force

Date: February 12, 2019

RE: Exposure of Property/Casualty Statement of Actuarial Opinion Instructions

On December 18, 2018, January 8, January 29, and February 12, 2019, the Casualty Actuarial and Statistical (C) Task Force (“CASTF”) held conference calls that were open to the public, for the purpose of discussing the exposure draft of the revisions to the Property/Casualty (P/C) Statement of Actuarial Opinion Instructions, currently exposed through February 15, 2019. The focus of CASTF in these discussions was on the changes proposed by the Executive (EX) Committee’s ad hoc group.

The purpose of this letter is to document items of consensus that emerged during those conference calls, as well as a major area of difference among CASTF members regarding one matter relevant to the exposure draft.

Items of Consensus Among CASTF Members

The CASTF members achieved agreement regarding the following four desirable changes to the pending draft:

1. In 1A Definitions, item iii: Leave the “or” in the first line.

2. Change the paragraph after 1A Definitions, item i-iv, to read:

“An exception to parts (i) & (ii) of this definition would be an actuary who is a member of the Academy evaluated by the Academy’s Casualty Practice Council and determined to be a Qualified Actuary for particular lines of business and business activities. Should an actuary qualify under this alternate route, the actuary must attach a copy of the approval letter from the Academy to the Actuarial Opinion each year.”

Membership in the Academy is a current requirement for those actuaries who need to be evaluated by the Academy’s Casualty Practice Council.

3. In 1A Definitions, item ii, add language to emphasize there may be restrictions on acceptable designations:

“has (a) obtained an NAIC Accepted Actuarial Designation, subject to any noted restrictions, or (b) became a member in good standing of the Casualty Actuarial Society prior to January 1, 2021; and”

The noted restrictions are expected to come from the NAIC’s Educational Standards and Assessment Project.

4. Revise the grandfathering section (after paragraph 1A Definitions, items i-iv) to emphasize there may be restrictions on acceptable designations as follows:


4/6/19


Replace “The listed actuarial designations earned prior to January 1, 2021 are grandfathered as accepted.” with the following: “The listed actuarial designations earned prior to January 1, 2021, subject to any noted restrictions, are grandfathered as accepted.”

CASTF believes there might be additional issues to discuss once we know the restrictions placed on the NAIC Accepted Actuarial Designations (resulting from the NAIC’s Educational Standards and Assessment Project). This might impact our proposal for grandfathering and/or cause other concerns. Once the NAIC’s Educational and Assessment Project is completed, we would appreciate discussing the instructions with the ad hoc group prior to the instructions being finalized.

The remainder of this letter explains why membership in the American Academy of Actuaries (Academy) should or should not be a requirement for satisfying the definition of Qualified Actuary in order to be a P/C Appointed Actuary.

Major Area of Difference Among CASTF Members: Whether to Require Membership in the Academy for Appointed Actuaries

The major difference expressed among the CASTF members on the calls of December 18, 2018, and January 8, 2019, pertained to whether or not membership in the Academy should be required for P/C Appointed Actuaries to meet the definition of a “Qualified Actuary”. In the pending draft, sub-item (iv) of the definition of a “Qualified Actuary” would require the actuary to be “a member of a professional actuarial association subject to the same Code of Professional Conduct promulgated by the Academy, the U.S. Qualification Standards, and the Actuarial Board for Counseling and Discipline when practicing in the U.S.” Some CASTF members wished to replace this with a requirement to be a Member of the American Academy of Actuaries (MAAA). Other CASTF members supported the above wording as currently drafted. The purpose of this portion of the letter is to identify the existence of this disagreement and to provide the key arguments for each of the positions on this issue.

CASTF has discussed whether membership in the Academy should be included in the definition of Qualified Actuary. The original definition proposed by the Executive (EX) Committee’s ad hoc group (“ad hoc group”) on December 29, 2017, included a membership requirement, but this was subsequently removed based on the ad hoc group’s review of comments on the proposal.

Following the latest exposure, some members of CASTF voiced support on its January 8, 2019, call for replacing the broad membership definition in item iv. with a more specific definition requiring membership in the Academy, as follows under Section 1A:

iv. is a Member of the American Academy of Actuaries (MAAA) member of a professional actuarial association subject to the same Code of Professional Conduct promulgated by the Academy, the U.S. Qualification Standards, and the Actuarial Board for Counseling and Discipline when practicing in the U.S..

Further discussion was held, with other members of CASTF voicing support for the currently proposed definition. Despite the lack of consensus on the definition, CASTF believes it is important for the Executive (EX) Committee’s ad hoc group to hear the arguments for and against requiring Academy membership.

Reasons Provided for Why Membership in the Academy Should Be a Requirement

1. The Academy is focused solely on the U.S. actuarial profession: Unlike other U.S. based actuarial organizations, the Academy serves members performing actuarial services in the U.S. only, with U.S. based membership criteria not duplicated by any other U.S. based organization. Non-residents and resident aliens of fewer than three years are required to describe their actuarial work experience and need for Academy


4/6/19


membership and obtain a letter of reference attesting to their work in and knowledge of U.S. standards and practices. Academy membership would assure that non-resident members of the SOA and CAS would have sufficient U.S. based experience to issue actuarial opinions in the U.S. Specific requirements can be found in items 1 and 2C in the Academy’s membership form (attached). The Academy is currently the only U.S. based actuarial organization that performs this screening of non-residents and resident aliens of fewer than three years. Without requiring Academy membership, there is no such assurance for either a Board of Directors or the state regulators.

2. Academy membership has been established as the standard for qualifying actuaries: The Academy was formed in 1965 by the other existing actuarial organizations, including the Casualty Actuarial Society (CAS) and the Society of Actuaries (SOA), to be the credential recognized as indicating qualification for professional accreditation. As a result, nearly all NAIC model laws and regulations, and Federal laws and regulations require Academy membership when an actuarial opinion is required. Not including Academy membership in the definition of P/C Qualified Actuary would deviate from the historical standard implemented at state and federal levels.

3. Academy membership supports the continued professionalism of actuaries. The Academy is the home for the Committee on Qualifications, the Actuarial Standards Board, and the Actuarial Board for Counseling and Discipline. The Academy also promotes high level of professionalism of actuaries in their practice through its Council on Professionalism, which includes representatives from the above three organizations as well as the CAS and SOA.

4. The Academy serves the public interest. The Academy’s mission is to serve the public interest by providing non-partisan objective actuarial experience on U.S. public policy issues at both the state and federal level. The NAIC has benefited from the Academy’s expertise for years in drafting model laws, regulations, tables, and guidelines.

5. The Academy specifically supports P/C Appointed Actuaries with tools to improve the quality and reliability of P/C actuarial opinions. The Academy annually produces a P/C Loss Reserve Law Manual, which is a compilation of insurance laws on loss and loss expense reserves in all 50 states, DC, and Puerto Rico. Academy members receive a $400 discount on the purchase of the manual. The Academy also offers a two-day Seminar on Effective P/C Actuarial Opinions providing a deeper understanding of the laws, regulations, and qualifications for issuing actuarial opinions. Academy members receive a $100 discount on the registration fee. (Regulators may attend at no cost). Finally, the Academy produces an annual practice note providing common practices for issuing the P/C actuarial opinions and reports.

6. Other countries require membership in their national actuarial organization. Appointed actuaries in Canada must be members of the Canadian Institute of Actuaries (CIA), regardless of how they received their basic education. Many actuaries in Canada obtain their education from the CAS and SOA, which are recognized through mutual recognition to grant membership in the CIA. We understand Germany and Mexico have similar requirement for membership in their societies for actuaries who did not receive their education in those countries.

7. Academy membership is practical and understandable. Membership in the Academy is easier for a Board of Directors and state regulators to understand and verify.

8. Academy membership does not diminish the value of basic and continuing education provided by the CAS and SOA: Both the CAS and SOA have established syllabi of basic education that, if approved, will remain a vital component of an actuary’s qualifications.


4/6/19


Reasons Provided for Why Membership in the Academy Should Not Be a Requirement

1. Membership in the Academy is not a current requirement in the P/C Statement of Actuarial Opinion Instructions. Not including such a membership requirement would preserve the status quo in this regard and would enable the CASTF to focus on the specific charges that motivated the proposed revisions to the Actuarial Opinion Instructions in the first place. Adding a membership requirement is another way to say all Appointed Actuaries must pay $675 Academy membership dues.

2. One purpose of redefining “Qualified Actuary” is to use an objective measure rather than to support membership in any particular organization. This appears to be the reason Sub-item (iv) in the pending draft’s definition of a “Qualified Actuary” requires the actuary to be “a member of a professional actuarial association subject to the same Code of Professional Conduct promulgated by the Academy, the U.S. Qualification Standards, and the Actuarial Board for Counseling and Discipline when practicing in the U.S.” This would, by itself, subject the actuary to the professionalism frameworks and disciplinary processes created by the Academy, whether or not the actuary is a member of the Academy. Regulators generally regard membership in the Academy positively. Membership in the Academy may entitle the actuary to take a more active role in the Academy’s work and decision-making processes, benefit from the Academy’s educational and networking opportunities, and participate in the policy discussions in which the Academy is engaged. In exchange, the actuary would pay Academy membership dues. However, the Academy’s membership dues and the benefits that they enable an actuary to access are not indispensable prerequisites to issuing a statutory Statement of Actuarial Opinion in connection with the loss and LAE reserves carried on an insurer’s P/C Annual Statement. That is, an individual is capable of being objectively qualified through the requisite combination of basic education, experience, and continuing education to issue such statutory Statements of Actuarial Opinion without paying Academy membership dues or taking advantage of the benefits offered by the Academy.

Beyond recognition of the professionalism frameworks and disciplinary processes created by the Academy, as well as the specific pathway offered through the Academy’s Casualty Practice Council, making Academy membership mandatory for all P/C Appointed Actuaries would create concerns about privileging one private actuarial organization over others. While supporters of such a mandate made the analogy between Academy membership for actuaries and State Medical Board licensure for medical practitioners, the more fitting analogy in the medical realm to Academy membership would be membership in the American Medical Association or a similar private professional (and policy advisory) group, which may have specific interests that are distinct from the interests of regulators or the public whom the regulators are explicitly tasked to protect. It is not the purpose of the Actuarial Opinion Instructions to make membership in a private interested-party organization a prerequisite for fulfilling an essentially public purpose – the preparation of statutory Statements of Actuarial Opinion – for which the intended users are regulators and the intended beneficiaries are members of the public.

3. Concerns about the qualifications of international actuaries should not be the basis for a new requirement that would apply to the vast majority of Appointed Actuaries who are based in the United States. If the Academy’s primary concern pertains to international actuaries who may have obtained basic education that may not have covered U.S. laws, whereas the Academy membership application requires such individuals to certify their familiarity with U.S. laws and practices in their actuarial practice area, then the way to resolve this would be the consensus change #2 agreed upon by the CASTF: to require the MAAA designation only from those actuaries who elect to be evaluated by the Academy’s Casualty Practice Council – since the actuaries who do not elect this option would be subject to the basic education requirements, which themselves include coverage of U.S. laws. However, to fulfill the above purpose, it is not necessary to require all or even the vast majority of P/C Appointed Actuaries to have Academy membership.


4/6/19


4. Academy membership for Appointed Actuaries is not an Accreditation Standard and is only in a few states’ laws. The Actuarial Opinion Instructions are ultimately subordinate to State law and are not the proper vehicle for creating new law or new requirements. It is true that a few States’ laws already mandate Academy membership for P/C Appointed Actuaries – but this is not the case for all States, and the laws of specific NAIC-Accredited States were cited in which Academy membership was not a requirement. For those States where Academy membership is currently a requirement, the laws of those States would prevail over the Actuarial Opinion Instructions in any event. However, the Actuarial Opinion Instructions should not unilaterally set a more stringent bar than currently exists pursuant to the laws of any NAIC-Accredited States. The NAIC has a specific process for amending its model statutes and regulations, and even subsequent to such amendments, decisions would need to be made at the NAIC level as to whether to designate the amended model laws as Accreditation standards, and decisions would also need to be made at the level of a State’s Legislature as to whether to enact any given model statute and at the level of a State’s regulatory authority as to whether to enact any given model regulation. These processes need to be followed by those wishing to introduce any new mandates for membership in any specific actuarial organization.

5. While supporters of mandating Academy membership have cited similar mandates in the areas of practice of Life and Health Insurance, it remains the case that model laws and regulations pertaining to statutory Statements of Actuarial Opinion are distinct between P/C, Life, and Health areas of practice. Often the differences are material and lengthy; they reflect significant inherent differences in the lines of business subject to the statutory Statements of Actuarial Opinion. Accordingly, any specific reasons for requiring Academy membership for Appointed Actuaries in Life and/or Health areas of practice may not be germane to the P/C area of practice. Each area of practice should be considered from the standpoint of the issues, developments, and products specific to that area, and attempts to “harmonize” requirements across areas of practice have historically not succeeded and will often not produce a desirable result due to failure to recognize and respect area-specific differences, nuances, and needs.

6. The current proposal is a principle-based approach. The main purpose of the contemplated revisions to the Actuarial Opinion Instructions was to create a pathway for recognition of NAIC-Accepted Actuarial Designations that will be evaluated through the objective Educational Standards and Assessment process. This is a principles-based approach that would consider the content of an actuarial organization’s offerings instead of making membership in any specific organization a per se requirement. Mandating membership in the Academy (or any singular organization identified by name) would be in conflict with such a principles-based approach and would revert to the approach of “hard-coding” a specific organization into the Actuarial Opinion Instructions. Such “hard-coding” is insufficiently flexible to address evolving developments in the profession, for responding to which a principles-based approach offers more versatility.

7. Ultimately the Actuarial Opinion Instructions support a State-based system of insurance regulation, where regulatory authority rests in the States. Some individual States may favor mandating Academy membership for Appointed Actuaries and may choose to do so within the framework of their own laws. (Indeed, several States have done this.) However, in any situation where material differences exist among the States with regard to such an issue, the preferred approach is to “agree to disagree” – allowing individual States to pursue their own approaches in accordance with their laws, while the Actuarial Opinion Instructions should reflect the baseline requirements that are compatible with the laws and regulatory philosophies in all NAIC-Accredited member jurisdictions. The very existence of material differences of position on the issue of requiring Academy membership for Appointed Actuaries can be seen as an argument for not requiring it at this time.

Should you have questions, please contact Phil Vigliaturo, Chair of CASTF.

Cc: Kris DeFrain (NAIC)

1850 M Street NW Suite 300 Washington, DC 20036 Telephone 202 223 8196 Facsimile 202 872 1948 www.actuary.org

January 22, 2019

Kris DeFrain, FCAS, MAAA, CPCUDirector of Research and Actuarial Services National Association of Insurance

Commissioners (NAIC) Central OfficeVia Email

Re: CASTF Regulatory Review of Predictive Models White Paper

Dear Kris:

As the American Academy of Actuaries’1 senior property/casualty fellow, I appreciate this opportunity to comment on the white paper exposure draft of the Casualty Actuarial and Statistical Task Force (CASTF) discussing best practices for the Regulatory Review of Predictive Models (RRPM). To provide input, we have sought guidance from two Academy groups: the Automobile Insurance Committee of the Casualty Practice Council (CPC) and the Big Data Task Force.

I agree with the CASTF that “Insurers’ use of predictive analytics along with big data has significant potential benefits to both consumers and insurers.” Though predictive analytics is still in its early stages of use in insurance ratemaking, benefits are being realized. Along the way, the insurance industry has committed resources to fund and staff the development of predictive analytics projects. Actuaries have played a central role in this development.

In 2017, the CPC conducted a daylong seminar at the NAIC’s Insurance Summit to help familiarize regulators with predictive modeling including how it relates to public policy issues.In 2018, the Academy produced a monograph, Big Data and the Role of the Actuary, which includes extensive sections on regulatory and professionalism considerations.

The RRPM white paper is comprehensive in its scope. The need for a set of best practices in the review process is well noted. Additionally, there are 92 potential information items identified in the paper for use in reviewing model submission. I offer the following for CASTF’s consideration:

1 The American Academy of Actuaries is a 19,500+ member professional association whose mission is to serve the public and the U.S. actuarial profession. For more than 50 years, the Academy has assisted public policymakers on all levels by providing leadership, objective expertise, and actuarial advice on risk and financial security issues. The Academy also sets qualification, practice, and professionalism standards for actuaries in the United States.

Attachment Three-B Casualty Actuarial and Statistical (C) Task Force

4/6/19



The CASTF’s charge to develop best practices for reviewing predictive rating models is an important goal. However, there is potential for the process to become unmanageable for both the modelers and the reviewers. Is there opportunity to explore whether the suggested RRPM process might be unwieldy for regulators? Would insurance consumers be better served by a more collaborative framework to assess issues?Confidentiality is addressed in two places. In the first instance, confidentiality is cited as a key regulatory principle, where appropriate. Later in the paper, it is addressed directly in that insurers are guided to know the specifics of state regulations regarding confidentiality of rate filings. Could there be a separate discussion of this issue that would seek to come up with alternative methods of model review that are acceptable to modelers and regulators?On page three of the RRPM, it states “GLM (generalized linear model) output is assumed, as part of the model design, to be 100% credible no matter the size of the underlying data set.” It should be noted that a GLM produces both a parameter estimate and a standard error around that parameter estimate. The standard error along with claim volume consideration is applied in the selection of proposed relativities.Information items A.1.a and A.1.d seem to be addressing the same concern. Could the differences between A.1.a and A.1.d be clarified? Information item A.5.a references the potential for submitting raw data. Could this prove problematic as to the security of insurance consumer data? Will regulators be attempting to recreate models? The sheer size of these data sets could make them difficult to provide. Would sample data set structures meet the needs of this information item? Finally, there might be issues regarding contractual relationships with third-party data providers and their resources.The actuarial standards of practice (ASOPs) are undoubtedly a valuable resource. The ASOPs establish standards for “appropriate” actuarial practice in the United States (ASOP No. 1, Introductory Actuarial Standard of Practice, Section 1). At a minimum, ASOP No. 12, Risk Classification (For All Practice Areas), and ASOP No. 41, Actuarial Communications, are applicable. Should those documents be explicitly considered in the best practices for RRPM?Item B.3.c addresses the intuitiveness of the predictor variables. Is it the intent of thisitem to prohibit variables that are not explicitly intuitive? Actuarial principles state that an intuitive argument is desirable, but not required. Item C.2.a appears to be addressing causal effect of the predictor variables. Could you please clarify this?In the introduction, there is a statement that “the insurer must anticipate … the reviewers’ interests because the reviewers will respond with unanticipated questions.” Anticipating the unanticipated can prove challenging. Likely, this comment is addressing a communication-related issue. Could you please clarify?

In closing, I wish to reiterate that the American Academy of Actuaries remains committed to working with CASTF as regulators strive to understand and monitor the growing role of predictive modeling in insurance rate development. We look forward to participating in the ongoing dialogue on the RRPM to help to achieve a thoughtful and effective review process.

If you have any questions about these comments, contact me ([email protected]) or Marc Rosenberg, senior casualty policy analyst, at 202-785-7865 or [email protected].


4/6/19



Sincerely,

Richard Gibson, MAAA, FCASSenior Casualty FellowAmerican Academy of Actuaries


4/6/19


From: Ricker, Michael D (CED) <[email protected]> Sent: Tuesday, January 22, 2019 2:24 PM To: DeFrain, Kris <[email protected]> Subject: RE: Regulatory Review of Predictive Models White Paper - Comments Due Jan. 15

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Kris, a couple comments from AK regarding the 10/25/18 draft: 1. In section VII, it’s not clear how table C, “The Filed Rating Plan”, coordinates with tables A and

B. Are the items in A and B not expected to be included in rate/rule filings? Where/when/how are regulators expected to be accessing those pieces of information if not within rate/rule filings? Perhaps language can be added to make the difference between tables C and A&B more clear; the current intro language in section VII seems to imply that all of the information presented in section VII (i.e. all three tables) is for the purpose of providing supporting documentation within a rate/rule filing.

2. C.1.a appears to be referring to a state-specific SERFF Requirement. Alaska, for one, does not have an “Actuarial Memorandum section on the SERFF Supporting Documentation tab”.

Michael Ricker Property & Casualty Actuary Alaska Division of Insurance | P.O. Box 110805 | Juneau, AK 99811-0805 Phone: 907.465.2564 | Fax: 907.465.3422 | DOI Main phone: 907.465.2515 http://www.commerce.alaska.gov/web/ins [email protected]


4/6/19


Dear Mr. Piazza,

Thank you for the opportunity to comment on the draft white paper “Regulatory Review of Predictive Models” by the Casualty Actuarial and Statistical Task Force (CASTF).

Below are areas of concern that we would like to address:

The white paper asks companies to provide evidence of a causal relationship between risk characteristics and expected cost.

From CASTF white paper: B.3.c - Provide an intuitive argument for why an increase in each predictor variable should increase or decrease frequency, severity, loss costs, expenses, or whatever is being predicted. C.2.a - Provide an explanation how the characteristics/rating variables, included in the filed rating plan, logically and intuitively relate to the risk of insurance loss (or expense) for the type of insurance product being priced.

From “ASOP 12: Risk Classification” While the actuary should select risk characteristics that are related to expected outcomes, it is not necessary for the actuary to establish a cause and effect relationship between the risk characteristic and expected outcome in order to use a specific risk characteristic.

The white paper asks companies to provide information and data such that a regulator would be reproducing, rather than reviewing, the filed model.

From CASTF white paper: A.5.a - If the raw data selected to build the model is in a format that can be made available to the regulator, provide it. A.3.f - What adjustments were made to raw data, e.g., transformations, binning and/or categorizations? If so, name the characteristic/variable and describe the adjustment. B.6.b - What software was used? Provide the name of the software vender/developer, software product and a software version reference.

From “ASOP 41: Actuarial Communication” In the actuarial report, the actuary should state the actuarial findings, and identify the methods, procedures, assumptions, and data used by the actuary with sufficient clarity that another actuary qualified in the same practice area could make an objective appraisal of the reasonableness of the actuary’s work as presented in the actuarial report.

Additional Comments We also consider raw data to be proprietary and we do not wish to provide such data due its confidential nature. It is of the utmost importance that any sample raw data provided be protected to conceal the private information of insureds.

The white paper places a large emphasis on state-specific results.

From CASTF white paper:


4/6/19


C.6.a - Provide state-specific, book-of business-specific univariate historical experience data consisting of, at minimum, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output(s) proposed to be used within the rating plan. C.6.b - Provide an explanation of any material (especially directional) differences between model indications and state-specific univariate indications. B.4.b – Some states require state-only data to test the plan, especially for analysis where using the state-only data contradicts the countrywide results. State-only data might be more applicable but could also be impacted by low credibility for some segments of risk. B.4.j - Describe how the model was tested for geographic stability, e.g., across states or territories within state.

From “A Practitioners Guide to Generalized Linear Models”: Depending on the underlying claim frequencies and the number of factors being analyzed, credible results on personal lines portfolios can generally be achieved with around 100,000 exposures (which could for example be 50,000 in each of two years, etc). Meaningful results can sometimes be achieved with smaller volumes of data (particularly on claim types with adequate claims volume), but it is best to have many 100,000s of exposures.

Comments: In most cases, state-specific results will lack credibility for a multivariate model to achieve meaningful results and therefore not be useful for comparisons. The state-specific data may be misused and result in a less predictive model if countrywide patterns are overridden with low-volume state-specific data. If every difference between state-specific results and countrywide results were investigated, this would dramatically increase required filing time and speed-to-market would be greatly reduced.

The white paper places a large emphasis on univariate indications.

From CASTF white paper: C.6.a - Provide state-specific, book-of business-specific univariate historical experience data consisting of, at minimum, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output(s) proposed to be used within the rating plan. C.6.b - Provide an explanation of any material (especially directional) differences between model indications and state-specific univariate indications. Multivariate indications may be reasonable as refinements to univariate indications, but likely not for bringing about reversals of those indications.

Comments: It is generally accepted in the industry that multivariate indications are more accurate than univariate indications. Univariate indications alone should not be used to override multivariate model results. A clarification is also needed for how the paper uses the term “univariate indication”.

Modelers generally refer to “univariates” as a comparison of actual results vs predicted results by variable for the statistic being modeled, such as pure premium. However, a univariate indication can also be based on a loss ratio analysis.

It would be problematic to require a loss ratio for all model filings when loss ratio is not the target variable. For consistency and interpretability, loss ratio analyses require bringing all premiums to a common set of rating plan factors. This requires extensive work for a countrywide analysis and possibly significant work even within a state if the experience


4/6/19


period includes several rate filings. This extraneous effort is not justified when the loss ratio is not the target variable.

The white paper contains items that difficult or inappropriate to provide.

From CASTF white paper: B.6.a - Provide the names, contact emails, phone numbers and qualifications of the key persons who: a. Led the project b. Compiled the data c. Built the model d. Performed peer review

Comments Inappropriate to ask for contact information of employees in public documents. Filings should be considered the work product of the insurance company and a single point of contact should be used for questions. In traditional actuarial filings, the names, qualifications, and contacts of every person that worked on the filing is not provided. In some regulatory settings, one qualified actuary will sign off on the work.

From CASTF white paper: o C.2.a - Include a discussion of the relevance each characteristic/rating variable has on consumer

behavior that would lead to a difference in risk of loss. Comments

This does not provide information that is useful with respect to the strenuous nature of the ask. From CASTF white paper:

C.3.a – Provide a comparison between relativities indicated by the model to both current relativities and the insurer's selected relativities for each risk characteristic/variable in the rating plan. Each significant difference should be highlighted and explained.

Comments: Onerous to include explanation for every difference. Comparison is not possible when rating algorithms/structures are different. Focus of model comparisons should be on overall model performance rather than individual characteristics.

Once again, thank you for the opportunity to comment. Sincerely, Allstate Property & Casualty Actuarial Leadership For any questions, please contact: Mike Woods, FCAS, CSPA Allstate Insurance Company 2775 Sanders Rd Northbrook, IL 60062 [email protected]


4/6/19


Kris DeFrain, FCAS, MAAA, CPCU Director, Research and Actuarial Services National Association of Insurance Commissioners (NAIC)

Sent via e-mail at [email protected]

January 22, 2019

Dear Ms. DeFrain:

Thank you for the opportunity to comment on the NAIC Casualty Actuarial and Statistical Task

Force (CASTF) proposed exposure draft regarding the Regulatory Review of Predictive Models currently

under consideration before the CASTF. The following comments are submitted on behalf of the

American Property Casualty Insurance Association (APCIA)1.

The APCIA greatly appreciates the work of the CASTF on the exposure draft. The following remarks are

presented with intended support for regulatory guidance in the form of best practices to benefit

regulators and insurers in the review of predictive models and analytics utilized by the insurance

company to justify a filed rating plan for the private passenger auto or homeowners insurance market.

The APCIA believes that best practices could foster comprehensive upfront dialogue between the filing

company and regulator that supports an efficient and effective review appropriately focused on

ensuring compliance with the applicable statutory or regulatory rating standards. However, APCIA

strongly believes the creation of best practices should not be an initiative to create new rating standards

that extend the statutory scope of the rate review process.

The CASTF had previously surveyed state insurance departments for documented practices, guidelines,

and checklists developed to aid state insurance department staff that review and approve rate filings

that may rely in some part on a generalized linear or other type of predictive model. Further, the

nationally recognized Actuarial Standards Board has developed professional standards of practice

1 Representing nearly 60 percent of the U.S. property casualty insurance market, the American Property Casualty Insurance Association (APCIA) promotes and protects the viability of private competition for the benefit of consumers and insurers. APCIA represents the broadest cross-section of home, auto, and business insurers of any national trade association. APCIA members represent all sizes, structures, and regions, which protect families, communities, and businesses in the U.S. and across the globe. APCIA is the result of the merger between the American Insurance Association (AIA) and the Property Casualty Insurers Association of America (PCIAA).


4/6/19


of 29

related to use of modeling in insurance applications that were established through a deliberative and

transparent process.

This current “best practices” draft includes numerous “essential” responses to questions that extend

well beyond what any insurance department currently requires as well as the best practices set forth in

established actuarial professional standards. The APCIA urges the CASTF to revert to identifying the

existing best practices currently employed by insurance regulators and to leverage the well-vetted best

practices established by the Actuarial Standards Board as the foundation for this effort. APCIA believes it

would be detrimental to the shared goals of regulators and industry to establish essential practices that

insurance departments in their current capacity have not and potentially cannot employ.

It is imperative that any best practice not create a one-size-fits-all prescriptive checklist that may unduly

restrict the use of advanced mathematical and actuarial techniques and the risk rating factors necessary

to most appropriately price companies’ insurance business. As the introduction section of the draft

reflects, insurers’ use of predictive analytics along with big data has significant potential benefits to both

consumers and insurers by transforming the insurer-consumer experience into a more meaningful

relationship. However, predictive analytics techniques are evolving rapidly, and their application is not

yet an industry norm. The extent to which insurers leverage predictive analytics in their insurance

application can be expected to vary for each company based on their own analysis and outlook of their

applicable book of business.

To establish true best practices, the APCIA recognizes the regulator’s essential role and duty over the

rating plans in their state in accordance with their specific statutory authority. Best practices can help

the regulator and insurance company establish a base understanding of the essential elements of a

model that may influence the regulatory review as to whether modeled rates are appropriately justified.

The expectation is that best practices will aid speed to market and competitiveness of the state

marketplace. With the adoption of such guiding best practices the state regulator and filing company

will be better able to identify the resources and areas of focus needed to assist in the review of

predictive models. To that end, the list of essential elements must be limited to those truly needed and

be deliberatively focused on areas of priority for what will amount to one component of the supporting

methodology for the company’s rating plan. So, while there will be cases where more information is

required, the table attached to this letter seeks to provide the task force with suggestions to cull and


4/6/19


of 29

streamline the proposed ninety-two elements of regulatory review. APCIA believes the current number

of elements would add significant time-consuming demands on the company and regulator without

further benefit and give cause to dissuade the use of advanced analytics and the advancement of

product innovation.

In addition, the APCIA would urge the CASTF to initiate additional discussions amongst the member

states to garner further understanding of how generalized linear models have been reviewed and the

specific filing requirements for insurers that employ them. Various states already obtain significant

information on model input data, the structure of the model, and the model outputs, which becomes

the basis for a thorough discussion of the underlying assumptions and modeling methodology; the

reasons for the approaches selected and the mathematical formulas used; and, comparison and contrast

of indicated relativities and confidence intervals.

The APCIA lastly recommends that the “best practices” not create a new intuitive rating standard for the

relationship between selected risk factors and expected loss or expense. We urge the CASTF to adhere

to Actuarial Standard of Practice No.12 Risk Classification (for all Practice Areas) Section 3.2.2 states:

“Causality—While the actuary should select risk characteristics that are related to expected outcomes, it

is not necessary for the actuary to establish a cause and effect relationship between the risk

characteristic and expected outcome in order to use a specific risk characteristic.”

The APCIA submits these comments with the intent to support adopted best practices that attain the

goals of product innovation, education, efficiency, and compliance. We encourage the CASTF to consider

that regulator requests for raw data and details on the development of model algorithms could stifle

innovation and speed to market efforts and jeopardize the proprietary nature of the data and privacy

concerns associated with sharing it. Requiring this level of detail from companies, as well as requiring

regulators to find time to review this level of detail within every filing, can severely slow down the

approval timeline. This can unduly create more burden on companies with greater levels of

segmentation and innovation in their products. In the very competitive marketplace that exists for

personal lines insurance, we believe consumers would be best served through more effective discussion

of the essential qualitative information about the rating variables, combined with more general support

for the predictive nature of the variables used.


4/6/19


of 29

The following table includes the comments of the APCIA related to the specific 92 elements of the

proposed regulatory review of predictive models. Input is additionally provided on other sections of the

exposed draft document.

Thank you again for the opportunity to comment. We look forward to working with the Task Force to

achieve a solution that benefits regulators, insurers and ultimately our consumers.

Sincerely,

David Kodama Lisa Brown

Assistant Vice President, APCIA Assistant General Counsel & Director, APCIA


4/6/19


o

f 29

A. S

elec

ting

Mod

el In

put

Info

rmat

ion

Impo

rtan

ce to

R

egul

ator

’s R

evie

w

"Ess

entia

l" o

r "M

ay B

e R

eque

sted

"

Com

men

tsA

PCIA

1.

Avai

labl

e D

ata

Sour

ces

A.1

.a

Prov

ide

deta

ils o

f all

data

sour

ces i

nclu

ding

th

e ex

perie

nce

perio

d fo

r in

sura

nce

data

and

whe

n th

e da

ta w

as la

st

reco

rded

or u

pdat

ed.

Esse

ntia

l

This

info

rmat

ion

can

be u

sed

to e

valu

ate

the

com

plet

enes

s of t

he d

ata

sour

ce, i

nteg

rity

of th

e da

ta so

urce

, rel

evan

ce o

f the

dat

a to

the

pred

ictiv

e tim

efra

me,

the

pote

ntia

l for

his

toric

al b

ias,

trans

pare

ncy

to

insu

red

of th

e da

ta so

urce

, and

the

abili

ty o

f the

insu

red

to m

ake

corr

ectio

ns to

the

data

sour

ce.

A.1

.bSp

ecify

the

com

pani

es

who

se d

ata

is in

clud

ed in

th

e da

tase

ts.

May

Be

Req

uest

ed

If th

e fil

er is

par

t of a

gro

up, d

o th

e da

tase

ts in

clud

e da

ta fr

om a

ffili

ated

co

mpa

nies

? If

so, w

hich

com

pani

es?

If th

e fil

er is

an

advi

sory

or

gani

zatio

n, w

hat c

ompa

nies

are

use

d? A

re th

e co

mpa

nies

incl

uded

in

the

data

rele

vant

and

com

patib

le to

the

com

pany

that

file

d th

era

ting

plan

?

Whe

n/w

hyw

ould

this

be

rele

vant

for p

urpo

ses o

f rev

iew

ing

how

the

mod

el is

use

d in

de

velo

pmen

t of t

he ra

ting

plan

?

A.1

.c

Prov

ide

the

geog

raph

ical

sc

ope

and

geog

raph

ic

expo

sure

dis

tribu

tion

of

the

data

.

Esse

ntia

lEv

alua

te w

heth

er th

e da

ta is

rele

vant

to th

e lo

ss p

oten

tial f

or w

hich

it is

be

ing

used

. For

exa

mpl

e, v

erify

that

hur

rican

e da

ta is

onl

y us

ed w

here

hu

rric

anes

can

occ

ur.

Que

stio

n sh

ould

focu

s atte

ntio

n on

just

ifica

tion

that

the

data

use

d is

app

ropr

iate

for

the

mod

eled

resp

onse

A.1

.d

List

eac

h da

ta so

urce

. Fo

r eac

h so

urce

, lis

t all

data

ele

men

ts u

sed

as

inpu

t to

the

mod

el th

at

cam

e fr

om th

at so

urce

.

Esse

ntia

lC

omm

enta

ry sh

ould

cla

rify

that

the

scop

e is

rega

rdin

gan

yva

riabl

e th

at e

nded

up

in

the

final

mod

el(s

ame

stan

dard

hel

d fo

r any

oth

er a

ctua

rial t

echn

ique

/met

hodo

logy

);cl

arify

the

“sou

rce”

is re

lativ

e to

the

insu

ranc

e co

mpa

ny.

A.1

.e

Spec

ify th

e ty

pe o

f dat

a (e

.g.,

acci

dent

yea

r or

polic

y ye

ar, t

ext,

num

eric

).

Esse

ntia

lC

ombi

ne u

nder

A.1

.a –

basi

c la

yout

, for

m, a

nd so

urce

of t

he d

ata

utili

zed

(sam

e st

anda

rd h

eld

for a

ny o

ther

act

uaria

l tec

hniq

ue/m

etho

dolo

gy)


4/6/19


o

f 29

A.1

.f

Expl

ain

if in

tern

al o

r ex

tern

al d

ata

was

use

d an

d if

exte

rnal

dat

a w

as

used

, dis

clos

e re

lianc

e on

dat

a su

pplie

d by

ot

hers

.

Esse

ntia

lIn

clud

e un

der A

.1.a

A.1

.g

Prov

ide

deta

ils o

f any

no

n-in

sura

nce

data

use

d (c

usto

mer

-pro

vide

d or

ot

her)

, inc

ludi

ng w

ho

owns

this

dat

a, h

ow

cons

umer

s can

ver

ify

thei

r dat

a an

d co

rrec

t er

rors

, whe

ther

the

data

w

as c

olle

cted

by

use

of a

qu

estio

nnai

re/c

heck

list,

whe

ther

it w

as

volu

ntar

ily re

porte

d by

th

e ap

plic

ant,

and

whe

ther

any

of t

he

varia

bles

are

subj

ect t

o th

e Fa

ir C

redi

t Rep

ortin

g A

ct. I

f the

dat

a is

from

an

out

side

sour

ce, w

hat

step

s wer

e ta

ken

to

verif

y th

e da

ta w

as

accu

rate

?

Esse

ntia

lIf

the

data

is fr

om a

third

-par

ty so

urce

, the

com

pany

shou

ld p

rovi

de

info

rmat

ion

on h

ow th

e so

urce

add

ress

es th

e qu

estio

ns in

this

co

nsid

erat

ion.

The

insu

ranc

e co

mpa

ny m

ay n

ot b

e ab

le to

dete

rmin

e ho

w c

onsu

mer

s ver

ify a

nd

corr

ect 3

rdpa

rty d

ata.

Thi

shas

nev

er b

een

a pr

ereq

uisi

te fo

r rat

e ap

prov

al.F

or

exam

ple,

insu

rers

are

not

requ

ired

to d

etai

l how

vehi

cle

owne

rs c

an a

nd d

o ve

rify

and

corr

ect e

rror

s in

thei

r MV

Rs.

Prov

idin

g cu

stom

ers a

n ab

ility

to c

halle

nge

and/

or

corr

ect d

ata

inpu

ts m

ay p

rove

diff

icul

t if t

he in

sure

r doe

sn’t

own

the

data

. Im

posi

ng

this

as a

n “e

ssen

tial”

elem

ent o

f the

revi

ew m

ay re

sult

in a

disi

ncen

tive

to u

se o

r eve

n co

nsid

er n

ew so

urce

s of i

nfor

mat

ion,

and

ther

efor

e, a

dis

ince

ntiv

e to

inno

vate

.

2.

Sub-

Mod

els

A.2

.a

Dis

clos

e re

lianc

e on

sub-

mod

el o

utpu

t use

d as

inpu

t to

this

mod

el. I

f a su

b-m

odel

was

relie

d up

on,

prov

ide

the

vend

or n

ame,

an

d th

e na

me

and

vers

ion

of

the

sub-

mod

el. I

f the

sub-

mod

el w

as b

uilt/

crea

ted

in-

hous

e, p

rovi

de c

onta

ct

info

rmat

ion

for t

he p

erso

n re

spon

sibl

e fo

r the

sub-

mod

el.

Esse

ntia

l

Exam

ples

of s

uch

sub-

mod

els i

nclu

de c

redi

t/fin

anci

al sc

orin

g al

gorit

hms a

nd h

ouse

hold

com

posi

te sc

ore

mod

els.

Sub-

mod

els c

an

be e

valu

ated

sepa

rate

ly a

nd in

the

sam

e m

anne

r as t

he p

rimar

y m

odel

un

der e

valu

atio

n.

The

“ess

entia

l” a

ttent

ion

shou

ld fo

cust

he fi

ling

com

pany

todi

sclo

se re

lianc

e an

d ex

plan

atio

n of

the

rele

vant

data

sour

cesa

nd/o

r mod

el/s

ub-m

odel

s,an

d ho

w th

ey a

reus

ed to

supp

ort t

he se

lect

ed ra

te fi

ling.

Whi

le w

ho b

uilt

the

sub-

mod

el a

nd h

ow it

was

built

may

be

of in

tere

st, i

t sho

uld

not b

e th

e “e

ssen

tial”

focu

s ove

r the

insu

ranc

e ap

plic

atio

n of

the

sub-

mod

el–

this

shou

ld b

e th

e st

anda

rd n

o di

ffere

nt fo

r any

oth

er

actu

aria

l or s

tatis

tical

tool

use

d in

the

rate

mak

ing.


4/6/19


o

f 29

A.2

.b

If u

sing

cat

astro

phe

mod

el

outp

ut, i

dent

ify th

e ve

ndor

an

d th

e m

odel

se

tting

s/as

sum

ptio

ns u

sed

whe

n th

e m

odel

was

run.

Esse

ntia

lFo

r exa

mpl

e, it

is im

porta

nt to

kno

w h

urric

ane

mod

el se

tting

s for

st

orm

surg

e, d

eman

d su

rge,

long

/sho

rt-te

rm v

iew

s.

A.2

.c

If u

sing

cat

astro

phe

mod

el

outp

ut (a

sub-

mod

el) a

s in

put t

o th

e G

LM u

nder

re

view

, dis

clos

e w

heth

er

loss

ass

ocia

ted

with

the

mod

eled

out

put w

as

rem

oved

from

the

loss

ex

perie

nce

data

sets

.

Esse

ntia

l

If a

wea

ther

-bas

ed su

b-m

odel

is in

put t

o th

e G

LM u

nder

revi

ew, l

oss

data

use

d to

dev

elop

the

mod

el sh

ould

not

incl

ude

loss

exp

erie

nce

asso

ciat

ed w

ith th

e w

eath

er-b

ased

sub-

mod

el. D

oing

so c

ould

cau

se

dist

ortio

ns in

the

mod

eled

resu

lts b

y do

uble

cou

ntin

g su

ch lo

sses

w

hen

dete

rmin

ing

rela

tiviti

es o

r los

s loa

ds in

the

filed

ratin

g pl

an.

For e

xam

ple,

redu

ndan

t los

ses i

n th

e da

ta m

ay o

ccur

whe

n no

n-hu

rric

ane

win

d lo

sses

are

incl

uded

in th

e da

ta w

hile

als

o us

ing

a se

vere

con

vect

ive

stor

m m

odel

in th

e ac

tuar

ial i

ndic

atio

n. S

uch

redu

ndan

cy m

ay a

lso

occu

r with

the

incl

usio

n of

fluv

ial o

r plu

vial

flo

od lo

sses

whe

n us

ing

a flo

od m

odel

, inc

lusi

on o

f fre

eze

loss

es

whe

n us

ing

a w

inte

r sto

rm m

odel

or i

nclu

ding

dem

and

surg

e ca

used

by

any

cat

astro

phic

eve

nt.

A.2

.d

If u

sing

out

put o

f any

sc

orin

g al

gorit

hms,

prov

ide

a lis

t of t

he v

aria

bles

use

d to

de

term

ine

the

scor

e an

d pr

ovid

e th

e so

urce

of t

he

data

use

d to

cal

cula

te th

e sc

ore.

Esse

ntia

lA

ny su

b-m

odel

shou

ld b

e re

view

ed in

the

sam

e m

anne

r as t

he

prim

ary

mod

el th

at u

ses t

he su

b-m

odel

’s o

utpu

t as i

nput

.

A.2

.e

Was

the

sub-

mod

el

prev

ious

ly a

ppro

ved

(or

acce

pted

) by

the

regu

lato

ry

agen

cy?

Esse

ntia

lIf

the

sub-

mod

el w

as p

revi

ousl

y ap

prov

ed, t

hat m

ay c

hang

e th

e ex

tent

of t

he su

b-m

odel

’s re

view

.


4/6/19


o

f 29

3.

Adju

stm

ents

and

Scr

ubbi

ng

A.3

.aPr

ovid

e pr

e-sc

rubb

ed d

ata

dist

ribut

ions

for e

ach

inpu

t.M

ay B

e R

eque

sted

Com

pare

thes

e di

strib

utio

ns to

A.3

.g

The

sugg

estio

n th

at th

is is

“es

sent

ial”

to th

e re

gula

tor’

s rev

iew

of a

rate

filin

g ra

ises

co

ncer

n. T

his w

illlik

ely

be a

sign

ifica

nt a

dmin

istra

tivel

y bu

rden

som

e in

quiry

that

w

ill re

quire

a h

igh

leve

l of e

xper

tise,

tim

e an

d ot

her r

esou

rces

for t

he re

gula

tort

o m

ake

use

of. I

twou

ld b

e m

ore

appr

opria

te to

ask

wha

t var

iabl

es h

ad m

eani

ngfu

l ad

just

men

ts a

nd th

en d

escr

ibe

thos

e ad

just

men

ts.A

dditi

onal

ly, t

hist

ype

of

info

rmat

ion

coul

dbe

con

side

red

high

ly c

onfid

entia

l and

shou

ld o

nly

be re

ques

ted

ifit

can

be p

rovi

ded

in a

pro

prie

tary

and

con

fiden

tial m

anne

r. Th

e “e

ssen

tial”

det

ail o

f the

da

tadi

strib

utio

ns fo

r the

rele

vant

inpu

ts a

nd o

utpu

ts c

an b

e pr

ovid

ed u

nder

A.1

.

A.3

.bH

ow w

as m

issi

ng d

ata

hand

led?

Esse

ntia

lC

ombi

ne A

.3.b

. –d.

Dat

a qu

ality

“sc

rubb

ing”

is a

n “e

ssen

tial”

cons

ider

atio

n fo

r any

da

tase

t. Is

the

stan

dard

the

sam

e fo

r mod

elle

d vs

non

-mod

elle

d da

ta?

A.3

.cIf

dup

licat

e re

cord

s exi

st,

how

wer

e th

ey h

andl

ed?

Esse

ntia

lSe

e co

mm

ent u

nder

A.3

.b.

A.3

.d

Wer

e an

y da

ta o

utlie

rs

iden

tifie

d an

d su

bseq

uent

ly

adju

sted

? N

ame

the

outli

ers

and

expl

ain

the

adju

stm

ents

m

ade

to th

ese

outli

ers.

Esse

ntia

lSe

e co

mm

ent u

nder

A.3

.b

A.3

.e

Wer

e pr

emiu

m, e

xpos

ure,

lo

ss o

r exp

ense

dat

a ad

just

ed (e

.g.,

deve

lope

d,

trend

ed, a

djus

ted

for

cata

stro

phe

expe

rienc

e or

ca

pped

) and

, if s

o, h

ow?

Do

the

adju

stm

ents

var

y fo

r di

ffere

nt se

gmen

ts o

f the

da

ta a

nd, i

f so,

wha

t are

the

segm

ents

and

how

was

the

data

adj

uste

d?

Esse

ntia

l

Look

for a

nom

alie

s in

the

data

that

shou

ld b

e ad

dres

sed.

For

ex

ampl

e, is

ther

e an

ext

rem

e lo

ss e

vent

in th

e da

ta?

If o

ther

pro

cess

es

wer

e us

ed to

load

rate

s for

spec

ific

loss

eve

nts,

how

is th

e im

pact

of

thos

e lo

sses

shou

ld b

e re

mov

ed c

onsi

dere

d?fr

om th

e in

put d

ata,

e.g

., la

rge

loss

es, f

lood

, hur

rican

e or

seve

re c

onve

ctiv

e st

orm

mod

els f

or

PPA

com

preh

ensi

ve o

r hom

eow

ners

’ los

s.

A.3

.f

Wha

t adj

ustm

ents

wer

e m

ade

to ra

w d

ata,

e.g

., tra

nsfo

rmat

ions

, bin

ning

an

d/or

cat

egor

izat

ions

? If

so

, nam

e th

e ch

arac

teris

tic/v

aria

ble

and

desc

ribe

the

adju

stm

ent.

Esse

ntia

lSe

e co

mm

ent u

nder

A.3

.a.C

ombi

ne w

ith C

.5.a


4/6/19


o

f 29

A.3

.gPr

ovid

e po

st-s

crub

bed

data

di

strib

utio

ns fo

r eac

h in

put.

May

Be

Req

uest

edC

ompa

re th

ese

dist

ribut

ions

to A

.3.a

See

com

men

t und

er A

.3.a

.

4.

Dat

a O

rgan

izat

ion

–RE

NAM

E D

ata

Revi

ew a

nd R

econ

cilia

tion

A.4

.a

Doc

umen

t the

met

hod

of

orga

niza

tion

for c

ompi

ling

data

, inc

ludi

ng p

roce

dure

s to

mer

ge d

ata

from

diff

eren

t so

urce

s and

a de

scrip

tion

of

any

prel

imin

ary

anal

yses

,da

ta c

heck

s, an

d lo

gica

l te

sts p

erfo

rmed

on

the

data

an

d th

e re

sults

of t

hose

test

s.

May

Be

Req

uest

edEs

sent

ial

This

shou

ld e

xpla

in h

ow d

ata

from

sepa

rate

sour

ces w

as m

erge

d.

The

“ess

entia

l” a

ttent

ion

to d

ata

orga

niza

tion

and

prep

arat

ion

can

dem

and

sign

ifica

nt

time

and

reso

urce

s with

out a

ttent

ion

to h

ow th

is su

ppor

ts th

e ra

te re

view

pro

cess

.B

est P

ract

ices

shou

ld ra

ther

focu

s on

how

the

data

supp

orts

the

ratin

g va

riabl

es in

the

filed

insu

ranc

e ap

plic

atio

n.

A.4

.b

Doc

umen

t the

pro

cess

for

revi

ewin

g th

e ap

prop

riate

ness

, re

ason

able

ness

, con

sist

ency

an

d co

mpr

ehen

sive

ness

of

the

data

, inc

ludi

ng a

ju

stifi

catio

n of

why

the

data

m

akes

sens

e.

Esse

ntia

l

For e

xam

ple,

if b

y-pe

ril m

odel

ing

is p

erfo

rmed

, the

doc

umen

tatio

n sh

ould

be

for e

ach

peril

and

mak

e in

tuiti

ve se

nse.

For

exa

mpl

e, if

“m

urde

r” o

r “th

eft”

rate

s are

use

d to

pre

dict

the

win

d pe

ril, p

rovi

de

supp

orta

nd a

logi

cal e

xpla

natio

n.

Cla

rific

atio

n ne

eded

–It

is n

ot u

nder

stoo

d w

hat i

s mea

nt b

y th

e es

sent

ial n

eed

for

“jus

tific

atio

n of

why

the

data

mak

e se

nse”

rela

tive

to th

e ov

erar

chin

g go

al to

just

ify

the

rate

filin

g.Th

e re

fere

nce

to a

n “i

ntui

tive

sens

e” fo

r any

by-

peril

mod

elin

g is

not

de

fined

and

unc

lear

.Reg

ulat

ory

requ

ests

for i

ntui

tive

argu

men

ts a

nd e

xpla

natio

nsof

in

tuiti

ve re

latio

nshi

ps a

re in

cons

iste

nt w

ith A

SOP

12 R

isk

Cla

ssifi

catio

n-3

.2.2

.and

sh

ould

not

be

incl

uded

as a

n es

sent

ial b

est p

ract

ice

for r

egul

ator

y re

view

.

A.4

.c

Dis

clos

e m

ater

ial f

indi

ngs

from

the

data

revi

ew a

nd

iden

tify

any

pote

ntia

l m

ater

ial l

imita

tions

, def

ects

, bi

as o

r unr

esol

ved

conc

erns

fo

und

or b

elie

ved

to e

xist

in

the

data

.

Esse

ntia

lC

ombi

ne A

.4.c

and

A.4

.d w

ith “

esse

ntia

l” fo

cus o

n ho

w th

e co

mpa

ny a

ddre

sses

any

m

ater

ial l

imita

tions

, err

ors,

bias

es in

the

mod

elin

g.

A.4

.d

For a

ny e

rror

s or m

ater

ial

limita

tions

in th

e da

ta,

expl

ain

how

they

wer

e co

rrec

ted.

Esse

ntia

l


4/6/19


of 2

9 5.

Fi

nal D

ata

Info

rmat

ion

A.5

.a

If th

e ra

w d

ata

sele

cted

to

build

the

mod

el is

in a

fo

rmat

that

can

be

mad

e av

aila

ble

to th

e re

gula

tor,

prov

ide

it.

May

Be

Req

uest

ed

Mul

tiple

con

cern

s tha

t “be

st p

ract

ices

” do

not

add

ress

the

expe

rtise

and

reso

urce

s re

quire

d to

revi

ew, a

sses

s and

util

ize

“raw

dat

a” fo

r pur

pose

of r

ate

revi

ew. P

rovi

ding

ra

w d

ata

will

rais

eco

ncer

n ab

out d

ata

secu

rity,

reco

rd re

tent

ion,

con

fiden

tialit

y an

d pr

oprie

tary

pro

tect

ions

.

B. B

uild

ing

the

Mod

el

Info

rmat

ion

Impo

rtan

ce to

Reg

ulat

or’s

R

evie

w "

Esse

ntia

l" o

r "M

ay

Be

Req

uest

ed"

Com

men

tsA

PCIA

1.

Hig

h-Le

vel N

arra

tive

for B

uild

ing

the

Mod

el

B.1

.a

Iden

tify

the

type

of m

odel

(e

.g. G

ener

aliz

ed L

inea

r M

odel

–G

LM, d

ecis

ion

tree,

Bay

esia

n G

ener

aliz

ed

Line

ar M

odel

, Gra

dien

t-B

oost

ing

Mac

hine

, neu

ral

netw

ork,

etc

.), d

escr

ibe

its

role

in th

e ra

ting

syst

em a

nd

prov

ide

the

reas

ons w

hy

that

type

of m

odel

is a

n ap

prop

riate

cho

ice

for t

hat

role

.

Esse

ntia

lIf

by-

peril

or b

y-co

vera

ge m

odel

ing

is u

sed,

the

expl

anat

ion

shou

ld b

e by

-per

il/co

vera

ge.

For S

ectio

n B

–C

onsi

dert

he w

orki

ng d

raft

of th

e A

ctua

rial S

tand

ards

Boa

rd o

n M

odel

ing

sect

ions

3.5

.1 M

itiga

tion

of M

odel

Ris

k -V

alid

atio

nan

d 3.

6 Pr

esen

tatio

n of

Res

ults

, inc

ludi

ng E

xpla

natio

n of

Lim

itatio

ns o

f Mod

els;

Dis

cuss

ion

of M

odel

s;C

ompa

rison

to P

rior R

epor

ts; a

nd D

escr

iptio

n of

Con

serv

atis

m o

r Opt

imis

m


4/6/19


of 2

9 B

.1.b

A d

escr

iptio

n of

why

the

mod

el (u

sing

the

varia

bles

in

clud

ed in

it) i

s app

ropr

iate

fo

r the

line

of b

usin

ess.

Esse

ntia

lIf

by-

peril

, by-

form

or b

y-co

vera

ge m

odel

ing

is u

sed,

the

expl

anat

ion

shou

ld b

e by

-per

il/co

vera

ge/fo

rm.

Com

bine

with

B.1

.a

B.1

.c

Des

crib

e th

e m

odel

revi

ew

proc

ess,

from

initi

al c

once

pt

to fi

nal m

odel

. Kee

p th

is in

ov

ervi

ew n

arra

tive

mod

e,

less

than

3 p

ages

.

Esse

ntia

lMay

Be

Req

uest

ed

Prov

ide

as p

art o

f B.1

.a. a

nd B

.1.b

.Thi

s sho

uld

be th

e ov

er-a

rchi

ng “

esse

ntia

l”

cons

ider

atio

n fo

r B.1

, for

whi

ch th

e na

rrat

ive

shou

ld c

aptu

re a

ll th

ese

sub-

elem

ents

.A

ttent

ion

shou

ld fo

cus n

ot o

n co

ncep

t dev

elop

men

t but

impl

emen

tatio

n an

d in

corp

orat

ion

of th

e fin

al m

odel

into

the

ratin

g pl

an.

B.1

.d

Des

crib

e w

heth

er lo

ss ra

tio,

pure

pre

miu

m o

r fr

eque

ncy/

seve

rity

anal

yses

w

as p

erfo

rmed

and

, if

sepa

rate

freq

uenc

y/se

verit

y m

odel

ing

was

per

form

ed,

how

pur

e pr

emiu

ms

wer

e de

term

ined

.

Esse

ntia

l

B.1

.eW

hat i

s the

mod

el’s

targ

et

varia

ble?

Esse

ntia

lA

cle

ar d

escr

iptio

n of

the

targ

et v

aria

ble

is k

ey to

un

ders

tand

ing

the

purp

ose

of th

e m

odel

.

B.1

.fPr

ovid

e a

det

aile

d

desc

riptio

n o

f th

e va

riabl

e se

lect

ion

proc

ess.

Esse

ntia

l

B.1

.g

Was

inpu

t dat

a se

gmen

ted

in a

ny w

ay, e

.g.,

was

m

odel

ing

perf

orm

ed o

n a

by-c

over

age

or b

y-pe

ril

basi

s or b

y-fo

rm?

Expl

ain

the

form

of d

ata

segm

enta

tion

and

the

reas

ons f

or d

ata

segm

enta

tion.

Esse

ntia

lTh

e re

gula

tor w

ould

use

this

to fo

llow

the

logi

c of

the

mod

elin

g pr

oces

s.

B.1

.h

Des

crib

e an

y lim

itatio

ns o

r co

ncer

ns in

the

anal

ysis

re

sulti

ng fr

om d

ata

issu

es

and

disc

uss t

he re

sulti

ng

impa

ct o

n th

e m

odel

ing

resu

lts.

Esse

ntia

lTh

is c

anbe

cap

ture

d in

the

info

rmat

ion

prov

ided

und

erA

.4.c


4/6/19


of 2

9 B

.1.i

Expl

ain

Hho

w d

ata

cred

ibili

ty (o

r lac

k th

ereo

f)

was

acc

ount

ed fo

r in

the

mod

el b

uild

ing?

Esse

ntia

l

Adj

ustm

ents

may

be

need

ed g

iven

mod

els d

o no

t exp

licitl

y co

nsid

er th

e cr

edib

ility

of t

he in

put d

ata

or th

e m

odel

’s

resu

lting

out

put;

mod

els t

ake

inpu

t dat

a at

face

val

ue a

nd

assu

me

100%

cre

dibi

lity

whe

n pr

oduc

ing

mod

eled

out

put.

Re-

stat

e: ‘E

xpla

in th

e cr

edib

ility

adj

ustm

ents

mad

e af

ter m

odel

ing’

2.

Med

ium

-Lev

el N

arra

tive

for B

uild

ing

the

Mod

el

B.2

.a

Des

crib

e an

y ju

dgm

ent u

sed

thro

ugho

ut th

e m

odel

ing

proc

ess.

Dis

clos

e as

sum

ptio

ns u

sed

in

cons

truct

ing

the

mod

el a

nd

prov

ide

supp

ort f

or th

ese

assu

mpt

ions

.

Esse

ntia

l

The

“des

crib

e an

y ju

dgm

ent”

is c

over

ed a

cros

s all

the

actu

aria

l sup

port

filed

, of

whi

ch th

e m

odel

ing

is b

ut a

par

t of t

he a

ctua

rial j

udge

men

t bei

ng a

sses

sed

in th

e ra

te

revi

ew.

It w

ould

be

nons

ensi

cal t

o ex

tract

a d

escr

iptio

n of

all

judg

emen

tsus

edou

t of

thei

r res

pect

ive

cont

ext.

Ref

er to

act

uaria

l sta

ndar

ds o

f pra

ctic

e on

dis

clos

ures

and

use

of

pro

fess

iona

l jud

gmen

t in

use

of m

odel

s

B.2

.b

If p

ost-m

odel

adj

ustm

ents

w

ere

mad

e to

the

data

and

th

e m

odel

was

reru

n,

expl

ain

the

deta

ils a

nd th

e ra

tiona

le. I

t is n

ot n

eces

sary

to

dis

cuss

eac

h ite

ratio

n of

ad

ding

and

subt

ract

ing

varia

bles

, but

the

regu

lato

r sh

ould

be

prov

ided

with

a

gene

ral d

escr

iptio

n of

how

th

at w

as d

one,

incl

udin

g an

y m

easu

res r

elie

d up

on.

Esse

ntia

lEv

alua

te th

e ad

ditio

n or

rem

oval

of v

aria

bles

and

the

mod

el

fittin

g.

B.2

.b –

B.2

.c: C

allin

g fo

r the

“es

sent

ial”

cap

ture

of d

etai

ls o

n th

e ite

rativ

e m

odel

runs

, tes

ts a

nd a

djus

tmen

ts c

an d

eman

d si

gnifi

cant

tim

e an

d re

sour

ce c

omm

itmen

t for

th

e co

mpa

ny a

nd re

gula

tor.

The

“ess

entia

l” p

riorit

y sh

ould

be

on th

e ke

y da

ta

adju

stm

ents

, var

iabl

ew

eigh

ting,

and

the

stat

istic

al (v

alid

atio

n, g

oodn

ess o

f fit,

R-

squa

re m

easu

res)

testi

ng.

B.2

.c

Des

crib

e th

e un

ivar

iate

te

stin

g an

d ba

lanc

ing

that

w

as p

erfo

rmed

dur

ing

the

mod

el-b

uild

ing

proc

ess,

in

clud

ing

a ve

rbal

su

mm

arya

n ex

plan

atio

nof

th

e th

ough

t pro

cess

es

invo

lved

.

Esse

ntia

lFu

rther

ela

bora

tion

from

B.2

.b.

See

com

men

t und

er B

.2.b


4/6/19


of 2

9 B

.2.d

Des

crib

e th

e 2-

way

test

ing

and

bala

ncin

g th

at w

as

perf

orm

ed d

urin

g th

e m

odel

-bui

ldin

g pr

oces

s,

incl

udin

g a

verb

al

sum

mar

yexp

lana

tion

of th

e th

ough

t pro

cess

es o

f in

clud

ing

(or n

ot in

clud

ing)

in

tera

ctio

n te

rms.

Esse

ntia

lFu

rther

ela

bora

tion

from

B.2

.a a

nd B

.2.b

.

B.2

.e

For t

he G

LM, w

hat w

as th

e lin

k fu

nctio

n us

ed?

Wha

t di

strib

utio

n w

as u

sed

for t

he

mod

el (e

.g.,

Pois

son,

G

auss

ian,

log-

norm

al,

Twee

die)

? Ex

plai

n w

hy th

e lin

k fu

nctio

n di

strib

utio

n w

as c

hose

n. P

rovi

de th

e fo

rmul

as fo

r the

dis

tribu

tion

and

link

func

tions

, in

clud

ing

spec

ific

num

eric

al

para

met

ers o

f the

di

strib

utio

n.

Esse

ntia

l

B.2

.fW

ere

ther

e da

ta si

tuat

ions

G

LM w

eigh

ts w

ere

used

? D

escr

ibe

thes

e.M

ay B

e R

eque

sted

Inve

stig

ate

whe

ther

id

entic

al

reco

rds

wer

e co

mbi

ned

to

build

the

mod

el.

3.

Pred

icto

r Var

iabl

es

B.3

.a

Prov

ide

the

nam

es,

desc

riptio

ns a

nd u

ses o

f ea

ch p

redi

ctor

var

iabl

e,

offs

et v

aria

ble,

con

trol

varia

ble,

pro

xy v

aria

ble,

ge

ogra

phic

var

iabl

e,

geod

emog

raph

ic v

aria

ble

and

all o

ther

var

iabl

es in

th

e m

odel

; exp

lana

tions

sh

ould

not

use

pr

ogra

mm

ing

lang

uage

or

cod

e.

Esse

ntia

l


4/6/19


of 2

9 B

.3.b

For e

ach

pred

icto

r va

riabl

e, st

ate

whe

ther

th

e va

riabl

e is

co

ntin

uous

, dis

cret

e or

B

oole

an.

Esse

ntia

lTh

is sh

ould

be

prov

ided

unde

r the

gen

eral

des

crip

tion

of th

e va

riabl

e da

ta.

B.3

.c

Prov

ide

an in

tuiti

ve

argu

men

tsup

port

for

why

how

an in

crea

se in

ea

ch p

redi

ctor

var

iabl

e sh

ould

may

incr

ease

or

decr

ease

freq

uenc

y,

seve

rity,

loss

cos

ts,

expe

nses

, or w

hate

ver i

s be

ing

pred

icte

d.

Esse

ntia

l

See

A.4

.b. R

equi

ring

an “

esse

ntia

l” in

tuiti

ve a

rgum

ent f

or a

pre

dict

or v

aria

ble

is n

ot

unde

rsto

od re

lativ

e to

act

uaria

l sta

ndar

ds o

f pra

ctic

e. R

efer

to A

SOP

12 R

isk

Cla

ssifi

catio

n-3

.2.2

.The

re n

eeds

to b

e fu

rther

com

men

tary

/gui

danc

e or

del

ete

the

“int

uitiv

e” st

anda

rd to

be

a m

ore

gene

ral d

escr

iptio

n, e

xpla

natio

n an

d su

ppor

t for

the

impa

ct e

ach

pred

icto

r var

iabl

e ha

s on

the

mod

elle

d ou

tput

.

B.3

.d

If th

e m

odel

er u

sed

a Pr

inci

pal C

ompo

nent

A

naly

sis (

PCA

) ap

proa

ch, p

rovi

de a

na

rrat

ive

abou

t tha

t pr

oces

s, ex

plai

n w

hy

PCA

was

use

d, a

nd

desc

ribe

the

step

-by-

step

pr

oces

s use

d to

tra

nsfo

rm o

bser

vatio

ns

(usu

ally

cor

rela

ted)

into

a

set o

f lin

early

un

corr

elat

ed v

aria

bles

. In

clud

e a

listin

g of

the

PCA

var

iabl

e an

d its

pr

inci

pal c

ompo

nent

s.

Esse

ntia

lR

enam

e th

is a

s a m

ore

gene

ric d

imen

sion

ality

redu

ctio

n an

d th

en li

st P

CA

as o

ne

exam

ple.

4.

Mas

sagi

ng D

ata,

Mod

el V

alid

atio

n an

d G

oodn

ess-

of-F

it M

easu

res

B.4

.a

Prov

ide

a de

scrip

tion

of

how

the

avai

labl

e ra

w

data

was

div

ided

be

twee

n m

odel

de

velo

pmen

t, te

st a

nd

valid

atio

n da

tase

ts.

Des

crib

e al

l ci

rcum

stan

ces u

nder

w

hich

the

test

ing

and

Esse

ntia

l

For S

ub-s

ectio

n B

.4 -

Con

side

r ASO

P 38

3.5.

2 M

odel

Out

put—

In v

iew

of t

he

inte

nded

use

of t

he m

odel

, the

act

uary

shou

ldex

amin

e th

e m

odel

out

put f

or

reas

onab

lene

ss, c

onsi

derin

g fa

ctor

s suc

h as

the

follo

win

g:a.

the

resu

lts d

eriv

ed fr

om

alte

rnat

e m

odel

s or m

etho

ds, w

here

ava

ilabl

e an

dap

prop

riate

;b. h

ow h

isto

rical

ob

serv

atio

ns, i

f app

licab

le, c

ompa

re to

resu

lts p

rodu

ced

byth

e m

odel

;c. t

he

cons

iste

ncy

and

reas

onab

lene

ss o

f rel

atio

nshi

ps a

mon

g va

rious

out

putr

esul

ts; a

nd,d

. th

e se

nsiti

vity

of t

he m

odel

out

put t

o va

riatio

ns in

the

user

inpu

t and

mod

el

assu

mpt

ions

.


4/6/19


of 2

9

valid

atio

n da

tase

ts w

ere

acce

ssed

.

B.4

.b

Des

crib

e th

e m

etho

ds

used

to a

sses

s the

st

atis

tical

si

gnifi

canc

e/go

odne

ss o

f th

e fit

of t

he m

odel

, suc

h as

lift

char

ts a

nd

stat

istic

al te

sts.

Dis

clos

e w

heth

er th

e re

sults

are

ba

sed

on te

stin

g da

ta,

valid

atio

n da

ta a

nd

hold

out s

ampl

es. E

nsur

e th

at th

e as

sess

men

t in

clud

es m

odel

pr

ojec

tion

resu

lts

com

pare

d to

his

toric

al

actu

al re

sults

to v

erify

th

at m

odel

ed re

sults

bea

ra

reas

onab

le re

latio

nshi

p to

act

ual r

esul

ts. D

iscu

ss

the

resu

lts.

Esse

ntia

l

Som

e st

ates

requ

ire st

ate-

only

dat

a to

test

the

plan

, esp

ecia

lly

for a

naly

sis w

here

usi

ng th

e st

ate-

only

dat

a co

ntra

dict

s the

co

untry

wid

e re

sults

. Sta

te-o

nly

data

mig

ht b

e m

ore

appl

icab

le

but c

ould

als

o be

impa

cted

by

low

cre

dibi

lity

for s

ome

segm

ents

of ri

sk.

Com

men

tary

shou

ld c

autio

n th

e re

gula

tor t

hat s

tate

-onl

y da

ta m

ay n

ot b

e st

atis

tical

ly

cred

ible

or a

ppro

pria

te.

B.4

.c

Des

crib

e an

y ad

just

men

ts th

at w

ere

mad

e in

the

data

with

re

spec

t to

scal

ing

for

disc

rete

var

iabl

es o

r bi

nnin

g th

e da

ta.

Esse

ntia

lR

efer

toA

.3.f.

It is

mor

e ap

prop

riate

to se

eka

“des

crip

tion”

of h

ow v

aria

bles

wer

e sc

aled

, bin

ned,

or t

rans

form

ed a

s opp

osed

to v

ery

time-

inte

nsiv

e ex

hibi

ts o

f pre

/pos

t-sc

rub

dist

ribut

ions

.

B.4

.dD

escr

ibe

any

trans

form

atio

ns m

ade

for

cont

inuo

us v

aria

bles

.Es

sent

ial

Com

bine

with

B.4

.c.a

nd p

ossi

bly

A.3

.f.


4/6/19


of 2

9 B

.4.e

Esse

ntia

l

Typi

cal p

-val

ues g

reat

er th

an 5

% a

re la

rge

and

shou

ld b

e qu

estio

ned.

Rea

sona

ble

busi

ness

judg

men

t can

som

etim

es

prov

ide

legi

timat

e su

ppor

t for

hig

h p-

valu

es. R

easo

nabl

enes

s of

the

p-va

lue

thre

shol

d co

uld

also

var

y de

pend

ing

on th

e co

ntex

t of t

he m

odel

, e.g

., th

e th

resh

old

mig

ht b

e lo

wer

whe

n m

any

cand

idat

e va

riabl

es w

ere

eval

uate

d fo

r inc

lusi

on in

the

mod

el.

Com

men

tary

shou

ld c

larif

yth

at fo

r var

iabl

es th

at a

re m

odel

ed c

ontin

uous

ly o

nly

the

stat

istic

s aro

und

the

mod

eled

par

amet

ers w

ould

be

prov

ided

. e.g

., w

ould

not

exp

ect t

o pr

ovid

e co

nfid

ence

inte

rval

s aro

und

each

leve

l of a

n A

OI c

urve

. 4.e

-4.n

Com

men

tary

sh

ould

con

side

r tha

tpro

vidi

ng p-values

for e

very

var

iabl

e m

ayno

t be

nece

ssar

y, a

nd

that

pro

vidi

ng th

e ov

eral

l lift

and

acc

urac

yte

sts m

aybe

suff

icie

nt fo

r rev

iew

ing

pred

ictiv

e m

odel

s.

Reg

ulat

ors s

houl

d be

ale

rted

that

cer

tain

pre

dict

ive

mod

els m

ay n

ot g

ener

ate

p-va

lues

or

F te

sts.

B.4

.f

Iden

tify

the

thre

shol

d fo

r st

atis

tical

sign

ifica

nce

and

expl

ain

why

it w

as

sele

cted

. Pro

vide

ave

rbal

def

ense

an

expl

anat

ion

for k

eepi

ng

the

varia

ble

for e

ach

disc

rete

var

iabl

e le

vel

whe

re th

e p-

valu

es w

ere

not l

ess t

han

the

chos

en

thre

shol

d.

Esse

ntia

lSe

e C

omm

ent f

or B

.4.e

.Se

e co

mm

ent u

nder

B.4

.e.

B.4

.g

For o

vera

ll di

scre

te

varia

bles

, pro

vide

type

3

chi-s

quar

e te

sts,

p-va

lues

, F te

sts a

nd a

ny

othe

r rel

evan

t and

m

ater

ial t

est.

Wer

e m

odel

dev

elop

men

t dat

a,

valid

atio

n da

ta, t

est d

ata

or o

ther

dat

a us

ed fo

r th

ese

test

s?

Esse

ntia

lSe

e C

omm

ent f

or B

.4.e

.Se

e co

mm

ent u

nder

B.4

.e.

B.4

.h

For c

ontin

uous

var

iabl

es,

prov

ide

conf

iden

ce

inte

rval

s, ch

i-squ

are

test

s, p-

valu

es a

nd a

ny

othe

r rel

evan

t and

m

ater

ial t

est.

Wer

e m

odel

dev

elop

men

t dat

a,

valid

atio

n da

ta, t

est d

ata

or o

ther

dat

a us

ed fo

r th

ese

test

s?

Esse

ntia

lSe

e C

omm

ent f

or B

.4.e

.Se

e co

mm

ent u

nder

B.4

.e.


4/6/19


of 2

9 B

.4.i

Des

crib

e ho

w th

e m

odel

w

as te

sted

for s

tabi

lity

over

tim

e.Es

sent

ial

Eval

uate

the

build

/test

/val

idat

ion

data

sets

for p

oten

tial m

odel

di

stor

tions

(e.g

., a

win

ter s

torm

in y

ear 3

of 5

can

dis

tort

the

mod

el in

bot

h th

e te

stin

g an

d va

lidat

ion

data

sets

).

Furth

er c

omm

enta

ry is

nee

ded

to c

larif

yw

hat“

esse

ntia

l” su

ppor

t is n

eede

d to

de

mon

stra

te th

e te

stin

g of

“st

abili

tyov

er ti

me”

.

B.4

.j

Des

crib

e ho

w th

e m

odel

w

as te

sted

for

geog

raph

ic st

abili

ty, e

.g.,

acro

ss st

ates

or

terr

itorie

s with

in st

ate.

Esse

ntia

lEv

alua

te th

e ge

ogra

phic

split

s for

pot

entia

l mod

el d

isto

rtion

s.Fu

rther

com

men

tary

is n

eede

d to

cla

rify

wha

t is m

eant

by

“geo

grap

hic

stab

ility

” or

w

hat t

ypes

of e

xhib

its w

ould

pro

vide

info

rmat

ion

rega

rdin

g th

is co

ncer

n.

B.4

.k

Des

crib

e ho

w o

verf

ittin

g w

as a

ddre

ssed

and

the

resu

lts o

f cor

rela

tion

test

s.

Esse

ntia

l

B.4

.l

Prov

ide

supp

ort

dem

onst

ratin

g th

at th

e G

LM a

ssum

ptio

ns a

re

appr

opria

te (f

or

exam

ple,

the

choi

ce o

f er

ror d

istri

butio

n).

Esse

ntia

lV

isua

l rev

iew

of p

lots

of a

ctua

l err

ors i

s usu

ally

suff

icie

nt.

B.4

.m

Prov

ide

the

form

ula

rela

tions

hip

betw

een

the

data

and

the

mod

el

outp

uts,

with

a d

efin

ition

of

eac

h m

odel

inpu

t and

ou

tput

. Pro

vide

all

nece

ssar

y co

effic

ient

s to

eval

uate

the

pred

icte

d va

lue

for

any

real

or

hypo

thet

ical

set o

f in

puts

.

Esse

ntia

lB

.4.l

and

B.4

.m w

ill sh

ow th

e m

athe

mat

ical

func

tions

in

volv

ed a

nd c

ould

be

used

to re

prod

uce

som

e m

odel

pr

edic

tions

.

Com

men

tary

shou

ld a

dvis

e th

e re

gula

tor t

hatp

redi

cted

val

ue m

ay n

ot b

e a

part

of th

em

odel

out

put;

only

rela

tiviti

es. A

n ou

tput

that

repr

esen

ts p

redi

cted

pur

e pr

emiu

m

from

a p

redi

ctiv

e m

odel

shou

ld n

ot b

e co

nfus

ed w

ith a

n ac

tuar

ial e

stim

ate

of

pros

pect

ive

pure

pre

miu

m. P

rovi

ding

this

cou

ld in

clud

e ad

ditio

nal d

etai

l abo

ut

mod

elin

g pr

actic

e in

exc

ess o

f wha

t has

bee

n hi

stor

ical

ly p

rovi

ded

or w

hat i

s nee

ded

to a

sses

s goo

dnes

s of f

it.

B.4

.n

Prov

ide

5-10

sam

ple

reco

rds a

nd th

e ou

tput

of

the

mod

el fo

r tho

se

reco

rds.

Esse

ntia

l


4/6/19


of 2

9 5.

“O

ld M

odel

” Ve

rsus

“N

ew M

odel

”

B.5

.a

An

expl

anat

ion

of w

hy

this

mod

el is

bet

ter t

han

the

one

it is

repl

acin

g.

How

was

that

con

clus

ion

form

ed?

Wha

t met

rics

wer

e re

lied

on fo

r m

easu

rem

ent?

Esse

ntia

lR

egul

ator

s sho

uld

expe

ct to

see

impr

ovem

ent i

n th

e ne

w c

lass

pl

an’s

pre

dict

ive

abili

ty o

r oth

er su

ffic

ient

reas

on fo

r the

ch

ange

.

B.5

.b

Wer

e 2

Gin

i coe

ffici

ents

co

mpa

red?

Wha

t was

the

conc

lusi

on d

raw

n fr

om

this

com

paris

on?

May

Be

Req

uest

edO

ne e

xam

ple

of a

com

paris

on m

ight

be

suff

icie

nt.

B.5

.c

Wer

e do

uble

lift

char

ts

anal

yzed

? W

hat w

as th

e co

nclu

sion

dra

wn

from

th

is a

naly

sis?

Esse

ntia

lO

ne e

xam

ple

of a

com

paris

on m

ight

be

suff

icie

nt.

Whe

n w

ould

this

be

rele

vant

? ->

“May

be

Req

uest

ed”

B.5

.d

Prov

ide

a lis

t of a

ll ne

w

pred

icto

r var

iabl

es in

the

mod

el th

at w

ere

not i

n th

e pr

ior m

odel

.

Esse

ntia

lU

sefu

l to

diff

eren

tiate

bet

wee

n ol

d an

d ne

w v

aria

bles

so th

e re

gula

tor c

an p

riorit

ize

mor

e tim

e on

fact

ors n

ot y

et re

view

ed.

B.5

.e

Prov

ide

a lis

t of

pred

icto

r var

iabl

es u

sed

in th

e ol

d m

odel

that

are

no

t use

d in

the

new

m

odel

. Why

wer

e th

ey

drop

ped

from

the

new

m

odel

?

Esse

ntia

l


4/6/19


of 2

9 6.

M

odel

er/S

oftw

are

B.6

.a

Prov

ide

the

nam

es,

cont

act e

mai

ls, p

hone

nu

mbe

rs a

nd

qual

ifica

tions

of t

he k

ey

pers

ons w

ho:

a.

Led

the

proj

ect

b.

Com

pile

d th

e da

tac.

B

uilt

the

mod

eld.

Pe

rfor

med

pee

r re

view

Esse

ntia

l

The

is n

ot “

esse

ntia

l” o

r rel

evan

t for

pur

pose

s of r

evie

win

g ho

w th

e m

odel

is u

sed

in

deve

lopm

ent o

f the

ratin

g pl

an.W

here

els

e is

such

info

rmat

ion

soug

ht?

For e

ach

filin

g, g

ener

ally

ther

e is

aqu

alifi

edac

tuar

y pr

ovid

ing

the

filin

g an

d si

gnin

g on

beh

alf

of th

e co

mpa

ny. W

here

as th

e co

mpa

ny m

ay o

ffer

this

info

rmat

ion,

it is

the

filin

g ac

tuar

y th

at m

ust d

eter

min

e if

the

mod

el/s

oftw

are

deve

lope

r mus

t be

enga

ged

in o

rder

to

resp

ond

suff

icie

ntly

to th

e re

gula

tor’

s inq

uirie

s.

B.6

.b

Wha

t sof

twar

e w

as

used

? Pr

ovid

e th

e na

me

of th

e so

ftwar

e ve

nder

/dev

elop

er,

softw

are

prod

uct a

nd a

so

ftwar

e ve

rsio

n re

fere

nce.

Esse

ntia

lTh

is sh

ould

be

capt

ured

in th

e su

mm

ary

narr

ativ

e fo

r the

mod

el.

B.6

.c

Whe

n di

d w

ork

to b

uild

th

e m

odel

beg

in a

nd

whe

n w

as th

e m

odel

bu

ild fi

naliz

ed?

May

Be

Req

uest

edEs

sent

ial

The

“ess

entia

l”fo

cus s

houl

d be

mor

e on

the

desc

riptio

n of

the

data

(e.g

.yea

rs o

f dat

a;as

of d

ate)

. If t

he m

odel

is a

com

mer

cial

pro

duct

, the

n th

e co

mpa

ny m

ay w

ant t

o of

fer

the

prod

uctio

n ye

ar, v

ersi

on/m

odel

nam

e.N

ever

thel

ess,

this

shou

ld n

ot b

e es

sent

ial t

o th

e re

view

and

exp

lana

tion

of h

ow th

e m

odel

func

tions

and

impa

cts t

he ra

ting

plan

.


4/6/19


of 2

9 C

. The

File

d Ra

ting

Plan

Info

rmat

ion

Impo

rtan

ce to

R

egul

ator

’s R

evie

w

"Ess

entia

l" o

r "M

ay

Be

Req

uest

ed"

Com

men

tsA

PCIA

1.

Gen

eral

Impa

ct o

f Mod

el o

n Ra

ting

Algo

rith

mTh

e Pr

edic

tive

Mod

el is

not

the

Pric

ing

Mod

el.T

he p

redi

ctiv

e m

odel

impa

cts t

he ra

te

indi

catio

n w

hich

influ

ence

s the

rate

sele

ctio

n w

hich

pro

mpt

s the

rate

just

ifica

tion.

C.1

.a

In th

e A

ctua

rial

Mem

oran

dum

sect

ion

on

the

SER

FF S

uppo

rting

D

ocum

enta

tion

tab,

for

each

mod

el re

lied

upon

, in

clud

e a

docu

men

t tha

t ex

plai

ns th

e m

odel

and

its

role

in th

e ra

ting

syst

em.

Esse

ntia

l

This

item

bec

omes

“Es

sent

ial”

if th

e ro

le o

f the

mod

el c

anno

t be

imm

edia

tely

dis

cern

ed b

y th

e re

view

er fr

om a

qui

ck re

view

of

the

rate

and

/or r

ule

page

s. (I

mpo

rtanc

e is

dep

ende

nton

stat

e re

quire

men

ts a

nd e

ase

of id

entif

icat

ion

by th

e fir

st la

yer o

f re

view

and

esc

alat

ion

to th

e ap

prop

riate

revi

ew st

aff.)

C.1

.b

Prov

ide

an e

xpla

natio

n of

how

the

mod

el w

as

used

to a

djus

t the

ratin

g al

gorit

hm.

Esse

ntia

lC

omm

enta

ry sh

ould

advi

se th

e re

gula

tor t

hat m

odel

sare

gen

eral

ly u

sed

for f

acto

r-ba

sed

indi

catio

ns, w

hich

are

then

use

d as

the

basi

s for

sele

cted

cha

nges

to th

e ra

ting

plan

. It i

s the

cha

nges

to th

e ra

ting

plan

that

cre

ate

impa

cts.

C.1

.c

Prov

ide

a co

mpl

ete

list

of a

ll ch

arac

teris

tics/

varia

bles

us

ed in

the

prop

osed

ra

ting

plan

, inc

ludi

ng

thos

e us

ed a

s inp

ut to

the

mod

el (i

nclu

ding

sub-

mod

els a

nd c

ompo

site

va

riabl

es) a

nd a

ll ot

her

char

acte

ristic

s/va

riabl

es

used

to c

alcu

late

a

prem

ium

. For

eac

h ch

arac

teris

tic/v

aria

ble,

in

dica

te if

it is

onl

y in

put

to th

e m

odel

, whe

ther

it

Esse

ntia

lEx

ampl

es o

f var

iabl

es u

sed

as in

puts

to th

e m

odel

and

use

d a

s se

para

te u

niva

riate

ratin

g ch

arac

teris

tics m

ight

be

crite

ria u

sed

to

dete

rmin

e a

ratin

g tie

r or h

ouse

hold

com

posi

te c

hara

cter

istic

.

This

shou

ld b

e co

vere

d al

read

y in

the

gene

ral d

escr

iptio

n of

the

inpu

t and

mod

el d

ata,

whi

ch is

sepa

rate

from

the

othe

r ris

k ch

arac

teris

tics a

nd v

aria

bles

that

may

be

filed

as

part

of th

e ra

ting

plan

.


4/6/19


of 2

9

is o

nly

a se

para

te

univ

aria

te ra

ting

char

acte

ristic

, or w

heth

er

it is

bot

h in

put t

o th

e m

odel

and

a se

para

te

univ

aria

te ra

ting

char

acte

ristic

. The

list

sh

ould

pro

vide

tra

nspa

rent

des

crip

tions

of

eac

h lis

ted

char

acte

ristic

/var

iabl

e.

C.1

.d

For e

ach

char

acte

ristic

/var

iabl

e us

ed a

s bot

h in

put t

o th

e m

odel

(inc

ludi

ng su

b-m

odel

s and

com

posi

te

varia

bles

) and

as a

se

para

te u

niva

riate

ratin

g ch

arac

teris

tic, e

xpla

in

how

thes

e ar

e te

mpe

red

or a

djus

ted

to a

ccou

nt

for p

ossi

ble

over

lap

or

redu

ndan

cy in

wha

t the

ch

arac

teris

tic/v

aria

ble

mea

sure

s.

Esse

ntia

l

Mod

elin

g lo

ss ra

tio w

ith th

ese

char

acte

ristic

s/va

riabl

es a

s con

trol

varia

bles

wou

ld a

ccou

nt fo

r pos

sibl

e ov

erla

p. T

he in

sure

r sho

uld

addr

ess t

his p

ossi

bilit

y or

oth

er c

onsi

dera

tions

, e.g

., tie

r pl

acem

ent m

odel

s ofte

n us

e ris

k ch

arac

teris

tics/

varia

bles

that

are

al

so u

sed

else

whe

re in

the

ratin

g pl

an.

C.1

.e

If th

e fil

ing

supp

ort

incl

udes

an

upda

te o

r re

plac

emen

t of a

n ex

istin

g m

odel

, ide

ntify

an

d ex

plai

n th

e ch

ange

s in

cal

cula

tions

, as

sum

ptio

ns, p

aram

eter

s an

d da

ta u

sed

to b

uild

th

e m

odel

s. Pr

ovid

e an

ex

plan

atio

n of

why

the

upda

ted/

repl

acem

ent

mod

el is

bet

ter t

han

the

one

it is

repl

acin

g,

incl

udin

g, h

ow th

at

conc

lusi

on w

as re

ache

d,

and

the

met

rics r

elie

d up

on to

reac

h th

at

conc

lusi

on.

Esse

ntia

lC

over

ed u

nder

sect

ion

B.5


4/6/19


of 2

9 2.

Re

leva

nce

of V

aria

bles

/ Re

latio

nshi

p to

Ris

k of

Los

s

C.2

.a

Prov

ide

an e

xpla

natio

n ho

w th

e ch

arac

teris

tics/

ratin

g va

riabl

es, i

nclu

ded

in th

e fil

ed ra

ting

plan

, lo

gica

lly a

nd in

tuiti

vely

re

late

to th

e ris

k of

in

sura

nce

loss

(or

expe

nse)

for t

he ty

pe o

f in

sura

nce

prod

uct b

eing

pr

iced

. Inc

lude

a

disc

ussio

n of

the

rele

vanc

e ea

ch

char

acte

ristic

/ratin

g va

riabl

e ha

s on

cons

umer

beh

avio

r tha

t w

ould

lead

to a

di

ffere

nce

in ri

sk o

f los

s (o

r exp

ense

).

Esse

ntia

lTh

is e

xpla

natio

n w

ould

not

be

need

ed if

the

conn

ectio

n be

twee

n va

riabl

es a

nd ri

sk o

f los

s (or

exp

ense

) has

alre

ady

been

ill

ustra

ted.

Cov

ered

und

erA

.4.b

and

B.3

.c. R

efer

toex

istin

g st

anda

rdsi

n A

SOP

12 R

isk

Cla

ssifi

catio

n-3

.2.2

This

shou

ld b

e th

e sa

me

stan

dard

for m

odel

ed c

hara

cter

istic

s/ra

ting

varia

bles

as f

or

non-

mod

eled

.

3.

Com

pari

son

of M

odel

Out

puts

to C

urre

nt a

nd S

elec

ted

Ratin

g Fa

ctor

s

C.3

.a

Prov

ide

a co

mpa

rison

be

twee

n re

lativ

ities

in

dica

ted

by th

e m

odel

to b

oth

curr

ent

rela

tiviti

es a

nd th

e in

sure

r's se

lect

ed

rela

tiviti

es fo

r eac

h ris

k ch

arac

teris

tic/v

aria

ble

in th

e ra

ting

plan

. Ea

ch si

gnifi

cant

di

ffere

nce

shou

ld b

e hi

ghlig

hted

and

ex

plai

ned.

Esse

ntia

l

“Sig

nific

ant d

iffer

ence

” m

ay v

ary

base

d on

the

risk

char

acte

ristic

/var

iabl

e an

d co

ntex

t. H

owev

er, t

he m

ovem

ent o

f a

sele

cted

rela

tivity

shou

ld b

e in

the

dire

ctio

n of

the

indi

cate

d re

lativ

ity; i

f not

, an

expl

anat

ion

is n

eces

sary

as t

o w

hy th

e m

ovem

ent i

s log

ical

.


4/6/19


of 2

9 C

.3.b

Wha

t cal

cula

tions

, ju

dgm

ents

and

ad

just

men

ts, i

f any

, w

ere

mad

e be

fore

us

ing

the

mod

el

outp

ut in

the

ratin

g sy

stem

? Id

entif

y an

y ad

just

men

ts th

at w

ere

mad

e to

the

indi

cate

d m

odel

to d

eriv

e th

e se

lect

ed m

odel

.

Esse

ntia

lC

over

edun

der C

.3.a

C.3

.c

If th

e m

odel

resu

lts in

sc

ores

, tie

rs, o

r ra

nges

of v

alue

s for

w

hich

indi

catio

ns a

re

then

der

ived

for e

ach

such

resu

lting

ca

tego

ry, p

rovi

de

expl

anat

ions

for f

iled

ratin

g va

lues

that

de

viat

e fr

om th

ese

indi

catio

ns a

nd

supp

ortin

g in

form

atio

n/

anal

yses

. For

ex

ampl

e, id

entif

y an

y ad

just

men

ts th

at w

ere

mad

e to

the

fact

ors

indi

cate

d fo

r eac

h ca

tego

ry o

f the

mod

el

outp

uts t

o de

rive

the

fact

ors s

elec

ted

for

the

ratin

g pl

an.

Esse

ntia

lTh

is is

esp

ecia

lly im

porta

nt if

dev

iatio

ns a

re m

ater

ial a

nd/o

r im

pact

one

con

sum

er p

opul

atio

n m

ore

than

ano

ther

.C

over

und

er C

.3.a

4.

Resp

onse

s to

Dat

a, C

redi

bilit

y an

d G

ranu

lari

ty Is

sues

C.4

.a

Wha

t con

side

ratio

n w

as g

iven

to th

e cr

edib

ility

of t

he

outp

ut d

ata?

Esse

ntia

l

At w

hat l

evel

of g

ranu

larit

y is

cre

dibi

lity

appl

ied.

If m

odel

ing

was

by-

cove

rage

, by-

form

or b

y-pe

ril, e

xpla

in h

ow th

ese

wer

e ha

ndle

d w

hen

ther

e w

as n

ot e

noug

h cr

edib

le d

ata

by c

over

age,

fo

rm o

r per

il to

mod

el.


4/6/19


of 2

9 C

.4.b

If ap

plic

able

, dis

cuss

th

e ra

tiona

le fo

r usi

ng

a m

odel

that

is m

ore

gran

ular

than

the

ratin

g pl

an.

Esse

ntia

lTh

is is

app

licab

le if

the

insu

rer h

ad to

com

bine

mod

eled

out

put

in o

rder

to re

duce

the

gran

ular

ity o

f the

ratin

g pl

an.

Com

men

tary

shou

ld c

larif

y ho

w g

ranu

larit

y is

an

“ess

entia

l” re

gula

tory

con

cern

for

the

ratin

g pl

an.T

his s

houl

d be

cov

ered

und

er th

e ge

nera

l nar

rativ

e of

how

the

mod

el

is im

plem

ente

d an

d in

corp

orat

ed in

to th

e ra

ting

plan

.

C.4

.c

If ap

plic

able

, dis

cuss

th

e ra

tiona

le fo

r usi

ng

a ra

ting

plan

that

is

mor

e gr

anul

ar th

an

mod

eled

out

put.

Esse

ntia

l

A m

ore

gran

ular

ratin

g pl

an im

plie

s tha

t the

insu

rer h

ad to

ex

trapo

late

cer

tain

ratin

g tre

atm

ents

, esp

ecia

lly a

t the

tails

of a

di

strib

utio

n of

attr

ibut

es, i

n a

man

ner n

ot sp

ecifi

ed b

y th

e m

odel

in

dica

tions

.

Com

men

tary

shou

ld c

larif

y ho

w g

ranu

larit

y is

an

“ess

entia

l” re

gula

tory

con

cern

for

the

ratin

g pl

an.T

his s

houl

d be

cov

ered

und

er th

e ge

nera

l nar

rativ

e of

how

the

mod

el

is im

plem

ente

d an

d in

corp

orat

ed in

to th

e ra

ting

plan

.

5.

Def

initi

ons o

f Rat

ing

Varia

bles

C.5

.a

Prov

ide

a tra

nspa

rent

pr

esen

tatio

n an

d ex

plan

atio

n of

bi

nnin

g de

cisi

ons t

hat

assi

gn ra

nges

of

mod

el o

utpu

ts to

pa

rticu

lar r

atin

g ca

tego

ries.

Esse

ntia

lC

ombi

ne w

ith A

.3.f

C.5

.b

Prov

ide

com

plet

e de

finiti

ons o

f any

ra

ting

tiers

or o

ther

in

term

edia

te ra

ting

cate

gorie

s tha

t tra

nsla

te th

e m

odel

ou

tput

s int

o so

me

othe

r stru

ctur

e th

at is

th

en p

rese

nted

with

in

the

rate

and

/or r

ule

page

s.

Esse

ntia

l


4/6/19


of 2

9 6.

Su

ppor

ting

Dat

a

C.6

.a

Prov

ide

stat

e-sp

ecifi

c, b

ook-

of-

busi

ness

-spe

cific

un

ivar

iate

his

toric

al

expe

rienc

e da

ta

cons

istin

g of

, at

min

imum

, ear

ned

prem

ium

s, in

curr

ed

loss

es, l

oss r

atio

s and

lo

ss ra

tio re

lativ

ities

fo

r eac

h ca

tego

ry o

f m

odel

out

put(s

) pr

opos

ed to

be

used

w

ithin

the

ratin

g pl

an.

May

Be

Req

uest

edEs

sent

ial

Com

men

tary

shou

ld c

autio

n re

gula

tor t

hat a

lterin

g un

ivar

iate

resu

lts fo

r mod

els b

uilt

with

cou

ntry

wid

e da

ta a

t a st

ate

leve

l may

not b

e st

atis

tical

ly c

redi

ble

or a

ppro

pria

te.

Loss

ratio

resu

lts m

ay n

otbe

com

para

ble

to re

sults

from

a m

odel

sinc

e th

ese

resu

lts

refle

ct th

e ex

istin

g ra

ting

plan

as w

ell a

s los

ses.

Prov

idin

g st

ate-speci

fic d

ata

for e

very

va

riabl

e m

ayno

t mak

e se

nse

beca

use

such

dat

a w

ould

not

be

cred

ible

at t

he st

ate

leve

l in

man

y in

stan

ces.

The

leve

l of i

nfor

mat

ion

may

als

o be

con

side

red

prop

rieta

ry.

C.6

.b

Prov

ide

an

expl

anat

ion

of a

ny

mat

eria

l (es

peci

ally

di

rect

iona

l) di

ffere

nces

bet

wee

n m

odel

indi

catio

ns

and

stat

e-sp

ecifi

c un

ivar

iate

in

dica

tions

.

May

Be

Req

uest

edEs

sent

ial

Mul

tivar

iate

indi

catio

ns m

ay b

e re

ason

able

as r

efin

emen

ts to

un

ivar

iate

indi

catio

ns, b

ut li

kely

not

for b

ringi

ng a

bout

reve

rsal

s of

thos

e in

dica

tions

. For

inst

ance

, if t

he u

niva

riate

indi

cate

d re

lativ

ity fo

r an

attri

bute

is 1

.5 a

nd th

e m

ultiv

aria

te in

dica

ted

rela

tivity

is 1

.25,

this

is p

oten

tially

a p

laus

ible

app

licat

ion

of th

e m

ultiv

aria

te te

chni

ques

. If,

how

ever

, the

uni

varia

te in

dica

ted

rela

tivity

is 0

.7 a

nd th

e m

ultiv

aria

te in

dica

ted

rela

tivity

is 1

.25,

a

regu

lato

r may

que

stio

n w

heth

er th

e at

tribu

te in

que

stio

n is

ne

gativ

ely

corr

elat

ed w

ith o

ther

det

erm

inan

ts o

f ris

k. C

redi

bilit

y of

stat

e da

ta sh

ould

be

cons

ider

ed w

hen

stat

e in

dica

tions

diff

er

from

mod

eled

resu

lts b

ased

on

a br

oade

r dat

a se

t. H

owev

er, t

he

rele

vanc

e of

the

broa

der d

ata

set t

o th

e ris

ks b

eing

pric

ed sh

ould

al

so b

e co

nsid

ered

.

Com

men

tary

shou

ld c

autio

n th

e re

gula

tor t

hatm

akin

g m

odel

mod

ifica

tions

at a

stat

e le

vel m

ay n

ot b

e ap

prop

riate

usi

ng st

ate

spec

ific

data

resu

lts. A

lterin

g un

ivar

iate

re

sults

for m

odel

s bui

lt w

ith c

ount

ryw

ide

data

at a

stat

e le

vel m

ayno

t be

stat

istic

ally

cr

edib

le o

r app

ropr

iate

. The

regu

lato

r sho

uld

be a

dvis

ed th

at a

lthou

gh th

ere

is

pote

ntia

l for

thes

e ty

pe o

f rev

ersa

ls to

reve

al a

con

cern

in th

e m

odel

, it i

s not

at a

ll un

com

mon

or u

nexp

ecte

d to

see

this

eff

ect.

e.g.

, a u

niva

riate

revi

ew in

dica

ting

high

er

pure

pre

miu

ms f

or h

omes

with

a so

phis

ticat

edal

arm

syst

em;a

n ef

fect

refle

ctin

gal

arm

s in

hom

es o

f hig

her t

han

aver

age

valu

e an

d a

“rev

ersa

l” e

xpec

ted

unde

r a

mul

tivar

iate

revi

ew. H

owev

er, r

elat

ions

hips

can

be

very

com

plic

ated

and

diff

icul

t to

teas

e ou

t in

prac

tice.

The

filin

g ac

tuar

y sh

ould

be

prep

ared

to p

rovi

deth

ese

expl

anat

ions

,cor

rela

tions

and

inte

ract

ions

in d

ata.

7.

Con

sum

er Im

pact

sH

ow d

oes s

ectio

n C

.7 a

pply

in o

ther

com

pone

nts o

f the

rate

filin

g? T

he “

esse

ntia

l”

focu

s sho

uld

be o

n ho

w th

e m

odel

in q

uest

ion

impa

cts t

he ra

ting

plan

.

C.7

.a

Iden

tify

mod

el

chan

ges a

nd ra

ting

varia

bles

that

will

ca

use

larg

e pr

emiu

m

disr

uptio

ns.

Esse

ntia

lC

omm

enta

ry sh

ould

focu

s reg

ulat

or’s

atte

ntio

n on

the

key

ratin

g fa

ctor

s tha

t cou

ld b

e th

e ca

use

ofa

larg

e pr

emiu

m sw

ing.

Ove

rall

rate

cha

nge

hist

ogra

ms c

an sh

ow

chan

ges i

n th

e ke

y se

lect

ed fa

ctor

s.


4/6/19


of 2

9 C

.7.b

Did

the

insu

rer

perf

orm

sens

itivi

ty

test

ing

to id

entif

y si

gnifi

cant

cha

nges

in

prem

ium

due

to sm

all

or in

crem

enta

l ch

ange

in a

sing

le

risk

char

acte

ristic

? If

so

, dis

cuss

and

pr

ovid

e th

e re

sults

of

that

test

ing.

May

Be

Req

uest

ed

One

way

to se

e se

nsiti

vity

is to

ana

lyze

a g

raph

of e

ach

risk

char

acte

ristic

’s/v

aria

ble’

s pos

sibl

e re

lativ

ities

. Loo

k fo

r si

gnifi

cant

var

iatio

n be

twee

n ad

jace

nt re

lativ

ities

and

eva

luat

e if

such

var

iatio

n is

reas

onab

le.

C.7

.c

Mea

sure

and

des

crib

e th

e im

pact

s on

expi

ring

polic

ies a

nd

desc

ribe

the

proc

ess

used

by

man

agem

ent

to m

itiga

te o

r get

co

mfo

rtabl

e w

ith

thos

e im

pact

s.

Esse

ntia

lTh

is is

not

rele

vant

to th

e re

view

of h

ow th

e m

odel

is im

plem

ente

d an

d in

corp

orat

ed

into

the

ratin

g pl

an

C.7

.d

Prov

ide

a ra

te

disr

uptio

n an

alys

is,

dem

onst

ratin

g th

e di

strib

utio

n of

pe

rcen

tage

impa

cts

on re

new

al b

usin

ess

(cre

ate

by re

ratin

g th

e cu

rren

t boo

k of

bu

sine

ss).

Incl

ude

the

larg

est d

olla

r and

pe

rcen

tage

impa

cts

aris

ing

from

the

filin

g, in

clud

ing

(des

irabl

y) th

e im

pact

s aris

ing

spec

ifica

lly fr

om th

e ad

optio

n of

the

mod

el

or c

hang

es to

the

mod

el a

s the

y tra

nsla

te in

to th

e pr

opos

ed ra

ting

plan

.

Esse

ntia

l

Whi

le th

e de

faul

t req

uest

wou

ld ty

pica

lly b

e fo

r the

dis

tribu

tion

of im

pact

s at t

he o

vera

ll fil

ing

leve

l, th

e re

gula

tor m

ay n

eed

to

delv

e in

to th

e m

ore

gran

ular

var

iabl

e-sp

ecifi

c ef

fect

s of r

ate

chan

ges i

f the

re is

con

cern

abo

ut p

artic

ular

var

iabl

es h

avin

g ex

trem

e or

dis

prop

ortio

nate

impa

cts,

or si

gnifi

cant

impa

cts t

hat

have

oth

erw

ise

yet t

o be

subs

tant

iate

d.Se

e A

ppen

dix

C fo

r an

exam

ple

of a

dis

rupt

ion

anal

ysis

.

This

is n

ot re

leva

nt to

the

revi

ew o

f how

the

mod

el is

impl

emen

ted

and

inco

rpor

ated

in

to th

e ra

ting

plan


4/6/19


of 2

9 C

.7.e

Prov

ide

expo

sure

di

strib

utio

ns fo

r ou

tput

var

iabl

es a

nd

show

the

effe

cts o

f ra

te c

hang

es a

t gr

anul

ar a

nd

sum

mar

y le

vels

.

May

Be

Req

uest

edEs

sent

ial

See

App

endi

x C

for a

n ex

ampl

e of

an

expo

sure

dis

tribu

tion.

The

regu

lato

r sho

uld

cons

ider

that

this

info

rmat

ion

may

be c

onsid

ered

pro

prie

tary

C.7

.f

Expl

ain

how

the

insu

rer

will

hel

p ed

ucat

e co

nsum

ers t

o m

itiga

te th

eir r

isk.

May

Be

Req

uest

edEs

sent

ial

This

is n

ot re

leva

nt to

the

revi

ew o

f how

the

mod

el is

impl

emen

ted

and

inco

rpor

ated

in

to th

e ra

ting

plan

C.7

.g

Iden

tify

sour

ces t

o be

us

ed a

t "po

int o

f sal

e"

to p

lace

indi

vidu

al

risks

with

in th

e m

atrix

of r

atin

g sy

stem

cl

assi

ficat

ions

. How

ca

n a

cons

umer

ve

rify

thei

r ow

n "p

oint

-of-

sale

" da

ta

and

corr

ect a

ny

erro

rs?

May

Be

Req

uest

edEs

sent

ial

Cou

ld b

e "E

ssen

tial"

if th

e va

riabl

es/ c

hara

cter

istic

s use

d co

uld

1) h

ave

publ

ic-p

olic

y im

plic

atio

ns, 2

) res

ult i

n er

rone

ous

info

rmat

ion

bein

g us

ed, o

r 3) r

esul

t in

man

y la

rge,

dis

rupt

ive

prem

ium

cha

nges

at r

enew

al. A

noth

er c

onsi

dera

tion

to ju

dge

“im

porta

nce”

is w

heth

er c

onsu

mer

s are

pro

activ

ely

invo

lved

(e

.g.,

use

of c

onsu

mer

cre

dit i

nfor

mat

ion

and

cred

it-re

port

accu

racy

issu

es).

This

is n

ot re

leva

nt to

the

revi

ew o

f how

the

mod

el is

impl

emen

ted

and

inco

rpor

ated

in

to th

e ra

ting

plan

C.7

.h

Iden

tify

ratin

g va

riabl

es th

at re

mai

n st

atic

ove

r a

cons

umer

’s li

fetim

e ve

rsus

thos

e th

at w

ill

be u

pdat

ed

perio

dica

lly.

Doc

umen

t gui

delin

es

for v

aria

bles

that

are

lis

ted

as st

atic

yet

for

whi

ch th

e un

derly

ing

cons

umer

attr

ibut

es

may

cha

nge

over

tim

e.

May

Be

Req

uest

edEs

sent

ial

This

is n

ot re

leva

nt to

the

revi

ew o

f how

the

mod

el is

impl

emen

ted

and

inco

rpor

ated

in

to th

e ra

ting

plan

C.7

.i

Prov

ide

the

regu

lato

r w

ith a

des

crip

tion

of

how

the

com

pany

w

ill re

spon

d to

co

nsum

ers’

inqu

iries

May

Be

Req

uest

edEs

sent

ial

This

is n

ot re

leva

nt to

the

revi

ew o

f how

the

mod

el is

impl

emen

ted

and

inco

rpor

ated

in

to th

e ra

ting

plan


4/6/19


of 2

9

abou

t how

thei

r pr

emiu

m w

as

calc

ulat

ed.

C.7

.j

Prov

ide

the

regu

lato

r w

ith a

mea

ns to

ca

lcul

ate

the

rate

ch

arge

d a

cons

umer

.

May

Be

Req

uest

ed

Espe

cial

ly fo

r a c

ompl

ex m

odel

or r

atin

g pl

an, a

scor

e or

pr

emiu

m c

alcu

lato

r via

Exc

el o

r sim

ilar m

eans

wou

ld b

e id

eal,

but t

his c

ould

be

elic

ited

on a

cas

e-by

-cas

e ba

sis.

Abi

lity

to

calc

ulat

e th

e ra

te c

harg

ed c

an a

llow

the

regu

lato

r to

perfo

rm

sens

itivi

ty te

stin

g w

hen

ther

e ar

e sm

all c

hang

es to

a ri

sk

char

acte

ristic

/var

iabl

e.

The

regu

lato

r sho

uld

cons

ider

that

the

com

pany

may

not

be

able

toha

nd o

ver a

prop

rieta

ry ra

ting

mod

el a

s par

t oft

he re

view

of t

he p

redi

ctiv

e m

odel

.In

othe

r cas

es,

insu

rers

may

not

be

able

topr

ovid

e re

gula

tors

with

the

mea

ns to

recr

eate

the

outp

ut o

f a

prop

rieta

ry m

odel

.

Reg

ulat

ors s

houl

d be

cau

tione

d th

at if

insu

ranc

e co

mpa

nies

are

requ

ired

to b

uild

an

Exce

l-type too

l for

regu

lato

rs to

be

able

to c

alcu

late

the

prem

ium

cha

rged

, it c

ould

hi

nder

the

filin

g re

view

pro

cess

and

det

er a

ny in

vest

men

t in

inno

vativ

e ra

ting/

mod

elin

g te

chni

ques

.

8.

Accu

rate

Tra

nsla

tion

of M

odel

into

a R

atin

g Pl

an

C.8

.a

Prov

ide

suff

icie

nt

info

rmat

ion

for t

he

revi

ewer

to b

e ab

le to

un

ders

tand

how

the

mod

el o

utpu

ts a

re

used

with

in th

e ra

ting

syst

em a

nd to

ver

ify

that

the

ratin

g pl

an, i

n fa

ct, r

efle

cts t

he

mod

el o

utpu

t and

any

ad

just

men

ts m

ade

to

the

mod

el o

utpu

t.

Esse

ntia

l

III. D

O R

EGU

LATO

RS N

EED

BEST

PRA

CTIC

ES T

O R

EVIE

W P

REDI

CTIV

E M

ODE

LS?

A G

LM c

onsi

sts o

f thr

ee e

lem

ents6

:• A

pro

babi

lity

dist

ribu

tion

from

the

expo

nent

ial f

amily

.

1

APCI

A: S

tric

tly sp

eaki

ng th

is is

not t

rue

as G

LM u

ses q

uasi-

likel

ihoo

d ra

ther

than

act

ual p

roba

bilit

y di

strib

utio

ns.

The

first

ele

men

t in

the

bulle

t poi

nt li

st sh

ould

be

a “v

aria

nce

func

tion”

, not

a p

roba

bilit

y di

strib

utio

n.

In a

dditi

on, G

LM o

utpu

t is a

ssum

ed, a

s par

t of t

he m

odel

des

ign,

to b

e 10

0% c

redi

ble

no m

atte

r the

size

of t

he u

nder

lyin

g da

ta se

t.

APCI

A: T

his s

tate

men

t is f

alse

. An

adv

anta

ge o

f GLM

is th

at w

e ca

n as

sign

stan

dard

err

ors a

nd c

onfid

ence

inte

rval

s to

the

estim

ator

s. I

thin

k th

e in

tent

ion

was

to c

autio

n th

at G

LM p

oint

-est

imat

e re

sults

are

not

go

spel

trut

h an

d ne

ed to

be

eval

uate

d fo

r cre

dibi

lity

of v

olum

e an

d di

agno

stic

s on

the

mod

el a

ssum

ptio

ns.


4/6/19


of 2

9 IV

. SC

OPE

Rec

omm

ende

d ed

its a

s fol

low

s:

The

focu

s of t

his p

aper

will

be

on G

LM

s use

d to

cre

ate

priv

ate

pass

enge

r au

tom

obile

and

hom

eow

ners

’ ins

uran

ce r

atin

g pl

ans.

Gui

danc

e an

d th

e kn

owle

dge

need

ed to

revi

ew p

redi

ctiv

e m

odel

s ide

ntifi

ed in

this

pap

er a

re, i

n- la

rge

part

,may

not

be

dire

ctly

tran

sfer

rabl

e to

oth

er ty

pes o

f pre

dict

ive

mod

els,

to o

ther

line

s of b

usin

ess,

or o

ther

insu

ranc

e en

deav

ors,

e.g

., co

mm

erci

al a

utom

obile

, wor

kers

’ com

pens

atio

n, m

arke

ting,

und

erw

ritin

g, o

r cla

ims.

Whe

n tr

ansf

errin

g gu

idan

ce to

oth

er li

nes o

f bus

ines

s or o

ther

insu

ranc

e en

deav

ors,

uni

que

cons

ider

atio

ns n

eed

to b

e gi

ven

to

the

cred

ibili

ty o

f the

dat

a an

d le

ss h

omog

eneo

us n

atur

e of

com

mer

cial

bus

ines

s,m

ay a

rise

depe

ndin

g on

the

cont

ext i

n w

hich

how

a p

redi

ctiv

e m

odel

is p

ropo

sed

to b

e de

ploy

ed, t

he u

ses t

o w

hich

it is

pro

pose

d to

be

put,

and

the

pote

ntia

l con

sequ

ence

s of a

n in

sure

r act

ing

on th

e ou

tput

of a

ny g

iven

pre

dict

ive

mod

el. T

his p

aper

doe

s not

del

ve in

to th

ese

poss

ible

con

side

ratio

ns b

ut re

gula

tors

shou

ld b

e pr

epar

ed to

add

ress

them

as t

hey

aris

e.

APCI

A: T

rans

fera

bilit

y - T

he c

onsid

erat

ions

for c

omm

erci

al li

nes m

ay d

iffer

from

per

sona

l lin

es, a

nd c

erta

inly

the

guid

ance

her

e re

late

d to

pre

dict

ive

mod

els i

ncor

pora

ted

into

file

d ra

ting

plan

s is e

ven

mor

e di

stan

ced

from

mod

els u

sed

in th

e co

ntex

t of m

arke

ting

or c

laim

s. A

dditi

onal

ly, h

avin

g a

one-

size-

fits a

ll ap

proa

ch to

eva

luat

ing

mod

els

acro

ss a

ll re

gula

tory

func

tions

, ins

uran

ce a

pplic

atio

ns, a

nd li

nes o

f bus

ines

s wou

ld b

e ov

er-c

onsu

min

g to

the

detr

imen

t of t

he sh

ared

goa

l for

this

effo

rt.

V. C

ON

FIDE

NTI

ALIT

Y

APCI

A: G

reat

er a

tten

tion

need

s to

be g

iven

to th

is se

ctio

n. B

efor

e su

ch “

best

pra

ctic

es”

can

be e

mpl

oyed

, the

regu

lato

r mus

t con

sider

that

ext

ensio

n or

enh

ance

men

t of c

onfid

entia

lity

prot

ectio

ns m

ay b

e ne

cess

ary

to e

xcha

nge

such

mod

elin

g in

telle

ctua

l pro

pert

y.

VI. G

UID

ANCE

FO

R RE

GULA

TORY

REV

IEW

OF

PRED

ICTI

VE M

ODE

LS (B

EST

PRAC

TICE

S)

APCI

A: R

egul

ator

y gu

idan

ce n

eede

d to

cla

rify

dete

rmin

atio

n of

unf

airly

disc

rimin

ator

y of

‘inp

uts’

and

‘out

puts

’. Ho

w d

o ‘o

utpu

ts’ v

ersu

s ‘in

puts

’ get

eva

luat

ed a

s fai

r/un

fair.

VIII.

PRO

POSE

D CH

ANG

ES T

O T

HE P

RODU

CT F

ILIN

G R

EVIE

W H

ANDB

OO

K

XIII.

APP

ENDI

X B

- - G

LOSS

ARY

OF

TERM

S

PCA

appr

oach

(Prin

cipa

l Com

pone

nt A

naly

sis) –

The

PCA

app

roac

h is

als

o kn

own

as fa

ctor

ana

lysi

s. T

hese

met

hods

cre

ate

mul

tiple

new

var

iabl

es fr

om c

orre

late

d gr

oups

of p

redi

ctor

s. T

hose

new

var

iabl

es e

xhib

it lit

tle o

r no

corr

elat

ion

betw

een

them

—th

ereb

y m

akin

g th

em p

oten

tially

mor

e us

eful

in a

GLM

. A P

CA in

a fi

ling

can

be d

escr

ibed

as “

a G

LM w

ithin

a G

LM.”

One

of t

he m

ore

com

mon

app

licat

ions

of P

CA is

geo

dem

ogra

phic

ana

lysi

s, w

here

man

y at

trib

utes

are

use

d to

mod

ify te

rrito

rial d

iffer

entia

ls o

n, fo

r exa

mpl

e, a

cen

sus b

lock

leve

l.

APCI

A: S

tric

tly sp

eaki

ng, f

acto

r ana

lysis

is a

diff

eren

t tec

hniq

ue th

at tr

ies t

o se

para

te o

ut ra

ndom

var

ianc

e fr

om la

tent

var

iabl

e va

rianc

e. R

e-ph

rase

to sa

y th

at th

ey a

re si

mila

r or r

elat

ed a

ppro

ache

s.


4/6/19



4/6/19


California DOI CommentsDraft: 10/25/18As adopted by the Casualty Actuarial and Statistical (C) Task Force on XX/XX/XX


Casualty Actuarial and Statistical (C) Task Force

Regulatory Review of Predictive Models

Table of Contents

I. Introduction ...............................................................................................................................................................2 II. What is a “Best Practice?”..........................................................................................................................................2 III. Do Regulators Need Best Practices to Review Predictive Models? ..............................................................................3 IV. Scope.........................................................................................................................................................................4 V. Confidentiality...........................................................................................................................................................5 VI. Guidance for Regulatory Review of Predictive Models (Best Practices) ......................................................................5 VII. Predictive Models – Information for Regulatory Review.............................................................................................6 VIII. Proposed Changes to the Product Filing Review Handbook ...................................................................................... 22 IX. Proposed State Guidance.......................................................................................................................................... 22 X. Other Considerations................................................................................................................................................ 22 XI. Recommendations Going Forward ........................................................................................................................... 22 XII. Appendix A – Best Practice Development ................................................................................................................ 23 XIII. Appendix B - - Glossary of Terms............................................................................................................................ 24 XIV. Appendix C – Sample Rate-Disruption Template...................................................................................................... 25 XV. Appendix D – Information Needed by Regulator Mapped into Best Practices............................................................ 28 XVI. Appendix E – References ......................................................................................................................................... 28


4/6/19




I. INTRODUCTION

Insurers’ use of predictive analytics along with big data has significant potential benefits to both consumers and insurers.by transforming the insurer-consumer experience into a more meaningful relationship. Predictive analytics can reveal insights into the relationship between consumer behavior and the cost of insurance, lower the cost of insurance for many, and provide incentives tools for the consumers to better control and mitigate loss. However, predictive analytic techniques are evolving rapidly and leaving many regulators without the necessary tools to effectively review insurers’ use of predictive models in insurance applications.

When a rate plan is truly innovative, the insurer must anticipate or imagine the reviewers’ interests because reviewers will respond with unanticipated questions and have unique educational needs. Insurers can learn from the questions, teach the reviewers, and so forth. When that back-and-forth learning is history, filing requirements and insurer presentations can be routinely organized to meet or exceed reviewers’ needs and expectations. Hopefully, this paperhelps bring more consistency and even uniformity to the art of reviewing predictive models within a rate filing.

The Casualty Actuarial and Statistical (C) Task Force (CASTF) has been charged with identifying best practices to serve as a guide to state insurance departments in their review of predictive models1 underlying rating plans. There were two charges given to CASTF by the Property and Casualty Insurance (C) Committee at the request of the Big Data (EX) Working Group:

A. Draft and propose changes to the Product Filing Review Handbook to include best practices for review of predictive models and analytics filed by insurers to justify rates.

B. Draft and propose state guidance (e.g., information, data) for rate filings that are based on complex predictive models.

This paper will identify best practices when reviewing predictive models and analytics filed by insurers with regulators to justify rates and provide state guidance for review of rate filings based on predictive models. Upon adoption of this paper by the Executive (EX) Committee and Plenary, the Task Force will evaluate how to incorporate these best practices into the Product Filing Review Handbook and will recommend such changes to the Speed to Market (EX) Working Group.

II. WHAT IS A “BEST PRACTICE?”

A best practice is a form of program evaluation in public policy. At its most basic level, a practice is a “tangible and visible behavior… [based on] an idea about how the actions…will solve a problem or achieve a goal” 2. Best practices are used to maintain quality as an alternative to mandatory legislated standards and can be based on self-assessment or benchmarking.3 Therefore, a best practice represents an effective method of problem solving.

A. Key Regulatory Principles

In this paper, best practices are based on the following principles that promote a comprehensive and coordinated review of predictive models across states:

1 In this paper, reference to “model” or “predictive model” are the same as “complex predictive model” unless qualified.2 Bardach, E. and Patashnik, E. (2016.) A Practical Guide for Policy Analysis, The Eightfold Path to More Effective Problem Solving.

Thousand Oaks, CA: CQ Press. See Appendix A for an overview of Bardach’s best-practice analysis. 3 Bogan, C.E. and English, M.J. (1994). Benchmarking for Best Practices: Winning Through Innovative Adaptation. New York: McGraw-

Hill.


4/6/19




1. State insurance regulators will maintain their current rate regulatory authority.

2. State insurance regulators will be able to share information to aid companies in getting insurance products to market more quickly.

3. State insurance regulators will share expertise and discuss technical issues regarding predictive models.

4. State insurance regulators will maintain confidentiality, where appropriate, regarding predictive models.

In this paper, best practices are presented in the form of guidance to regulators who review predictive models and to insurance companies filing rating plans that incorporate predictive models. Guidance will identify specific information useful to a regulator in the review of a predictive model, comment on what might be important about that information and, where appropriate, provide insight as to when the information might identify an issue the regulator needs to be aware of or explore further.

III. DO REGULATORS NEED BEST PRACTICES TO REVIEW PREDICTIVE MODELS?

The term “predictive model” refers to a set of models that use statistics to predict outcomes4. When applied to insurance, the model is chosen to estimate the probability or expected value of an outcome given a set amount of input data; for example, models can predict the frequency of loss, the severity of loss, or the pure premium. The generalized linear model (GLM)5 is a commonly used predictive model in insurance applications, particularly in building an insurance product’s rating plan.

Depending on definitional boundaries, predictive modeling can sometimes overlap with the field of machine learning. In this modeling space, predictive modeling is often referred to as predictive analytics.

Before GLMs became vogue, rating plans were built using univariate methods. Univariate methods were considered intuitive and easy to demonstrate the relationship to costs (loss and/or expense). Today, many insurers consider univariate methods too simplistic since they do not take into account the interaction (or dependencies) of the selected input variables and they may imply an assumption of a constant variance across the range of a target variable.

According to many in the insurance industry, GLMs introduce significant improvements over univariate-based rating plans by automatically adjusting for correlations among input variables. Today, the majority of predictive models used in private passenger automobile and homeowners’ rating plans are GLMs. However, GLM results are not always intuitive,and the relationship to costs may be difficult to explain. This is one of the primary reasons regulators can benefit from best practices.

A GLM consists of three elements6:

A probability distribution from the exponential family.

1

As can be seen in the description of the three GLM components above, it may take more than a casual introduction to statistics to comprehend the construction of a GLM. As stated earlier, a downside to GLMs is that it is more challenging to interpret the GLMs output than with univariate models. In addition, GLM output is assumed, as part of the model design, to be 100% credible no matter the size of the underlying data set. Because of this presumption in credibility, which may or may not be valid in practice, the modeler and the regulator reviewing the model would need to engage inthoughtful consideration when incorporating GLM output into a rating plan to ensure that model predictiveness is not compromised by any lack of actual credibility.

4 A more thorough exploration of different predictive models will be found in many statistics’ books, including Geisser,

Seymour(September 2016). Predictive Inference: An Introduction. New York: Chapman & Hall.5 The generalized linear model (GLM) is a flexible family of models that are unified under a single method. Types of GLM include logistic

regression, Poisson regression, gamma regression and multinomial regression. 6 More information on model elements can be found in most statistics’ books.


4/6/19




To further complicate regulatory review of models in the future, modeling methods are evolving rapidly and not limited to GLMs. As computing power grows exponentially, it is opening up the modeling world to more sophisticated forms of data acquisition and data analysis. Insurance actuaries and data scientists are seeking increased predictiveness by using even more complex predictive modeling methods. Examples of these are predictive models utilizing random forests, decision trees, neural networks, or combinations of available modeling methods (often referred to as ensembles). These evolving techniques will make the regulators’ understanding and oversight of filed rating plans incorporating predictive models even more challenging.

In addition to the growing complexity of predictive models, many state insurance departments do not have in-house actuarial support or have limited resources to contract out for support when reviewing rate filings that include use of predictive models. The Big Data (EX) Working Group identified the need to provide states with guidance and assistance when reviewing predictive models underlying filed rating plans.7 The Working Group circulated a proposal addressing aid to state insurance regulators in the review of predictive models as used in private passenger automobile and homeowners’ insurance rate filings. This proposal was circulated to all of the Working Group members and interested parties on December 19, 2017, for a public comment period ending January 12, 2018.8 The Big Data Working Groupeffort resulted in the new CASTF charges (see the Introduction section) with identifying best practices that provide guidance to states in the review of predictive models.

So, to get to the question asked by the title of this section: Do regulators need best practices to review predictive models? It might be better to ask this question another way: Are best practices in the review of predictive models of value to regulators and insurance companies? The answer is “yes” to both questions. Best practices will aid regulatory reviewers by raising their level of model understanding. However, best practices are not intended to create standards for filings that include predictive models. Rather, best practices will assist the states in identifying the model elements they should be looking for in a filing that will aid the regulator in understanding why the company believes that the filed predictive model improves the company’s rating plan, making that rating plan fairer to all consumers in the marketplacemaking the company more competitive in the marketplace. To make this work, both regulators and industry need to recognize that:

Best practices merely provide guidance to regulators in their essential and authoritative role over the rating plans in their state.

All states may have a need to review predictive models whether that occurs with approval of rating plans or in amarket conduct exam. Best practices help the regulator identify elements of a model that may influence the regulatory review as to whether modeled rates are appropriately justified. Each regulator needs to decide if the insurer’s proposed rates are compliant with state laws and regulations and whether to act on that information.

Best practices will lead to improved quality in predictive model reviews across states, aiding speed to market and competitiveness of the state marketplace.

Best practices provide a framework for states to share knowledge and resources to facilitate the technical review of predictive models.

Best practices aid training of new regulators and/or regulators new to reviewing predictive models. (This is especially useful for those regulators who do not actively participate in NAIC discussions related to the subject of predictive models.)

Each regulator adopting best practices will be better able to identify the resources needed to assist their state in the review of predictive models.

Lastly, from this point on in this paper, best practices will be referred to as “guidance.” This reference is in line with the intent of this paper to support individual state autonomy in the review of predictive models.

IV. SCOPE

The focus of this paper will be on GLMs used to create private passenger automobile and homeowners’ insurance rating plans.

7 Minutes of the Big Data (EX) Working Group, March 9, 2018: https://secure.naic.org/secure/minutes/2018_spring/ex_it_tf.pdf?598 All comments received by the end of January were posted to the NAIC website March 12 for review.

Commented [LKW1]: The regulator’s role is not to improve the competitiveness of a particular company in the marketplace, but rather to ensure that rates are fair and the marketplace as a whole is competitive.


4/6/19




Guidance and the knowledge needed to review predictive models identified in this paper are, in large part, transferrable to other types of predictive models, to other lines of business, or other insurance endeavors, e.g., commercial automobile, workers’ compensation, marketing, underwriting, or claims. When transferring guidance to other lines of business or other insurance endeavors, unique considerations may arise depending on the context in which a predictive model is proposed to be deployed, the uses to which it is proposed to be put, and the potential consequences of an insurer acting on the output of any given predictive model. This paper does not delve into these possible considerations but regulators should be prepared to address them as they arise.

V. CONFIDENTIALITY

Insurers and regulators should be aware that a rate filing might become part of the public record. Each state determines the confidentiality of a rate filing, supplemental material to the filing, when filing information might become public, the procedure to request that filing information be held confidentially, and the procedure by which a public records request is made. It is incumbent on an insurer to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filing.

VI. GUIDANCE FOR REGULATORY REVIEW OF PREDICTIVE MODELS (BEST PRACTICES)

Encourage and maintain a competition competitive among insurersmarketplace.

Protect the confidentiality of filed predictive models and supporting information (according to state law).

Review a predictive model efficiently.

Obtain a clear understanding of the characteristics that are input to a predictive model (and its sub-models), their relationship to each other and their relationship to non-modeled characteristics/variables used to calculate a risk’s premium.

Determine that individual input characteristics to a predictive model are related to the expected loss or expense differences in risk. Each input characteristic should have an intuitive or demonstrable actual relationship to expected loss or expense.

Determine that the data used as input to the predictive model is accurate, including a clear understanding how missing values, erroneous values and outliers are handled.

Determine that any adjustments to the raw data are handled appropriately, including but not limited to, trending, development, capping, removal of catastrophes.

Determine that individual input characteristics to a predictive model (and its sub-models) are not unfairly discriminatory and do not reflect proxies for prohibited characteristics.

Obtain a clear understanding of how the selected predictive model was built and why the insurer believes this type of model works in a private passenger automobile or homeowner’s insurance risk application.

Determine that individual output characteristics from a predictive model are related to expected loss or expense differences in risk. Each output characteristic should have a demonstrable actual relationship to expected loss or expense.

Obtain a clear understanding of how model output interacts with non-modeled characteristics/variables used to calculate a risk’s premium.

Determine that individual outputs from a predictive model and their associated selected relativities are not unfairly discriminatory or otherwise inappropriate.

Obtain a clear understanding of how the predictive model was integrated into the insurer’s state rating plan and how it improves the state ratingthat plan, (this latter element is only applicable when a new or revised model is introduced into an existing rating plan).

For predictive model refreshes, determine whether sufficient validation was performed to ensure the model is still a good fit.

Formatted: Font: (Default) Times New Roman, 10 pt,

Formatted: Normal, Left, No bullets or numbering

Formatted: Font: (Default) Times New Roman, 10 pt, Formatted: Font: (Default) Times New Roman, 10 pt,

Formatted: Font: Italic

Commented [WL2]: This should reflect model refreshes as well. Always applicable.

Formatted: Normal, Left, No bullets or numbering

Formatted: Font: (Default) Times New Roman, 10 pt, Formatted: Font: (Default) Times New Roman, 10 pt,

Formatted: Font: Italic


4/6/19




Determine the extent the model causes premium disruption for individual policyholders, and how the insurer will explain the disruption to individual consumers that inquire about it.

Determine the means available to a consumer to correct or contest individual data input values that may be in error.

Obtain a clear understanding how often each risk characteristics used as input to the model is updated and whether the model is periodically rerun to reflect changes to non-static characteristics.

Given an insurer’s rating plan relies on a predictive model and knowing all characteristics of a risk, a regulator should be able to audit/calculate the risk’s premium without consultation with the insurer.

VII.PREDICTIVE MODELS – INFORMATION FOR REGULATORY REVIEW

This section of the paper identifies the information a regulator may need to review a predictive model used by an insurer to support a filed P/C insurance rating plan. The list is lengthy but not exhaustive. It is not intended to limit the authority of a regulator to request additional information in support of the model or filed rating plan. Nor is every item on the list intended to be a required for every filing. However, the items listed should help guide a regulator to obtain sufficient information to determine if the rating plan meets state specific filing and legal requirements.

Though the list seems long, the insurer should already have internal documentation on the model for more than half of the information listed. The remaining items on the list require either minimal analysis (approximately 25%) or deeper analysis to generate the information for a regulator (approximately 25%).

Commented [LKW3]: This is more of a general requirement for any rating variable, and not specific to predictive model reviews.


4/6/19




A. Selecting Model Input

Information

Importance to Regulator’s

Review"Essential" or

"May Be Requested"

Comments

1. Available Data Sources

A.1.a

Provide details of all data sources including the experience period for insurance data and when the data was last recorded or updated.

Essential

This information can be used to evaluate the completeness of the data source, integrity of the data source, relevance of the data to the predictive timeframe, the potential for historical bias,transparency to insured of the data source, and the ability of the insured to make corrections to the data source.

A.1.a.i

Provide reconciliation between raw insurance data and external insurance reports.

EssentialAccuracy of insurance data should be reviewed as well.

A.1.a.ii

Provide the evaluation date of both insurance data, non-insurance or external data.

Essential

This information can be used to identify the staleness of data, especially external/non-insurance data, which may limit its relevance. The external/non-insurance data may not currently undergo the same scrutiny as insurance data and as the use of big data becomes more prevalent in insurance rating, this aspect of review will become more critical.

A.1.b Specify the companies whose data is included in the datasets.

May Be RequestedEssential

If the filer is part of a group, do the datasets include data from affiliated companies? If so, which companies? If the filer is an advisory organization, what companies are used? Are the companies included in the data relevant and compatible to the company that filed the rating plan?

A.1.cProvide the geographical scope and geographic exposure distribution of the data.

Essential

Evaluate whether the data is relevant to the loss potential for which it is being used. For example, verify that hurricane data is only used where hurricanes can occur.

A.1.d

List each data source. For each source, list all data elements used as input to the model that came from that source.

Essential

A.1.eSpecify the type of data (e.g., accident year or policy year, text, numeric).

Essential


4/6/19




A.1.f

Explain if internal or external data was used and if external data was used, disclose reliance on data supplied by others.

Essential

A.1.g

Provide details of any non-insurance data used (customer-provided or other), including who owns this data, how consumers can verify their data and correct errors, whether the data was collected by use of a questionnaire/checklist, whether it was voluntarily reported by the applicant, and whether any of the variables are subject to the Fair Credit Reporting Act. If the data is from an outside source, what steps were taken to verify the data was accurate?

EssentialIf the data is from a third-party source, the company should provide information on how the sourceaddresses the questions in this consideration.

2. Sub-Models

A.2.a

Disclose reliance on sub-modeloutput used as input to this model. If a sub-model was relied upon, provide the vendor name, and the name and version of the sub-model. If the sub-model was built/created in-house, provide contact information for the person responsible for the sub-model.

Essential

Examples of such sub-models include credit/financial scoring algorithms and household composite score models. Sub-models can be evaluated separately and in the same manner as the primary model under evaluation.

A.2.b

If using catastrophe model output, identify the vendor and the model settings/assumptions used when the model was run.

EssentialFor example, it is important to know hurricane model settings for storm surge, demand surge, long/short-term views.

A.2.c

If using catastrophe model output (a sub-model) as input to the GLMunder review, disclose whether loss associated with the modeled output was removed from the lossexperience datasets.

Essential

If a weather-based sub-model is input to the GLM under review, loss data used to develop the model should not include loss experience associated withthe weather-based sub-model. Doing so could cause distortions in the modeled results by double counting such losses when determining relativities or loss loads in the filed rating plan. For example, redundant losses in the data may occur when non-hurricane wind losses are included in the data whilealso using a severe convective storm model in the actuarial indication. Such redundancy may also occur with the inclusion of fluvial or pluvial floodlosses when using a flood model, inclusion of freeze losses when using a winter storm model orincluding demand surge caused by any catastrophic event.


4/6/19




A.2.d

If using output of any scoring algorithms, provide a list of the variables used to determine the score and provide the source of the data used to calculate the score.

EssentialAny sub-model should be reviewed in the same manner as the primary model that uses the sub-model’s output as input.

A.2.eWas the sub-model previously approved (or accepted) by the regulatory agency?

Essential If the sub-model was previously approved, that may change the extent of the sub-model’s review.

3. Adjustments and Scrubbing

A.3.a Provide pre-scrubbed data distributions for each input. May Be Requested Compare these distributions to A.3.g

A.3.b

Provide the percentage of exposures and premium for missing information from the model data by category.How was missing data handled?

Essential

A.3.c If duplicate records exist, how were they handled? Essential

A.3.d

Were any data outliers identified and subsequently adjusted? Name the outliers and explain the adjustments made to these outliers.

Essential

A.3.e

Were premium, exposure, loss or expense data adjusted (e.g., developed, trended, adjusted for catastrophe experience or capped) and, if so, how? Do the adjustments vary for different segments of the data and, if so, what are the segments and how was the data adjusted?

Essential

Look for anomalies in the data that should be addressed. For example, is there an extreme loss event in the data? If other processes were used to load rates for specific loss events, those losses should be removed from the input data, e.g., large losses, flood, hurricane or severe convective storm models for PPA comprehensive or homeowners’ loss.

A.3.f

What adjustments were made to raw data, e.g., transformations, binning and/or categorizations? If so, name the characteristic/variable and describe the adjustment.

Essential

A.3.g Provide post-scrubbed data distributions for each input. May Be Requested Compare these distributions to A.3.a

4. Data Organization

A.4.a

Document the method of organization for compiling data, including procedures to merge data from different sources and a description of any preliminary analyses, data checks, and logical tests performed on the data and the results of those tests.

Essential This should explain how data from separate sources was merged.


4/6/19




A.4.b

Document the process for reviewing the appropriateness, reasonableness, consistency and comprehensiveness of the data, including a justification of why the data makes sense.

Essential

For example, if by-peril modeling is performed, the documentation should be for each peril and make intuitive sense. For example, if “murder” or “theft” rates are used to predict the wind peril, provide support and a logical explanation.

A.4.c

Disclose material findings from the data review and identify any potential material limitations, defects, bias or unresolved concerns found or believed to exist in the data.

Essential

A.4.dFor any errors or material limitations in the data, explain how they were corrected.

Essential

5. Final Data Information

A.5.aIf the raw data selected to build the model is in a format that can be made available to the regulator, provide it.

May Be Requested

Commented [LKW4]: Please provide more explanation as to what is meant by these overarching terms.


4/6/19




B. Building the Model

Information



"May Be Requested"

Comments

1. High-Level Narrative for Building the Model

B.1.a

Identify the type of model (e.g. Generalized Linear Model – GLM, decision tree, Bayesian Generalized Linear Model, Gradient-Boosting Machine, neural network, etc.), describe its role in the rating systemand provide the reasons why that type of model is an appropriate choice for that role.

Essential If by-peril or by-coverage modeling is used, the explanation should be by-peril/coverage.

B.1.bA description of why the model (using the variables included in it) is appropriate for the line of business.

EssentialIf by-peril, by-form or by-coverage modeling is

used, the explanation should be by-peril/coverage/form.

B.1.c

Describe the model review process, from initial concept to final model. Keep this in overview narrative mode, less than 3 pages.

Essential

B.1.d

Describe whether loss ratio, pure premium or frequency/severity analyses was performed and, if separate frequency/severity modeling was performed, how pure premiums were determined.

Essential

B.1.e What is the model’s target variable? Essential A clear description of the target variable is key to understanding the purpose of the model.

B.1.f Provide a detailed description of the variable selection process. Essential

B.1.g

Was input data segmented in any way, e.g., was modeling performed on a by-coverage or by-peril basis or by-form? Explain the form of data segmentation and the reasons for data segmentation.

Essential The regulator would use this to follow the logic of the modeling process.

B.1.h

Describe any limitations or concerns in the analysis resulting from data issues and discuss the resulting impact on the modeling results.

Essential


4/6/19




B.1.i

How was data credibility (or lack thereof) was accounted for in the model building? How did the company determine the granularity of the rating variables?

Essential

Adjustments may be needed given models do not explicitly consider the credibility of the input data or the model’s resulting output; models take input data at face value and assume 100% credibility when producing modeled output.

2. Medium-Level Narrative for Building the Model

B.2.a

Describe any judgment used throughout the modeling process.Disclose assumptions used in constructing the model and provide support for these assumptions.

Essential

B.2.b

If post-model adjustments were made to the data and the model was rerun, explain the details and the rationale. It is not necessary to discuss each iteration of adding and subtracting variables, but the regulator should be provided with a general description of how that was done, including any measures relied upon.

Essential Evaluate the addition or removal of variables and the model fitting.

B.2.c

Describe the univariate testing and balancing that was performed during the model-building process, including a verbal summary of the thought processes involved.

Essential Further elaboration from B.2.b.

B.2.d

Describe the 2-way testing and balancing that was performed during the model-building process, including a verbal summary of the thought processes of including (or not including) interaction terms.

Essential Further elaboration from B.2.a and B.2.b.

B.2.e

For the GLM, what was the link function used? What distribution was used for the model (e.g., Poisson, Gaussian, log-normal, Tweedie)? Explain why the particular link function and distribution was were chosen. Provide the formulas for the distribution and link functions, including specific numerical parameters of the distribution.

Essential

B.2.f Were there data situations where GLM weights were used? Describe these. May Be Requested Investigate whether identical records were

combined to build the model.


4/6/19




3. Predictor Variables

B.3.a

Provide a complete data dictionary, including the names, descriptions and uses of each predictor variable, offset variable, control variable, proxy variable, geographic variable, geodemographic variable and all other variables in the model(including sub-models and external models); explanations should not use programming language or code.

Essential

B.3.a.i

Provide a list of predictor variables considered but not used in the final model, and the rationale for their removal.

May Be Requested

The rationale for this requirement is to identify variables that the company finds to be predictive but ultimately may reject for reasons other than loss-cost considerations (e.g., price optimization)

B.3.b

For each predictor variable within each model or sub-model, state whether the variable is continuous, discrete or Boolean.

Essential

B.3.b.iProvide a correlation matrix for all predictor variables included in the model and sub-model(s).

Essential

While GLMs accommodate collinearity, the correlation matrix provides more information about the magnitude of correlation between variables.

B.3.c

Provide an intuitive argument for why an increase in each predictor variable should increase or decrease frequency, severity, loss costs, expenses, or any element or characteristic whatever is being predicted.

Essential

B.3.d

If the modeler used a Principal Component Analysis (PCA) approach, provide a narrative about that process, explain why PCA was used, and describe the step-by-step process used to transform observations (usually correlated) into a set of linearly uncorrelated variables. Include a listing of the PCA variable and its principal components.

Essential


4/6/19




4. Massaging Data, Model Validation and Goodness-of-Fit Measures

B.4.a

Provide a description of how the available raw data was divided between model development(training), test, and validation datasets. Describe all circumstances under which the testing and validation datasets were accessed.

Essential

B.4.b

Describe the methods used to assess the statistical significance/goodness of the fit of the model, such as lift charts and statistical tests. Disclose whether the results are based on testing data, validation data and holdout samples. Ensure that the assessment includes model projection results compared to historical actual results to verify that modeled results bear a reasonable relationship to actual results. Discuss the results.

Essential

Some states require state-only data to test the plan, especially for analysis where using the state-only data contradicts the countrywide results. State-only data might be more applicable but could also be impacted by low credibility for some segments of risk.

B.4.c

Describe any adjustments that were made in the data with respect to scaling for discrete variables or binning the data.

Essential

B.4.d Describe any transformations made for continuous variables. Essential

B.4.e

For each discrete variable level, provide the parameter value, confidence intervals, chi-square tests, p-values and any other relevant and material tests. Were model development data, validation data, test data or other data used for these tests?

Essential

Typical p-values greater than 5% are large and should be questioned. Reasonable business judgment can sometimes provide legitimate support for high p-values. Reasonableness of the p-value threshold could also vary depending on the context of the model, e.g., the threshold might be lower when many candidate variables were evaluated for inclusion in the model.

B.4.f

Identify the threshold for statistical significance and explain why it was selected. Provide a verbal defense for keeping the variable for each discrete variable level where the p-values were not less than the chosen threshold.

Essential See Comment for B.4.e.

B.4.g

For overall discrete variables, provide type 3 chi-square tests, p-values, F tests and any other relevant and material test. Were model development data, validation data, test data or other data used for these tests?



4/6/19




B.4.h

For continuous variables, provide confidence intervals, chi-square tests, p-values and any other relevant and material test. Were model development data, validation data, test data or other data used for these tests?


B.4.i Describe how the model was tested for stability over time. Essential

Evaluate the build/test/validation datasets for potential model distortions (e.g., a winter storm in year 3 of 5 can distort the model in both thetesting and validation datasets).

B.4.jDescribe how the model was tested for geographic stability, e.g., across states or territories within state.

Essential Evaluate the geographic splits for potential model distortions.

B.4.kDescribe how overfitting was addressed and the results of correlation tests.

Essential

B.4.l

Provide support demonstrating that the GLM assumptions are appropriate (for example, the choice of error distribution).

Essential Visual review of plots of actual errors is usually sufficient.

B.4.m

Provide the formula relationship between the data and the model outputs, with a definition of each model input and output. Provide all necessary coefficients to evaluate the predicted value for any real or hypothetical set of inputs.

EssentialB.4.l and B.4.m will show the mathematical functions involved and could be used to reproduce some model predictions.

B.4.nProvide 5-10 sample records and the output of the model for those records.

Essential

5. “Old Model” Versus “New Model”

B.5.a

Provide aAn explanation of why this model is better than the one it is replacing. How was that conclusion formed? What metrics were relied on for measurement?

EssentialRegulators should expect to see improvement in the new class plan’s predictive ability or other sufficient reason for the change.

B.5.bWere 2 two Gini coefficients compared? What was the conclusion drawn from this comparison?

May Be Requested One example of a comparison might be sufficient.

B.5.cWere double lift charts analyzed? What was the conclusion drawn from this analysis?

Essential One example of a comparison might be sufficient.

B.5.dProvide a list of all new predictor variables in the model that were not in the prior model.

EssentialUseful to differentiate between old and new variables so the regulator can prioritize more time on factors not yet reviewed.


4/6/19




B.5.e

Provide a list of predictor variables used in the old model that are not used in the new model. Provide a detailed explanation of Why why were they were dropped from the new model?.

Essential

6. Modeler/Software

B.6.a

Provide the names, contact emails, phone numbers and qualifications of the key persons who:

a. Led the projectb. Compiled the datac. Built the modeld. Performed peer review

Essential

B.6.b

What software was used? Provide the name of the software vendervendor/developer, software product and a software version reference.

Essential

B.6.cWhen did work to build the model begin and when was the model build finalized?

Essential

Commented [WL5]: What is the rationale of this request? Suggest adding that rationale in the Comments section.


4/6/19




C. The Filed Rating Plan

Information



"May Be Requested"

Comments

1. General Impact of Model on Rating Algorithm

C.1.a

In the Actuarial Memorandum section on the SERFF Supporting Documentation tab, for each model and sub-model (including external models) relied upon, include a document that explains the model and its role in the rating system.

Essential

This item becomes “Essential” if the role of the model cannot be immediately discerned by the reviewer from a quick review of the rate and/or rule pages. (Importance is dependent on state requirements and ease of identification by the first layer of review and escalation to the appropriate review staff.)

C.1.bProvide an explanation of how the model was used to adjust the rating algorithm.

Essential

C.1.c

Provide a complete list of all characteristics/variables used in the proposed rating plan, including those used as input to the model (including sub-models and composite variables) and all other characteristics/variables used to calculate a premium. For each characteristic/variable, indicate if it is only input to the model, whether it is only a separate univariate rating characteristic, or whether it is both input to the model and a separate univariate rating characteristic. The list should provide transparent descriptions of each listed characteristic/variable.

Essential

Examples of variables used as inputs to the model and used as separate univariate rating characteristics might be criteria used to determine a rating tier or household composite characteristic.

C.1.d

For each characteristic/variable used as both input to the model (including sub-models and composite variables) and as a separate univariate rating characteristic, explain how these are tempered or adjusted to account for possible overlap or redundancy in what the characteristic/variable measures.

Essential

Modeling loss ratio with these characteristics/variables as control variables would account for possible overlap. The insurer should address this possibility or other considerations, e.g., tier placement models often use risk characteristics/variables that are also used elsewhere in the rating plan.


4/6/19




C.1.e

If the filing support includes an update or replacement of an existing model, identify and explain the changes in calculations, assumptions, parameters and data used to build the models. Provide an explanation of why the updated/replacement model is better than the one it is replacing, including, how that conclusion was reached, and the metrics relied upon to reach that conclusion.

Essential

2. Relevance of Variables / Relationship to Risk of Loss

C.2.a

Provide an explanation how the characteristics/rating variables,included in the filed rating plan,logically and intuitively relate to the risk of insurance loss (or expense) for the type of insurance product being priced. Include a discussion of the relevance each characteristic/rating variable has on consumer behavior that would lead to a difference in risk of loss (or expense).

EssentialThis explanation would not be needed if the connection between variables and risk of loss (or expense) has already been illustrated.

3. Comparison of Model Outputs to Current and Selected Rating Factors

C.3.a

Provide a comparison between relativities indicated by the model toboth current relativities and the insurer's selected relativities foreach risk characteristic/variable inthe rating plan. Each significant difference should be highlighted and explained.

Essential

“Significant difference” may vary based on the risk characteristic/variable and context. However, the movement of a selected relativity should be in the direction of the indicated relativity; if not, an explanation is necessary as to why the movement is logical.

C.3.b

What calculations, judgments and adjustments, if any, were made before using the model output in the rating system? Identify any adjustments that were made to the indicated indications produced by the model to derive the selected selections, modeland provide an explanation for the necessity of such adjustments.

Essential


4/6/19




C.3.c

If the model results in scores, tiers, or ranges of values for which indications are then derived for each such resulting category, provide explanations for filed rating values that deviate from these indicationsand supporting information/analyses. For example, identify any adjustments that were made to the factors indicated for each category of the model outputs to derive the factors selected for the rating plan.

EssentialThis is especially important if deviations are material and/or impact one consumer population more than another.

4. Responses to Data, Credibility and Granularity Issues

C.4.a What consideration was given to the credibility of the output data? Essential

At what level of granularity is credibility applied. If modeling was by-coverage, by-form or by-peril, explain how these were handled when there was not enough credible data by coverage, formor peril to model.

C.4.bIf applicable, discuss the rationale for using a model that is more granular than the rating plan.

EssentialThis is applicable if the insurer had to combine modeled output in order to reduce the granularity of the rating plan.

C.4.cIf applicable, discuss the rationale for using a rating plan that is more granular than modeled output.

Essential

A more granular rating plan implies that the insurer had to extrapolate certain rating treatments, especially at the tails of a distribution of attributes, in a manner not specified by the model indications.

5. Definitions of Rating Variables

C.5.a

Provide a transparent presentation and explanation of binning decisions that assign ranges of model outputs to particular rating categories.

Essential

C.5.b

Provide complete definitions of any rating tiers or other intermediate rating categories that translate the model outputs into some other structure that is then presented within the rate and/or rule pages.

Essential

Commented [WL6]: Confusing and difficult to read. Suggest rewrite.


4/6/19




6. Supporting Data

C.6.a

Provide state-specific, book-of-business-specific univariate historical experience data,separately for each year included in the model, consisting of, at minimum, earned exposures, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output(s) proposed to be used within the rating plan. For each data element, explain whether it is raw or adjusted, and if the latter, provide detailed explanation for the adjustments.

EssentialFor example, were losses developed/undeveloped, trended/untrended,capped/uncapped, etc.

C.6.b

Provide an explanation of any material (especially directional) differences between model indications and state-specific univariate indications.

Essential

Multivariate indications may be reasonable as refinements to univariate indications, but likely not for bringing about reversals of those indications. For instance, if the univariate indicated relativity for an attribute is 1.5 and the multivariate indicated relativity is 1.25, this is potentially a plausible application of the multivariate techniques. If, however, the univariate indicated relativity is 0.7 and the multivariate indicated relativity is 1.25, a regulator may question whether the attribute in question is negatively correlated with other determinants of risk. Credibility of state data should be considered when state indications differ from modeled results based on a broader data set. However, the relevance of the broader data set to the risks being priced should also be considered.

7. Consumer Impacts

C.7.aIdentify model changes and rating variables that will cause large premium disruptions.

Essential

C.7.b

Did the insurer perform sensitivity testing to identify significant changes in premium due to small or incremental change in a single risk characteristic? If so, discuss and provide the results of that testing.

May Be Requested

One way to see sensitivity is to analyze a graph of each risk characteristic’s/variable’s possible relativities. Look for significant variation between adjacent relativities and evaluate if such variation is reasonable and credible.

C.7.c

Measure and describe the impacts on expiring policies and describe the process used by management to mitigate or get become comfortable with those impacts.

Essential


4/6/19




C.7.d

Provide a rate disruption/dislocationanalysis, demonstrating the distribution of percentage impacts on renewal business (create by rerating the current book of business). Include the largest dollar and percentage impacts arising from the filing, including (desirably) the impacts arising specifically from the adoption of the model or changes to the model as they translate into the proposed rating plan.

Essential

While the default request would typically be for the distribution of impacts at the overall filing level, the regulator may need to delve into the more granular variable-specific effects of rate changes if there is concern about particular variables having extreme or disproportionate impacts, or significant impacts that have otherwise yet to be substantiated.See Appendix C for an example of a disruption analysis.

C.7.e

Provide exposure distributions for output variables and show the effects of rate changes at granular and summary levels.

Essential See Appendix C for an example of an exposure distribution.

C.7.fExplain how the insurer will help educate consumers to mitigate their risk.

Essential

C.7.g

Identify sources to be used at "point of sale" to place individual risks within the matrix of rating system classifications. How can a consumer verify their own "point-of-sale" data and correct any errors?

Essential

Could be "Essential" if the variables/ characteristics used could 1) have public-policy implications, 2) result in erroneous information being used, or 3) result in many large, disruptive premium changes at renewal. Another consideration to judge “importance” is whether consumers are proactively involved (e.g., use of consumer credit information and credit-report accuracy issues).

C.7.h

Identify rating variables that remain static over a consumer’s lifetime versus those that will be updated periodically. Document guidelines for variables that are listed as static,yet for which the underlying consumer attributes may change over time.

Essential

C.7.i

Provide the regulator with a description of how the company will respond to consumers’ inquiries about how their premium was calculated.

Essential

C.7.jProvide the regulator with a means to calculate the rate charged a consumer.

May Be RequestedEssential

Especially for a complex model or rating plan, a score or premium calculator via Excel or similar means would be ideal, but this could be elicited on a case-by-case basis. Ability to calculate the rate charged can allow the regulator to perform sensitivity testing when there are small changes to a risk characteristic/variable.

Commented [WL7]: Our concern is if this is not Essential, filers may be led to believe they can create “black boxes” creating rates that regulators would not be able to validate against filed rating plans.


4/6/19




8. Accurate Translation of Model into a Rating Plan

C.8.a

Provide sufficient information for the reviewer to be able to understand how the model outputs are used within the rating systemand to verify that the rating plan, in fact, reflects the model output and any adjustments made to the model output.

Essential

VIII. PROPOSED CHANGES TO THE PRODUCT FILING REVIEW HANDBOOK

TBD – placeholder to include best practices for review of predictive models and analytics filed by insurers to justify rates

IX. PROPOSED STATE GUIDANCE

TBD –placeholder for guidance for rate filings that are based on predictive model

X. OTHER CONSIDERATIONS

During the development of this paperguidance, a topics arose that are not addressed in this paper. These topics may need addressing during the regulator’s review of a predictive model. A few of these issues may be discussed elsewhere within NAIC. All of these issues, if addressed, will be addressed by each state on a case-by-case basis. The topics for considerationinclude:

TBD

XI. RECOMMENDATIONS GOING FORWARD

TBD


4/6/19




XII.APPENDIX A – BEST PRACTICE DEVELOPMENT

Best-practices development is a method for reviewing public policy processes that have been effective in addressing particular issues and could be applied to a current problem. This process relies on the assumptions that top performance is aresult of good practices and these practices may be adapted and emulated by others to improve results9.

The term “best practice” can be a misleading one due to the slippery nature of the word “best”. When proceeding with policy research of this kind, it may be more helpful to frame the project as a way of identifying practices or processes that have worked exceptionally well and the underlying reasons for their success. This allows for a mix-and-match approach for making recommendations that might encompass pieces of many good practices10.

Researchers have found that successful best-practice analysis projects share five common phases:

A. Scope

The focus of an effective analysis is narrow, precise and clearly articulated to stakeholders. A project with a broader focus becomes unwieldy and impractical. Furthermore, Bardach urges the importance of realistic expectations in order to avoid improperly attributing results to a best practice without taking into account internal validity problems.

B. Identify Top Performers

Identify outstanding performers in this area to partner with and learn from. In this phase, it is key to recall that a best practice is a tangible behavior or process designed to solve a problem or achieve a goal (i.e. reviewing predictive models contributes to insurance rates that are not unfairly discriminatory). Therefore, top performers are those who are particularly effective at solving a specific problem or regularly achieve desired results in the area of focus.

C. Analyze Best Practices

Once successful practices are identified, analysts will begin to observe, gather information and identify the distinctive elements that contribute to their superior performance. Bardach suggests it is important at this stage to distill the successful elements of the process down to their most essential idea. This allows for flexibility once the practice is adapted for a new organization or location.

D. Adapt

Analyze and adapt the core elements of the practice for application in a new environment. This may require changing some aspects to account for organizational or environmental differences while retaining the foundational concept or idea. This is also the time to identify potential vulnerabilities of the new practice and build in safeguards to minimize risk.

E. Implementation and evaluation

The final step is to implement the new process and carefully monitor the results. It may be necessary to make adjustments, so it is likely prudent to allow time and resources for this. Once implementation is complete, continued evaluation is important to ensure the practice remains effective.

9 Ammons, D. N. and Roenigk, D. J. 2014. Benchmarking and Interorganizational Learning in Local Government. Journal of Public Administration Research and Theory, Volume 25, Issue 1. P 309-335. https://doi.org/10.1093/jopart/muu01410 Bardach, E. and Patashnik, E. 2016. A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving. Thousand Oaks, CA. CQ Press.


4/6/19




XIII. APPENDIX B - - GLOSSARY OF TERMS

Offset vs. control factors - TBD

Probability Distribution - TBD

Exponential Family - TBD

Linear Predictor - TBD

Link Function - TBD

Univariate Model - TBD

Generalized Linear Model - TBD

Private Passenger Automobile Insurance – TBD

Homeowners Insurance – TBD

Rating algorithm – TBD

Rating plan – TBD

Rating system – TBD

PCA approach (Principal Component Analysis) – The PCA approach is also known as factor analysis. These methods create multiple new variables from correlated groups of predictors. Those new variables exhibit little or no correlation between them—thereby making them potentially more useful in a GLM. A PCA in a filing can be described as “a GLM within a GLM.” One of the more common applications of PCA is geodemographic analysis, where many attributes are used to modify territorial differentials on, for example, a census block level.

Fair Credit Reporting Act – The Fair Credit Reporting Act (FCRA), 15 U.S.C. § 1681 (FCRA) is U.S. Federal Government legislation enacted to promote the accuracy, fairness and privacy of consumer information contained in the files of consumer reporting agencies. It was intended to protect consumers from the willful and/or negligent inclusion of inaccurate informationin their credit reports. To that end, the FCRA regulates the collection, dissemination and use of consumer information, including consumer credit information.11 Together with the Fair Debt Collection Practices Act (FDCPA), the FCRA forms the foundation of consumer rights law in the United States. It was originally passed in 1970 and is enforced by the US Federal Trade Commission, the Consumer Financial Protection Bureau and private litigants.

Overfitting - TBD

Geodemographic - Geodemographic segmentation (or analysis) is a multivariate statistical classification technique for discovering whether the individuals of a population fall into different groups by making quantitative comparisons of multiplecharacteristics with the assumption that the differences within any group should be less than the differences between groups. Geodemographic segmentation is based on two principles:

1. People who live in the same neighborhood are more likely to have similar characteristics than are two people chosen at random.

2. Neighborhoods can be categorized in terms of the characteristics of the population that they contain. Any two neighborhoods can be placed in the same category, i.e., they contain similar types of people, even though they are widely separated.

Etc.

11 Dlabay, Les R.; Burrow, James L.; Brad, Brad (2009). Intro to Business. Mason, Ohio: South-Western Cengage Learning. p. 471. ISBN 978-0-538-44561-

0.


4/6/19




XIV. APPENDIX C – SAMPLE RATE-DISRUPTION TEMPLATE

NOTE: Uncapped Capped (If Applicable)

Minimum % Change Minimum % ChangeMaxmium % Change Maxmium % ChangeTotal Number of Insureds (Auto-Calculated)

1994Total Number of Insureds (Auto-Calculated)

1994

Uncapped Rate DisruptionPercent-Change Range Number of Insureds in Range Percent-Change Range Number of Insureds in Range

19

Capped Rate Disruption (If Applicable)

Template Updated October 2018State Division of Insurance - EXAMPLE for Rate Disruption

19

EXAMPLE Uncapped Rate Disruption


4/6/19




EXAMPLE Capped Rate Disruption

State Division of Insurance - EXAMPLE for Largest Percentage Increase Template Updated October 2018

Uncapped Change Uncapped Dollar Change Current Premium

Capped Change (If Applicable) Capped $ Change (If Applicable) Proposed Premium

Characteristics of Policy (Fill in Below)

Vehicle: BI Limits: PD Limits: UM/UIM Limits: MED Limits:

Attribute% Impact

(Uncapped)Dollar Impact (Uncapped)

TOTAL 15.00% $82.50

Corresponding Dollar Increase (for Insured Receiving Largest Percentage Increase)

For Auto Insurance:

At minimum, identi fy age and gender of each named insured, amount of insurance, terri tory, construction type, protection class , any prior loss his tory, and any other key attributes whose treatments are affected by this fi l ing.

Most Significant Impacts to This Policy

Largest Percentage Increase

Automobile policy: Territory:

COMP Deductible: COLL Deductible:

NOTE: as needed. Tota l percent and dol lar impacts should reconci le to the va lues presented above in this exhibi t.

What lengths of pol icy terms does the insurer offer in this book of bus iness?

Check a l l options that apply below.

12-Month Policies121212--Month PoliciesPPMonth PoliciesMonth Policies

6-Month Policies3-Month Policies

Other (SPECIFY)


4/6/19




State Division of Insurance - EXAMPLE for Largest Dollar Increase Template Updated October 2018

Uncapped Change Current Premium Uncapped Percent Change

Capped Change (If Applicable) Proposed Premium Capped % Change (If Applicable)Characteristics of Policy (Fill in Below)

Vehicle: BI Limits: PD Limits: UM/UIM Limits: MED Limits:

Attribute% Impact

(Uncapped)Dollar Impact (Uncapped)

for PD

TOTAL 12.00% $306.60

For Auto Insurance:

At minimum, identi fy age and gender of each named insured, amount of insurance, terri tory, construction type, protection class , any prior loss his tory, and any other key attributes whose treatments are affected by this fi l ing.

Corresponding Percentage Increase (for Insured Receiving Largest Dollar Increase)Largest Dollar Increase

Most Significant Impacts to This Policy

NOTE: as needed. Tota l percent and dol lar impacts should reconci le to the va lues presented above in this exhibi t.

Automobile policy: Territory:

COMP Deductible: COLL Deductible:


4/6/19




XV.APPENDIX D – INFORMATION NEEDED BY REGULATOR MAPPED INTO BEST PRACTICES

XVI. APPENDIX E – REFERENCES


4/6/19


To: NAIC Casualty Actuarial and Statistical Task Force From: Casualty Actuarial Society Machine Learning Task Force Members Date: January 15, 2019 Subject: Concerns Related to Regulatory Review of Predictive Models White Paper This purpose of this document is to raise concerns voiced by members of the Casualty Actuarial Society’s (CAS) Machine Learning Task Force (MLTF) with respect to the 10/25/2018 draft Regulatory Review of Predictive Models white paper of the NAIC’s Casualty Actuarial and Statistical Task Force. It should be noted that the MLTF does not speak on behalf of the CAS or its leadership, and within the MLTF there have been a variety of responses to the draft. This document collects some of those perspectives, which may at times be in conflict, with the intent of fully expressing our group’s concerns and considerations with respect to statistical modeling practice. Our primary concerns can be categorized as follows:

1. Scope (GLMs vs. machine learning) 2. Confidentiality of models 3. Focal point of regulation (rates vs. models) and depth of clarifying questions

Scope of White Paper The SCOPE paragraph the white paper may be excessively broad. While it first states that the scope will “focus on GLMs used to create private passenger automobile and homeowners’ insurance rating plans,” it then expands the scope universally with the statement that “Guidance and knowledge needed to review predictive models identified in this paper are, in large part, transferrable to other types of predictive models, to other lines of business, or other insurance endeavors.” The intended application to other types of models is confirmed in the High-Level Narrative for Building the Model, as item B.1.a asks to “Identify the type of model (e.g. Generalized Linear Model - GLM, decision tree, Bayesian Generalized Linear Model, Gradient-Boosting Machine, neural network, etc.)” Given that the paper addresses model review as it relates specifically to GLMs and private passenger auto with the hope that the guidance should be generalizable, it is somewhat concerning that more language in the paper was not devoted to some of the potentially significant ways that other types of models and other lines of business can deviate from the conventions of “GLM / PPA.” In some instances, these deviations can be such that questions marked as “essential” in this document may be extremely onerous, extremely difficult if not impossible to answer, misleading, or lacking in usefulness in other lines and for other types of


4/6/19


models. For this reason, we would suggest adding additional cautionary language (or revise the SCOPE section) to avoid potential misunderstandings. As an example of these issues, consider private passenger automobile coverage. Coverage is widely available and actuaries are accessing increasingly personal data to develop more refined GLM rating plans. The regulatory issues of personal privacy, fair credit reporting, and discrimination rise to the surface. Contrast this with the homeowners’ line, which faces a climate-driven coverage crisis. Measuring contagion risk has become as important as measuring individual risk. New data sources include vegetative fuel loads, topography, and weather data. Homeowners actuaries have to deal with extremely high severities and poorly fitting existing models. Innovation will be needed to maintain a functioning voluntary market. For these reasons, we recommend that the Task Force should limit the scope of the recommendations, provide up-front caveats regarding the use of the term “essential” to describe various data request items, and provide some additional discussion regarding the ways that rate filing review may differ depending on the model and line of business.

Confidentiality of Models The creation of innovative pricing models requires a major investment on the part of insurance companies. They make these investments in the hope of achieving a sustainable competitive advantage and consider these models as intellectual property. Public access to the inner workings of proprietary pricing models and company data acts as a serious deterrent to the investment in new models. The Task Force white paper acknowledges the importance of confidentiality, but offers no protections beyond what currently exists in state laws. It puts the burden on the insurers “to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filings.” We are concerned with the fact that the white paper offers no new protections to balance against the additional information requirements that will result from the proposed best practices. We would recommend that the Task Force add a column to the best practices table to indicate which items are likely to require confidentiality.


4/6/19


Focus of Regulation and Depth of Questions Members of our group expressed concern about the focus of the white paper in a number of complementary ways, in part based on their own experiences with the rate filing process. Our members have been involved in preparing rate filings and responding to questions from regulatory officials, providing reviews of models on behalf of insurers, and participating directly in the rate filing review process on behalf of regulators. Several members expressed concern that the best practices recommended in the white paper had the implied intent of allowing the regulator to build the model, themselves, rather than being designed for the purpose of reviewing the models. For these members, the statistical and data reporting required for a model review should be much less onerous than the requirements for creating a model from scratch. To that end, they voiced concern as follows:

“State insurance regulators are tasked with enforcing laws governing insurance rates, not the models that company management may have consulted during the pricing process. Internal decisions about credibility, binning, granularity, and limitation of rate disruption have long been part of the pricing process, even during the historical period of univariate pricing. The internal company pricing process has not typically been required in routine rate filings, but is typically requested only on an exception basis in the case of large rate changes. Please do not create best practices where the internal decision making process becomes “essential” in each filing. While regulators may feel that they need to understand the underlying models in order to understand the rating plans, it is possible to consider a change rating plan without an understanding of the technical elements of the underlying model. We submit that a rate change or rating plan could be judged to be acceptable if it is more efficient than the current rates, non-discriminatory, limits disruption, and is certified by an actuary. This approach puts the focus of regulation of the rating plan rather than the model.”

Others of our group took the view that often, significant technical detail is required for a thorough vetting of a model, especially in cases where particular variables, modeling decisions, or techniques used by the modeler are new or unusual in that line of business. In such cases the questions suggested by the white paper may, in fact, be inadequate to fully address the question of whether a decision produces rates that are not unfairly discriminatory. However, in the majority of models, such depth of questioning is not warranted. Questions of significant depth are only requested if a high-level review indicates that such questions are required. So, on the one hand, it is important that filers should be aware that such difficult questions are possibilities, particularly if there are unusual modeling decisions, because this


4/6/19


knowledge may allow them to prepare filings with forethought that makes answering such questions much less onerous. On the other hand, we are concerned that it is not necessary to ask such onerous questions in every instance. More to the point, there is some concern listing the majority of questions as “essential” and asking many questions for every single filing may have the unwanted effect of replacing careful statistical consideration with a “data-driven approach” to regulation. The risk of such an approach is that models may pass standard metrics of performance while failing badly at less-standard-but-more-highly-relevant measures of e.g. predictiveness. Such models would not be easily identified by a data-driven approach, and for this reason a data-driven approach to regulation is no substitute for careful analysis. We therefore recommend that significant consideration be given to whether each of the questions is “essential” if it is marked as such. Perhaps additional categories, such as “essential in certain circumstances” could provide a more nuanced view, allowing sufficient leeway for standard modeling approaches that are within commonly accepted bounds while providing adequate guidance regarding those times when it is essential to ask additional questions. In addition, given the complexity of many machine learning techniques relative to GLMs, many of the recommended questions may be impractical or impossible to answer. Further consideration must be given to the extent to which models, themselves, can be effectively regulated in such cases, or whether regulatory focus should (within reason) be on the resultant rates rather than the models that produced them.

Summary of Recommended Changes 1. Limit the scope of the recommendations provided in the white paper, 2. Provide up-front caveats regarding the use of the term “essential” to describe various

data request items, 3. Provide some additional discussion regarding the ways that rate filing review may differ

depending on the model and line of business, 4. Add a column to the best practices table to indicate which items are likely to require

confidentiality, 5. Significant consideration should be given to whether each of the questions is “essential”

if it is marked as such. Additional categories, such as “essential in certain circumstances” could provide a more nuanced view, allowing sufficient leeway for standard modeling approaches that are within commonly accepted bounds while providing adequate guidance regarding those times when it is essential to ask additional questions, and


4/6/19


6. Further consideration must be given to the extent to which models, themselves, can be effectively regulated in such cases, or whether regulatory focus should (within reason) be on the resultant rates rather than the models that produced them.


4/6/19


1

TO: NAIC Casualty Actuarial & Statistical (C) Task Force FROM: The Cincinnati Insurance Companies SUBJECT: Comments on Regulatory Review of Predictive Models White Paper DATE: January 22, 2019

Thank you for this opportunity to comment on the Casualty Actuarial & Statistical (C) Task Force (CASTF) white paper on best practices for the regulatory review of predictive models, which the CASTF voted to expose for public comment during its November 15, 2018 meeting held in conjunction with the NAIC Fall Meeting in San Francisco. INTRODUCTION. The use of predictive analytics has the potential to create significant benefits for both consumers and insurers by dramatically improving pricing precision, thereby promoting a more refined and balanced insurance transaction and transforming the insurer-consumer experience into a more meaningful relationship. Predictive analytics can reveal insights into consumer behavior, lower the cost of insurance for many, and provide tools for the consumer to better control and mitigate loss. Given these benefits, it is important that regulators have the necessary tools to effectively review an insurer’s use of predictive models in insurance applications.

With the forgoing precepts in mind, the CASTF was charged with identifying best practices to serve as a guide to state insurance departments in their review of predictive models underlying rating plans. Specifically, the CASTF was given two charges by the NAIC Property and Casualty Insurance (C) Committee at the request of the NAIC Big Data (EX) Working Group:

1) Draft and propose changes to the Product Filing Review Handbook to include best practices for review of predictive models and analytics filed by insurers to justify rates.

2) Draft and propose state guidance (e.g., information, data) for rate filings that are based on complex predictive models.

The initial step in this process has been completed: the drafting of the CASTF white paper to identify best practices for reviewing predictive models and analytics filed by insurers with regulators to justify rates and provide state guidance for review of rate filings based on predictive models.1 We now offer our comments on the CASTF draft white paper.

1 Upon adoption of the CASTF white paper by the NAIC Executive (EX) Committee and Plenary, the CASTF will evaluate how to incorporate the best practices identified in the white paper into the Product Filing Review Handbook and will recommend such changes to the NAIC Speed to Market (EX) Working Group.


4/6/19


2

Please note at the outset that we will be suggesting that the CASTF abandon its rules-based approach in favor of a principles-based approach for identifying best practices for regulatory review of predictive models. THE WHITE PAPER IS RULES-BASED AND OVERLY PRESCRIPTIVE. The heart of the 27-page white paper is a fifteen-page section of rules for the regulatory review of predictive models. As explained in the white paper, this section of the document identifies the information a regulator may need to review a predictive model used by an insurer to support a filed P/C insurance rating plan. As the drafters of the white paper acknowledge, the list of 94 prescriptive rules is “lengthy but not exhaustive” and may “seem long.”2 These conclusions inform our own assessment of the white paper: it is far too prescriptive and complicated and would further seem inconsistent with the goal of providing clear, meaningful guidelines for both modeler and regulator. The document is also at odds with what the regulated community would expect from a set of “best practices” – a set of consumer-oriented core guiding principles designed to ensure that an insurer’s predictive models are fairly constructed, not unfairly discriminatory and applied in ways that will sharpen an insurer’s ability to adequate segment and price risks, thus creating significant benefits for both consumers and insurers. Here are just a few of the concerns our actuaries have identified from their review of the CASTF’s rules-based white paper:

1. Many of the 94 rules included in the white paper have overlapping goals. These rules should be collapsed into a set of core principles instead of a series of detailed and prescriptive rules. For example, there are a variety of rules that discuss methods for determining whether a given variable should be included in a model. A better approach would be the pronouncement of a general principle for variable inclusion.

2. Some of the rules deemed “essential” are arguably too specific to be applicable in many cases or go

beyond the scope of necessary details to understand the model (e.g. PCA breakdown discussion, GLM weights, verbose description of all interaction testing).

3. The document seems focused on generalized linear models (GLMs) when methods of performing analysis are constantly evolving. For example, would a regulator actually want all of the “if” statements that would be generated by a random forest model? In addition, many newer models do not produce diagnostics such as p-values. Elastic net models, which are similar to traditional GLMs, have p-values that can be calculated, but these p-values do not represent statistical confidence as they do for a GLM.

4. Producing documentation for all the rules deemed "essential" would result in a very large volume of documentation. Would most regulators be able to properly review the generated model documentation? Proper review of this material would require regulators to allocate full-time staff to this task.

2 The 94 rules are described in a multi-paged table and are organized under 8 subject headings. Each of the rules are numbered using a three-character alphanumeric sequence: upper case letter, number, lower case letter (example: A.1.a.).


4/6/19


3

5. The 94 rules in the white paper would create an undue burden on modeling teams. We estimate this would increase our model development workload by roughly 33%. That is, an additional modeler would be required for every three current modelers.

6. The white paper’s rules-based approach, which imposes 94 prescriptive modeling rules on insurers, would reduce competition in the marketplace by increasing the amount of effort required to build and file models. It would also increase reliance on consultants and discourage insurers from developing internal models. It would also result in increased expense ratios and reduced innovation.

7. Many of the rules seem to suggest that an insurer should document and explain paths not taken. We think that the focus of regulatory review should be on the proposed model, as opposed to what is not being proposed.

For a more detailed list of our concerns with the 94 prescriptive rules, please see Exhibit A, submitted with this comment letter. We also share the concerns raised by our national P&C trade association, the American Property Casualty Insurance Association (APCI), in its comment letter detailing the many flaws in the 94 prescriptive rules. THE CASTF SHOULD ADOPT A PRINCIPLES-BASED APPROACH FOR REVIEW OF PREDICTIVE MODELS. We believe that a principles-based approach, premised upon a set of core guiding principles, would provide regulators with standards against which they could effectively review an insurer’s use of predictive models in insurance applications and in a timely fashion. We also believe that the regulatory review of predictive models should be more focused on preventing harm to consumers than on explaining how a predictive model functions. The principles-based approach we envision would achieve this balance by taking into consideration core guiding principles which would ensure that modelers follow a set of markers that would guard against unfair discrimination. Here are some examples of the types of core guiding principles which we believe are more focused on consumers and are worthy of the CASTF’s consideration:

1. Ensure that a predictive model does not promote, encourage or permit unfair/improper discrimination.

2. Ensure that a predictive model does not promote, encourage or permit improper strategy (price optimization, for example).

3. Ensure that the covariates and model predictions included in a model bear a reasonable resemblance to the subject matter being modeled.

4. Require adoption of internal company controls designed to periodically review all predictive models for violations of the core principles described above.

We therefore propose that the CASTF issue a call to regulators, the industry and other interested parties to submit proposed core principles for regulatory review of predictive models to the CASTF in advance of the NAIC Spring National Meeting in Orlando (April 6-9, 2019), and that the CASTF hold a discussion or hearing at the Orlando meeting on the question of whether it would be more effective and appropriate to employ a principles-based approach for regulatory review of predictive model than the current rules-based approach now under consideration.


4/6/19


4

THE NAIC HAS ALREADY EMBRACED PRINCIPLES-BASED REGULATION. The NAIC is no stranger to principles-based insurance regulation, which it embraced in 2017 to modernize the regulation of life insurance reserve calculations. As life insurance products increased in their complexity and companies developed new and innovative life insurance product designs that changed their risk profile, it was necessary to develop new reserve valuation methodologies to address these changes. These needs led the NAIC to replace its rules-based reserve calculation model with a principles-based valuation model. As the NAIC’s Center for Insurance Policy and Research explains in its “Key Issue” paper on principle-based reserving:

“Prior to PBR, static formulas and assumptions were used to determine these reserves as prescribed by state laws and regulations. However, sometimes this rule-based approach leaves an insurer with excessive reserves for certain insurance products and inadequate reserves for others. The solution is to "right-size" reserve calculations by replacing a rule-based approach with a principle-based approach.”3

Likewise, we believe that a principles-based approach would work better for regulatory review of predictive models. Using a set of core guiding principles instead of a prescriptive set of rules would “right size” predictive model regulation and avoid excessive regulation of new and innovative products. CLOSING COMMENTS. Thank you for considering our comments. We encourage the CASTF to consider the benefits of taking a principles-based approach to predictive model regulation and to take the measures we suggested above to take up the issue at the NAIC Spring Meeting in April 2019. Respectfully submitted,

Teresa Cracas Senior Vice President Chief Risk Officer

Luyang Fu Vice President Pricing & Predictive Analytics

Xiangfei Zeng Vice President Personal Pricing & Modeling

Thomas Hogan Vice President Corporate Counsel

Scott Gilliam Vice President Government Relations

Please direct all inquiries and follow up to Scott Gilliam, Vice President—Government Relations

Office: 513-870-2811 | Mobile: 513-607-5717 | Email: [email protected]

3 See “KEY ISSUE: The National System of State Regulation and Principle-Based Reserving, available online at https://naic.org/cipr_topics/principle_based_reserving_pbr.htm.


4/6/19


EXHIBIT A

NAIC Casualty Actuarial & Statistical (C) Task Force—White Paper on Regulatory Review of Predictive Models

Section VII. PREDICTIVE MODELS – INFORMATION FOR REGULATORY REVIEW

***Additional Concerns of The Cincinnati Insurance Companies***

General Comment. Requiring a company to state the data type of covariates should only be required when it affects model output and is necessary for model review. Often, the key information conveyed by data types are obvious from model structure/output. In other cases, the data type isn't strictly required for evaluation of the model. For example, many tree-based models treat numeric columns as an ordered categorical in the sense that the tree still chooses discrete split points for the numeric column. Rule B.2.e. The rule states that the formula for the distribution and link functions should be provided. We do not think providing statistical formulas necessarily aids in the review process. For commonly used distributions/models a reference should be sufficient.

Rule A.3.d. What purpose would be served by listing outliers? A better alternative would be to just describe the process for handling outliers. For example, loss capping is commonly applied. It would be unrealistic to list every record where a loss was capped. Rule A.4.a. Describing the process of how to join data seems unnecessary. Rule B.1.i. This rule provides: "models take input data at face value and assume 100% credibility." This is not true for all models. GLMs do treat categorical covariates as fully credible, but models such as elastic net and random forest do not. In addition, many implemented credibility procedures do not have rigorous foundations for the standard and credibility standards are often judgmental selected. Furthermore, selected complements often do not fall into a classical credibility framework. Rule B.3.c. Intuitive arguments should not be required. The question is whether the variables are predictive and not unfairly discriminatory. Rule B.3.d. Why is a narrative required to explain the reason PCA was used? PCA is a common technique that is well known for producing uncorrelated versions of the covariates. This property is even stated in the document. Furthermore, PCA is only one of many data transformations and we do not think it requires special treatment. Rule B.4.b. The rule provides: "Ensure that the assessment includes model projection results compared to historical actual results to verify that the modeled results bear a reasonable relationship to actual results." Shouldn't a model built on historical data bear a strong resemblance to the data? Or does this mean historical results of a prior model? Ideally, a new model would show somewhat different behavior as the new model would hopefully perform better as compared to the historical model. Rules B.4.e, B.4.g, and B.4.h. Many models such as elastic nets, neural networks, and tree-based models do not produce p-values or similar statistics. Rule B.4.m. Providing all formulas/coefficients is unrealistic for some models, e.g., random forest and deep neural networks. Technically all formulas/coefficients are able to be provided but would not be comprehensible or meaningful for model review.

EXHIBIT A


4/6/19


January 14, 2019

NAIC Casualty Actuarial and Statistical Task Force Attn: Kris DeFrain, FCAS, MAAA, CPCU

Via Email – [email protected]

Re: Comments on the Draft White Paper – Best Practices – Regulatory Review of Predictive Models

Dear Ms. DeFrain:

Fair Isaac Corporation (FICO®) is pleased to provide our comments on the recently released draft white paper, Best Practices – Regulatory Review of Predictive Models.

FICO is an independent analytics provider, not a data company, that is dependent on other firms (e.g., credit reporting agencies, insurance companies, lenders) to provide the appropriate and necessary data for our analysis and predictive model development. With a focus on innovation that effectively rewards all parties – insurers, lenders, and consumers alike – FICO is recognized as the pioneer in developing the algorithms and underlying analytics used to produce credit scores, credit-based insurance scores, and other risk management scores. FICO fully understands the value of regulatory scrutiny and the need for regulatory flexibility to help ensure that consumers continue to benefit from these scores by enjoying quick, fair access to credit and to more affordable insurance. In previous years, access to affordable insurance involved a lengthy decision process based, in some cases, on subjective and inconsistent underwriting factors.

In 1993, FICO introduced the first commercially-available credit-based insurance scores to all US insurers as an additional risk segmentation factor that could be used in their private passenger auto and home insurance underwriting and pricing programs. On behalf of several hundred FICO® Insurance Score clients, over these past 25 years we’ve met with state departments of insurance and have testified before dozens of state legislative committees. Our goal in each of these interactions was to provide regulatory support for our clients’ use of FICO Insurance Scores by answering all questions to the best of our ability and by offering as much insight into FICO’s proprietary modeling analytics and technologies as possible.

For nearly two decades, in support of rate filings throughout the nation by our FICO® Insurance Score clients, FICO has provided model documentation—specific consumer credit characteristics, attributes and weights for the filed model—as well as reason code/factor definitions, and a general discussion of our model development process to all requesting departments of insurance able to provide the necessary protections. In addition, FICO has modified our insurance score models as required by those states with either statutory or regulatory mandates. FICO also offers an insurance score educational website (insurancescores.fico.com) that has been accessed by consumers, regulators, legislators, insurers, agents and other interested parties throughout the nation.

It is FICO’s hope that our work over these past two decades with state departments of insurance on behalf of our clients will be fully recognized and allowed to continue unabated – grandfathered under the


4/6/19


current regulatory review process, as it were, and not impacted by undue burdens on FICO and our hundreds of clients.

Kris DeFrain, FCAS, MAAA, CPCU January 14, 2019 Page 2

Having shared a bit of FICO’s background and our FICO® Insurance Score client support strategies, the remainder of our comments will focus on our Scores business model and the negative implications the recommendations within the draft white paper could have on our Scores business.

The intellectual property underlying much of our predictive modeling and analytics technology has been developed by FICO data scientists over the past six decades. This development work has taken an enormous amount of time, money, research, know-how, and testing. The algorithms used in the various models cut across multiple products and solutions. For example, much of the technology and intellectual property that is used as part of the models and algorithms for FICO’s credit risk scores is also used as part of our other scoring models, including our credit-based insurance scores – FICO® Insurance Scores.

While some of FICO’s intellectual property relating to its scoring models and algorithms is protected by patents, the specifications that describe how the scoring models work and other core parts of the algorithms are FICO’s carefully guarded trade secrets. The continued success of FICO’s Scores business depends on the maintenance of this confidential and proprietary information and these trade secrets—including all non-public aspects of our current and future insurance scoring models.

FICO’s scoring-related trade secrets have substantial independent economic value to the company precisely because they are not generally known by others, including any potential competitors, that could unfairly obtain economic value from their disclosure or use. The value in FICO’s scoring trade secrets and proprietary methods for scoring would be put at risk if the company were required to disclose that information without full and continuing confidentiality. The unique scoring formula intellectual property assets, which are securely guarded and protected by contract and law, are paramount to an independent analytics provider such as FICO. Forcing disclosure of these intellectual property assets would dissipate the value of these assets.

Given the necessary protection of FICO’s intellectual property and trade secrets, our belief is that the depth and breadth of the regulatory review of predictive models proposed by the draft white paper presents serious market-restriction issues for FICO…..and for the hundreds of FICO® Insurance Score clients doing business in all states that allow for the industry’s significant use of credit-based insurance scores within their well-considered and comprehensive rating programs.

As mentioned previously, we believe the state regulatory practices under which FICO has supported our clients for the past two decades are appropriate and quite sufficiently protect all interests – consumers, regulators, and insurers.

The draft white paper’s only references to protection for the intellectual property and trade secrets of an independent analytics provider like FICO are too vague to offer any real protection. The proposal, as highlighted here, leaves the decision about confidentiality of a company’s intellectual property and trade secrets entirely within the discretion of each state regulator.

1. The fourth Key Regulatory Principle: State insurance regulators will maintain confidentiality, where appropriate, regarding predictive models.


4/6/19


Kris DeFrain, FCAS, MAAA, CPCU January 14, 2019 Page 3

2. Section V. CONFIDENTIALITY warns rate filers:

Insurers and regulators should be aware that a rate filing might become part of the public record. Each state determines the confidentiality of a rate filing, supplemental material to the filing, when filing information might become public, the procedure to request that filing information be held confidentially, and the procedure by which a public records request is made. It is incumbent on an insurer to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filing.

Nearly a decade ago, FICO was forced to withdraw its credit-based insurance scoring models from new or amended filings in the State of Florida over a very similar issue – lack of appropriate confidentiality protections. Because the Florida Office of Insurance Regulation rules forced FICO to withdraw our models from use by insurers in Florida, there was a significantly negative impact on FICO’s insurance clients and the hundreds of thousands of consumers who benefitted from the use of FICO Insurance Score models in Florida.

Since FICO cannot be left in a precarious position with respect to the protection of its intellectual property, if the drafted white paper is adopted, as written, by any state without necessary trade secrets and otherintellectual property protections in place, FICO may be forced to remove our FICO Insurance Score models from use by our insurance clients in that state, creating wholly unnecessary market disruption.

We look forward to working with the NAIC Casualty Actuarial and Statistical Task Force toward a regulatory review approach that protects the interests of all stakeholders, including the vast numbers of US consumers who benefit from the insurance industry’s continued use of predictive models to enhance their underwriting and pricing policies based on proven risk characteristics.

Sincerely,

Lamont D. Boyd, CPCU, AIMInsurance Industry Director, Scores and Analytics

[email protected] 602-317-6143 (mobile)

FICO Insurance Scores Consumer website at insurancescores.fico.com offers consumers, agents, regulators, legislators and others a thorough understanding of FICO’s credit-based insurance scores, the insurance industry’s use of our insurance scores, and general credit management tips.


4/6/19



4/6/19


January 15, 2019 Mr. Richard Piazza Chief Actuary, Actuarial ServicesLouisiana Department of Insurance P.O. Box 94214 Baton Rouge, LA 70804 Via Email

Dear Mr. Piazza: I appreciate your time the other day to allow LexisNexis Risk Solutions to offer feedback on the Regulatory Review of Predictive Models White Paper as an Interested Party. While our company historically has been known for our Attract Models (predictive models using credit-based insurance scoring used in Auto, Property and Commercial Lines), we have developed hundreds of custom models used by carriers, and are in the process of developing new and innovative models that will be filed in the near future (and many of which have been socialized with you and your staff). Additional information about our company and abilities can be found at https://risk.lexisnexis.com/products/predictive-modeling The following bullet points should be considered high-level feedback and commentary. We would like to meet with you further for a more in-depth discussion for input on your whitepaper.

We believe that the intent of this task force is to create uniformity across states to streamline the process of model filing, evaluation and implementation across states. We feel this is a step in the right direction as we would expect that filing reviews could be expedited.

The majority of items noted in the whitepaper are found in many state model filing check lists. When we develop models, our process includes documenting this type of information for both our own historical model development notes as well as for regulatory filings.

The proposed list is relevant to primarily one type of model (GLMs), which seems to be predominantly used in insurance. Modern methods, such as model blending or GBMs, are not well covered by these questions.

One area of potential concern is that the model requirements focus on the “science” part of the modeling process. Predictive modeling also involves a bit of “art”, including a modeler’s work to iterate variables, explore interaction effects, utilize multiple techniques, use sequential analysis, binning, etc. It may be worth providing some guidance to still allow for the “art” vs. pure science.

The majority of our modeling work at LexisNexis Risk Solutions is spent handling “dirty” or inconsistent data including extracting structure, imputing missing values, and handling incorrect data (to name a few). This step is very important since it will help us build data knowledge needed to apply the scientific standard of estimation based on a true statistical model. Modeling is difficult even if you fix the algorithm of choice, and just focus on the variable selection problem especially when dealing with large data sets with large number of inputs. However, using our knowledge of the data and experience, we can resolve to some heuristic techniques to find adequate solutions.


4/6/19


o For example, one of the rules is defined by the 5% threshold on p-value (in big samples, the threshold can be even more restricted). The p-value is the probability of finding the observed, or more extreme, result when the null hypothesis is true. Since the null hypothesis is usually associated with “no effect” of an attribute, a smaller p-value means that there is strong evidence that such attribute has indeed an impact on the target. However, many times (especially when sample sizes are small), the statistical approach may not capture the significance of some attributes and shows p-values greater than 5%. Based on previous experience, statistical modelers at LexisNexis Risk Solutions can identify important attributes that did not pass the 5% threshold and use alternative approaches such as bootstrapping or marginal impact to accept attributes with moderate p-values in the model.

Relative to the use of raw data (noted in Section A.5.a), LexisNexis Risk Solutions provides aggregate descriptive statistics related the loss experience of underlying modeling datasets. We work closely with state regulators to comply with various statues and regulations. However, we can only share data with regulators within the confines of our carrier contractual limitations. As good stewards of consumer data, we are extremely cautious with customer data and are often constrained by data privacy laws in the detail we can share. We work diligently with the insurer and regulator in each state regarding the confidentiality of the information submitted with the model filing.

Our biggest concern is sharing our extremely valuable intellectual property (e.g. our step-by- step statistical methods and key decisions). For example, the exact details on data scrubbing, variable selection methods, etc. would become publically available in some states. In particular, answering all of these questions would involve publishing algorithms and data processing, and the value of work on publicly available data, e.g. NFIRS, could plummet as all derived attributes would be exposed to copying. In banking, Basel requirements have similar goals and output, but since all work is kept internal, there is no issue with sharing intellectual property.

o We realize that allowing confidentiality of modeling filings is primarily set by state statute. Some of these allow auto and home models to be confidential, but stop short on other lines of business (e.g. Commercial). What we would propose is, if a state would require the recommended information, and if the state does NOT allow model intellectual property to be considered confidential, the state would accept the information via a different method (e.g. email to the reviewer) and kept out of the SERFF system.

Thank you again for allowing us to submit feedback. Please let me know if you have any questions. Sincerely, Gary T. Sanginario, CPCU Director, Product ManagementAnalytics Products and State Relations


4/6/19


To: Casualty Actuarial & Statistical (C) Task Force

From: Brent Kabler, Research Supervisor, Missouri Department of Insurance, Financial Institutions & Professional Registration

Re: Draft white paper Regulatory Review of Predictive Models

01-11-2019

First, I extend my appreciation to the CASTF for the obvious effort put into the draft. One thing that jumped out at me is that the document, taken as a whole, doesn’t seem to have an obvious audience. Namely, is this written for a non-technical audience that will have limited familiarity with statistical concepts? Or is the intended audience primarily working actuaries? If it is the former, I strongly encourage a significant rewrite of the draft to avoid jargon that will be entirely unfamiliar to a non-technical audience, and try as much as possible to convey concepts in a way that is broadly accessible as possible. I fully realize that is easier said than done, but as written a non-technical audience won’t be able to make much sense of a majority of the draft.

That aside, my primary concern with the draft is that it fails to address in any kind of serious way a problem that will increase significantly with the adoption of data mining techniques and the increasing availability of very large data sets that dwarf anything available even just a couple of decades ago. Data mining will (as has been shown across a wide variety of fields) dramatically increase the rate of false positives – the technique will inevitably churn up numerous associations between variables that are simply random non-meaningful correlations resulting purely from chance. I believe this is a near certainty. Secondly, the apparent complete disregard of causality that seems common among practitioners of data mining techniques will significantly magnify the problem. Causality forms the basis of the standard model of all natural and social sciences. I would argue that evaluations of models should consider the nature of observed relationships within the context of prior substantive knowledge. While I fully realize that the SOP accords a fairly diminished role for causality, I don’t think those standards justify dispensing with it entirely.

These problems are in no way esoteric or “purely academic.” Numerous disciplines have grown increasingly concerned with the “replicability crisis,” or the fact that a substantial proportion of published research cannot be replicated and are in fact simply false positives. It is worth nothing that these disciplines engage in far less data mining (if at all) than is common now among insurer rating practices. Indeed, in many disciplines, data mining can be considered outright academic fraud, depending on the nature of the disclosures made and egregiousness of the practice. This problem has grown so significant and is so widely recognized that the American Statistical Association took the unusual step of releasing a pretty strong warning against such practices (more on this below).

These comments in no way seek to upend standard actuarial practice. Rather, they are designed to raise awareness of the problematic nature inherit to data mining and hopefully to stimulate discussion about appropriate remediation measures. Data mining will no doubt remain an enduring feature of rating, given the strong competitive market pressures that incentivize rapid innovation. However, the practice raises some thorny problems for regulators seeking to ensure the validity of rating structures.

The truth is that data mining stands the standard scientific hypothetical-deductive method on its head (though it can’t rightly be called an inductive approach either). The standard scientific model usually starts with a body of substantive (or causal) knowledge. Rigorous hypotheses are derived from gaps in such knowledge. In it most rigorous form, all statistical tests are specified prior to conducting such tests, and the implications of each test are understood in terms of what they will reveal of a causal nature. Changing the tests after the fact


4/6/19


is discouraged, for strong methodological reasons. The correlation alone is not the final arbiter of the validity of findings, but causal understanding is employed to assess which correlations may be entirely due to chance, what are non-causal relationships, and which are most likely to be enduring causal relationships. This approach of employing prior knowledge to assess the validity of results is formalized most rigorously in methods known as Bayesian statistics.

Data mining generally proceeds in exactly the opposite manner. Rates are generally made in a vacuum of substantive or causal knowledge. Data mining employs various algorithms that may perform many thousands of statistical tests as they comb through enormous quantities of data that encompass countless variables. Results are interpreted from a series of purely post hoc explanations (or really, rationalizations), which often consist of little more than pure guesses as to the nature of the relationships that are churned out.

Such methods systematically undermine the level of confidence (in the statistical sense) that can be placed in results. Consider that statistical tests measure the probability that an some relationship observed in a data set is merely due to chance – a association that is entirely random (due, for example, to sampling error). The most common measure is the p-value, a meaningful probability that a relationship will be observed between variables when there is in fact no relationship beyond random chance. Generally, p-values are set to a maximum of .05, meaning that a relationship will be rejected unless there is less than a 5 percent chance that it would occur due to random chance alone. That is, if one hundred different relationships with a p-value of .05 are discovered, the chances are that five of them are non-meaningful chance relationships. When data mining churns out possibly many thousands of correlations, it will significantly increase the problem of such false positives.1

The literature is filled with countless examples of erroneous results produced by more or less random “interrogation” of data sets (which, again, are far less egregious than outright data mining). The CDC recently completed yet another study that found no relationships between vaccines and autism. As noted above, the study was designed in adherence to rigorous standards – testing protocols specifying all statistical tests were adopted at the very outset, and the study did not seek to alter or modify such tests over the course of the study. The study found, as expected, no statistical relationship between the presence or absence of vaccines at time of diagnosis, no observable effect of the timing of vaccines, etc.

A subsequent external “researcher” subsequently “found” a relationship between vaccines and autism diagnoses in a small cohort – African-American males that had received vaccines later than the medically recommended schedule. Examining this cohort alone, all measures proved statistically “significant.” This finding of course unsettled all of the usual suspects, was deemed proof of a “massive cover-up” at the CDC, and even resulted in a Congressional inquiry.

But no serious researcher accepts the finding as at all meaningful. As noted above, the more tests that are run, delving down to more and more narrowly defined subpopulations, the more likely random associations will be uncovered. The test wasn’t specified at the outset, but was clearly the result of simply running random statistical tests across different cohorts until a relationship is found, unguided by any prior knowledge. Nor is there any theoretical reason to accept this finding. After all, what possible causal mechanism would result in a vaccine-autism connection among just this cohort and no other?

It is clear that many historical rating variables are causally understood. The relationship between young drivers and crash risk is generally acknowledged. In part, the elevated risk is attributable to lack of experience. However, we also know that the young have a significantly higher crash risk than older individuals with the same driving experience (say, 30 year-olds that first obtain a license). As such, we can 1 A process sometimes referred to as “torturing the data until it confesses.”


4/6/19


reasonable infer that youthfulness per se is a risk factor. There is substantial support for this in the psychological literature, which amply documents higher risk-taking behavior, lack of foresight, and impulsiveness among the young. There is also strong evidence for why marital status may be related to risk.

However, there is little if any understanding of how credit score might be related to risk, or vehicle history scores, or any of the other countless rating variables of more recent provenance that have been uncovered via data mining techniques. If we lack any causal understanding that might afford a more Bayesian evaluation of model validity, we have to fall back on the nature of the statistical relationship itself which, as noted above, is increasingly likely to be attributable to pure chance with the explosion of data mining.

That said, I’m not necessarily of the old school of statistical thought that strongly asserts that data mining should never be performed, or that it is invalid on its face (though many statisticians do believe it should never be done). Data mining can have uses for exploratory analysis, or ways of suggesting promising avenues of research. But I am of the school that believes that data mining should never be the final analysis. Nor should analysis dispense entirely with notions of causality. Lack of rigor has produced such findings as

1. Individuals that have more rigid “back – white” cognitive styles are actually less able to physically distinguish different colors (Nosek, Spies, and Matt Motyl. 2012)

2. Bible “codes” existing as statistical proximity of various word pairs perfectly predict the future (Witztum, Rips and Rosenberg, 1994), a “finding” that literally launched a billion dollar industry even though thoroughly debunked as an egregious form of data mining,

3. Aspirin therapy for cardiovascular disease is more likely to result in death for Geminis and Libras but is beneficial for other astrological signs (from findings of a recent international study of survivors of heart attack, involving more than 134,000 patients in over 20 countries)

4. Listening to the Beatles’ When I’m 64 can literally change peoples’ actual chronological age (Simmons, Nelson and Simonsohn, 2011).

As noted above, the American Statistical Association expressed some degree of alarm at approaches similar to (though again, far less egregious than) data mining (Wasserstein and Lazer, 2016). In a formal statement of the ASA, the association warned against a purely “cookbook” approach to statistics: “a p-value near .05 taken by itself offers only weak evidence of the null hypothesis” (page 129). In addition, the ASA emphasized the centrality of causal prior knowledge in interpreting statistical results: “Researchers should bring many contextual factors into play to derive scientific inferences, including the design of the study, the quality of the measurements, the external evidence for the phenomenon under study [i.e. causal or theoretical knowledge], and the validity of the assumptions that underlie the data analysis” (page 130, emphasis added).

Lastly, the ASA warned strongly against an over reliance on data mining: ‘Cherry-picking promising findings, also known by such terms as data dredging, significance chasing…and ‘p-hacking,’ leads to a sspurious excess of statistically significant results…and should be vigorously avoided” (page 131, emphasis added).

But as we have seen, data mining goes well beyond simple offhand “cherry-picking” of findings. Indeed, it is a systematic and methodical system of cherry-picking. It is unclear how regulators should respond to the explosion of data mining, but it seems clear that the issue should be directly confronted. In general, the ASA recommends a model of full disclosure that requires that regulators acquire far more


4/6/19


information than is present in a typical rate filing. Regulators must be aware not only of the final model, but also the history of model building including the total number of hypotheses tested and all p-values computed.

I would encourage the CASTF to confront these issues far more directly in the draft, including fully outlining the nature of the problem and any remedial measures designed to guard against false positives. In part, the draft already offers some degree of remediation, including the importance of hold out or model validation data sets. This portion should be further accentuated in relation to data mining, and perhaps made mandatory for acceptance of any models predicated on data mining. Additional measures should also be addressed. For example, reported p-values should be adjusted based on the number of tests performed. One such commonly accepted test is the Bonferroni correction, in which the p-value is adjusted upwards in relation to the number of tests performed. Other adjustment methods are available in standard statistical texts.

In addition, full disclosure of model development history is absolutely essential. Proper interpretation of statistical results absolutely requires full knowledge of every stage of model development, and not just the final result. In part, the draft does some justice to this essential need. I would encourage making it a fundamental principle and include a robust discussion in the body of the draft.

Lastly, I strongly believe that regulators need to do better with respect to issues of causality. Actuarial standards of practice state that “While the actuary should select risk characteristics that are related to expected outcomes, it is not necessary for the actuary to establish a cause and effect relationship between the risk characteristic and expected outcome in order to use a specific risk characteristic” ( Actuarial Standard of Practice No. 12, Section 3.2.2, emphasis added). Unfortunately, this statement has most often been interpreted to mean that actuaries (and regulators) can dispense with causality entirely. As noted above, proper statistical interpretation absolutely requires some knowledge of causality. While there are some statements in the draft that could be interpreted as addressing this issue (however obliquely) I encourage the CASTF to tackle this issue head-on. It is entirely appropriate in standard statistical methods of interpretation to bring all prior knowledge and understanding to bear in assessing the validity of a statistical model. As the ASA emphasized in the quote above, it is an essential part of any such interpretation.

Again, I am not suggesting that actuarial practice be upended. It would be quite costly for insurers to adopt the most rigorous and methodical and above all slow methods of the sciences. But the positives of innovation must be tempered with proper evaluation by regulators of models. A false positive, by definition, violates the universal standard of treating like risks the same. These issues aren’t purely academic in nature, but clearly have practical implications for model review. I would strongly encourage they be confronted fully in much the same way that other disciplines are beginning to.


4/6/19


Bibliography

Nosek, Brian A., Jeffrey R. Spies, and Matt Motyl. 2012. Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science. 7(6): 615-631

Simmons, Joseph P, Leif Nelson, and Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psycological Science. 22: 1359 – 1366. Text available at http://people.psych.cornell.edu/~jec7/pcd%202015-16%20pubs/Simmons%20PsySci%202011.pdf

Witztum, Doron, Eliyahu Rips and Yoav Rosenberg. 1994. Equidistant letter sequences in the book of Genesis. Statistical Science. 9: 429-438.

Wasserstein, Ronald L. and Nicole A. Lazar. 2011. The ASA’s statement on p-values: Context, process and purpose. The American Statistician. 70: 129-133.


4/6/19


January 15, 2019 NAIC Casualty Actuarial and Statistical (C) Task Force c/o Kris DeFrain - [email protected] 1100 Walnut Street, Suite 1500 Kansas City, MO 64106-2197 Re: NAMIC Comments on CASTF’s Predictive Model White Paper Dear Task Force Chair, Vice Chair, Task Force Members, and Other Interested Regulators, Please accept the following comments of the National Association of Mutual Insurance Companies (hereinafter “NAMIC”)1 on behalf of its member companies regarding the exposed Predictive Model White Paper. NAMIC wishes to thank the Task Force for the ability to provide comments on the white paper and this extremely important concept of predictive modeling as it impacts the industry and consumers. Further, NAMIC wants to commend the Task Force for its diligence and thoroughness in attempting to ascertain “best practices” for predictive modeling analysis in the property/casualty insurance market. Despite the well-intentioned drafting of predictive model review parameters, there remains concerns that this document does not provide practical real-world examples to achieve the necessary goals of consumer protection and approval of filings so that companies may continue to compete in the marketplace in a timely and efficient fashion. Quite simply, the white paper calls for an inordinate amount of compliance expenditure and leaves very little discretion in the handling of even rudimentary filings. The compliance costs and human capital associated with responding to the various and detailed criteria will be enormous and potentially expose proprietary trade secrets to excessive review and dissemination. All of this may be occurring without the slightest scintilla of concern or other potential regulatory trigger being posited. Some generalized and specific concerns would include the following.

NAMIC is the oldest property/casualty insurance trade association in the country, with more than 1,400-member companies

representing 41 percent of the total market. NAMIC supports regional and local mutual insurance companies on main streets across America and many of the country’s largest national insurers. NAMIC member companies serve more than 170 million policyholders and write more than $253 billion in annual premiums. Our members account for 54 percent of homeowners, 43 percent of automobile, and 35 percent of the business insurance markets. Through our advocacy programs we promote public policy solutions that benefit NAMIC member companies and the policyholders they serve and foster greater understanding and recognition of the unique alignment of interests between management and policyholders of mutual companies.


4/6/19


Inordinate Compliance Costs The document consists of approximately 27 pages of “best practices” containing more than 86 “essential” and separate items of analysis and review of filings to be considered by each insurance department. Each of these “essential” items can contain multiple steps to complete making the actual compliance threshold much higher. NAMIC respectfully suggests that most of these “essential” steps should provide for more regulator discretion and be converted to “May Be Requested.” As written, the scope of the white paper would inevitably and dramatically increase compliance costs in responding to inquiries about rudimentary filings that will drain resources that could be more efficiently and effectively utilized in policyholder services and operational concerns of the company.2 Speed to Market Inevitably, NAMIC would submit that this paper could cause a significant reduction in timely filing approval causing not only the aforementioned inordinate compliance costs but the ability for companies to compete in the marketplace. If filings are delayed, that delay ultimately harms consumers as market competition is reduced.3

2 By way of example, names and contact information for those who built the model, explanations of how the insurer will help educate consumers to mitigate risk when some factors such as age or gender cannot be “mitigated,” and when the models were begun and finalized, might be information a regulator would like to see but it should by no means be considered “essential” to review and approve a filing.

3 By way of example, items such as the following seem to be a best thought but not necessarily a mandatory best practice: (1) “Provide a complete list of all characteristics/variables used in the proposed rating plan, including those used as input to the model and all other characteristics/variables used to calculate a premium.” (2) “For each characteristic/variable used as both input to the model and as separate univariate rating characteristic, explain how these are tempered or adjusted.” (3) “Provide state specific, book of business specific univariate historical experience data consisting of, at minimum, earned premiums, incurred losses, loss ratios and loss ratio relativities for each category of model output proposed to be used within the rating plan.” The last example completely goes against the idea of multivariate model-based rating plans. If an insurer still needs to be able to support the model with state-specific univariate results, the concepts conflict causing substantial more compliance adherence. Likewise, with “[p]rovide an explanation of any material differences between model indications and state specific univariate indications.” In this situation, if an insurer built a companywide model for deployment in 50 states, it would have to explain all material differences that the model had with individual state univariate analysis. This is excessive, unnecessary, costly, and moves the process in completely the wrong direction. The same concern applies to the comments section of this line that also states, “Credibility of state data should be considered when state indications differ from modeled results based on a broader data set.”


4/6/19


Department Expertise and Increased Budgetary Costs NAMIC believes that in addition to the previous considerations, additional thought should be expended on the compliance costs for the regulators themselves. While this work is exhibited to be a “best practice” white paper, it is not remote logic that states will extrapolate and adopt this paper as their approval manual. Otherwise, they could be left open to criticism that they failed to follow an “essential” act contained in the paper. It would then appear from these parameters that a great deal of staff, outside expertise, and increased budgetary costs are going to be incurred. A fiscal impact analysis should be ascertained before this paper is adopted. NAMIC is not suggesting that any filing that has reasonable triggers or cause for regulator concern should not be deeply and thoroughly examined for compliance with existing laws. However, CASTF will be causing this rigorous scrutiny of each and every filing. The exposure and ultimate adoption of this paper cannot be accomplished in a vacuum without considering ancillary impacts. Another consideration is that with such complexity of models, it is disconcerting to see guidance in the white paper that states “[g]iven an insurer’s rating plan relies on a predictive model and knowing all characteristics of a risk, a regulator should be able to audit/calculate the risk’s premium without consultation with the insurer.” We fail to see how a best practice would ever encompass failing to discuss a concern with the regulated entity to at least obtain their response as to their position. Basic fairness and due process notions would dictate an outreach when there are discrepancies or unanswered questions. Too Prescriptive Many of the “essential” requirements seem to be too prescriptive as to intent. While in a perfect best-case vision, some of these scenarios may be determined to be useful on a case-by-case basis, many go too far in the required analysis. Some require regulators to inquire about predictor variables in an old model that may no longer be used in a newer version. This requirement seems to disregard the dynamic nature of modeling and may lead to improper assertions and conclusions of the regulatory body over a matter that is no longer utilized. Additionally, there is a discussion about measuring and describing the impacts on expiring policies and describing the “process used by management to mitigate or get comfortable with those impacts.” There does not appear to be a legal requirement for “mitigation” and that concept is not clearly spelled out within the document. This leaves the matter open to broad and problematic interpretation. The same would go with the concept of “get comfortable” with an impact. These are not clarified terms and do not belong in this document. One can envision a regulator making a unilateral decision that management simply never got comfortable with what they implemented. This would provide simply no factual usage in determining the legal and regulatory parameters to ascertain filing approval. It will clearly slow down the process.


4/6/19


The general legal parameters for rate filings require them to be adequate, not unfairly discriminatory, or excessive. It is difficult to extrapolate why these concepts previously mentioned and contained in the white paper assist in ascertaining those thresholds. The concept of “large premium disruptions” and providing analysis for an anticipated negative impact appears to be already assumed prior to the analysis despite legal requirements and financial prudential mandates that rates be adequate. This inquiry is considered “essential” before lack of actuarial justification has even been identified. Items such as “[e]xplain how the insurer will help educate consumers to mitigate their risk” is another such paradox. This does not appear to be a legal requirement but rather a public policy pronunciation. It is not a best practice to inject such discussion into a rate filing. Likewise, an inquiry that states “[p]rovide the regulator with a description of how the company will respond to consumers’ inquiries about how their premium was calculated,” is deemed “essential” but is not really connected with determining the adequacy, excessiveness, or other criteria for a rate filing. It appears that a market-conduct-type inquiry is already being interjected into a rate filing analysis that will only take up needless time to respond appropriately and is premature when no trigger or actual perceived harm has been demonstrated. Confidentiality, Proprietary Information, Trade Secrets, Contractual Terms, and Information Sharing Exposing confidential and proprietary trade secrets, confidential information, and other business practices simply for accumulation of data in a rate filing, when otherwise unnecessary, is problematic for all involved. States may information share the data by law with other regulators. The data provided for these “essential” requirements subjects the regulator to increased Freedom of Information Act requests, subpoenas, and other types of litigation when there has been no demonstrated harm to consumers or trigger for the inquiry. Additionally, some proprietary models may have contractual terms that prevent disclosure and therefore an interference with contractual relations may occur. Without a demonstrated necessity, exposing this data to additional dissemination appears to be hindering its protection. The NAIC should update and strengthen its information-sharing platforms and protocols to absorb such complex and proprietary data if this is the path forward. It is unfortunate that the only discussion of confidentiality in the document makes essentially two points. One, that the information filed with departments “might become public” and two, that “it is incumbent upon the insurer to be familiar with each state’s laws regarding the confidentiality of information submitted with their rate filing.” There is no concomitant duty mentioned for regulators to protect confidential and proprietary information and the phraseology essentially attempts to alleviate that burden entirely. As custodians of this sensitive and complex information, more should be stated in the paper and, in fact, done to protect the same especially since it is the regulator who is demanding this broad and sensitive information without any particular threshold of concern.


4/6/19


Confusing or Non-Illuminated Terminology and Word Selection

Additionally, NAMIC believes a revisit of terminology used throughout the document should occur. As mentioned herein, terms such as “clear understanding,” “premium disruption,” “intuitive argument,” “large premium disruption,” “significant changes in premium,” and “get comfortable” are but some of the verbiage used throughout the paper that do not have quantifiable definitions. Consequently, they will be subject to broad interpretation by the reviewer that can only lead to more potential follow-up and compliance requirements in the absence of a concern or issue with the filing itself. These broad terms, including the use of “essential,” will result in every rate filing being a complex and time-consuming compliance endeavor. Again, it is ultimately the consumer who will be potentially harmed when innovative products cannot timely reach the marketplace.

Therefore, in closing, NAMIC would again thank the Task Force for its tireless work on this topic. However, while the best of intentions was undoubtedly attempted in this draft, significant concerns remain about its efficacy in today’s marketplace. We would ask that you take our considerations under advisement and consider another draft of this white paper alleviating the prescriptive rigid dynamic that has been created. NAMIC would offer its assistance to the Task Force and looks forward to working with the Task Force on this critical work where needed.

Sincerely,

Andrew Pauley, CPCU Government Affairs Counsel National Association of Mutual Insurance Companies (NAMIC)


4/6/19


To:

DeFrain, Kris, Piazza, Rich Vigliaturo, Phillip Darby, Sandra McGill, Mark Stolyarov, Gennady Davis, Daniel J. Eric Hintikka From: Gordon Hay Date: November 16, 2018

Re: Draft CASTF Model Review White Paper

Tomasz Serbinowski commented:

“Several places in the draft suggest that the regulator should require an intuitive or logical support for the selected explanatory variables. Why this sounds reasonable, it probably is not. Historically, auto insurers were allowed the use of gender as a rating variable. I don't know of any acceptable logical or intuitive explanation of that. I would venture a guess that an insurer could not offer any explanation other than the data showing that losses vary by gender.”

I have given this some thought, and I recommend:

We should guide reviewers to challenge variables for which the rate filer provides no explanation that rings true intuitively and logically. Some candidate variables are simply not sensible compared to more intuitive and logical alternatives.

However, “fairness” has always had more than one definition, leading to a balance or trade-off between fairness by solely statistical/actuarial criteria versus fairness that reflects socio-economic or cultural values. Sometimes an issue gets legislative or regulatory outcomes that differ depending on local conditions that may evolve over time. I think the white paper needs to explicitly acknowledge that “intuitive and logical” can be subjective and subject to change. States may need to give reviewers written guidance on variables that are prohibited and/or outline a procedure for regulating the introduction or sun-setting of variables that may be sound actuarially and statistically but not so sound politically.

My thoughts getting to that recommendation?

“Intuitive” and “Logical” may be desirable to all, but definitions will get into philosophy. In any given context, judgments based on intuition or logic inevitably depend on subjective perceptions and/or


4/6/19


beliefs. There has always been a difference between “unfairly discriminatory” in a statistical sense versus concerns about fairness that get political or legal treatment using logic that could follow statistical evidence, compromise with it, spin it, or simply discount it.

In practice, rate filers should have avoided using variables for rating, underwriting or other purposes that are legally or politically unacceptable, whether statistically predictive or not. So “race” to my knowledge is explicitly omitted everywhere, and gender should be omitted where it is prohibited. There may always be variables that are found to measure something unacceptable, making statistical validation beside the point. E.g. insurance applications for personal credit data have been under scrutiny for decades, and are prohibited for personal lines rating by some states. As modelers reach more deeply into “big data,” new predictive variables keep emerging. Some geodemographic variables with “poor optics” might be avoided for appearances sake, avoiding any arguments over their potential correlation with some prohibited variable.

Rate filing reviewers have always been wary of new classification variables, but seeing new variables several times a decade. Currently there’s a proliferation of new data sources and candidate variables, and inevitably some of them will be “unfairly discriminatory” due to some combination of statistical, legal or political criteria. States will differ in their positions regarding regulatory hurdles for innovation, including acceptability criteria for new variables. If left to the individual reviewer’s discretion, it isn’t realistic to expect consistent review criteria from one filing to the next, or sustained maturity of review from one reviewer to the next. If a State has taken positions on what types of variables are legally and politically acceptable in multi-variate rating systems, it might be a “best practice” to provide model reviewers (and rate filers) with a written guideline.

We should still guide reviewers to challenge variables for which the rate filer provides no explanation that rings true intuitively and logically. Some variables are genuine nonsense. Let’s say we’re trying to predict commercial GL loss ratios and make a Location Index for each zip code that’s fitted (with other non-geographic variables) using three census data elements: Number of death care services establishments, Number of office and stationary stores and Percentage of population between ages 80-84 from US Census data. Does that pass the “intuitive and logical” test, or do we ask for an explanation with reasons why these variables in particular? Is selection using some statistical technique going to suffice, or do we guide the reviewer to request an “intuitive and logical” explanation?


4/6/19


1

Comments on Draft NAIC CASTF White Paper on the Regulatory Review of Predictive Models Gennady Stolyarov II, FSA, ACAS, MAAA, CPCU, ARe, ARC, API, AIS, AIE, AIAF

Lead Actuary, Property and Casualty Insurance Property and Casualty Section, Nevada Division of Insurance

Kris DeFrain, FCAS, MAAA, CPCU Director, Research and Actuarial Services National Association of Insurance Commissioners (NAIC) Sent via e-mail at [email protected] January 13, 2019 Dear Ms. DeFrain: Thank you for the opportunity to comment on the draft of the NAIC Casualty Actuarial and Statistical (C) Task Force white paper on the Regulatory Review of Predictive Models. My comments below are subdivided into a section of substantive remarks, followed by a section recommending some minor editorial changes. Substantive Remarks While the paper remains in a draft form with various sections that will still need to be added, expanded, or refined, I consider the conceptual framework and specific elements of guidance presented in the current exposure draft to reflect an essentially correct approach. Having participated in the group of regulatory actuaries who contributed to this exposure draft, I am aware of the extent of thoughtful and thorough discussion and consideration of each element that went into this process. While different States have different legal and regulatory frameworks, and the intent of this paper is to be consistent with all of them, the discussions among the paper’s drafters elicited recognition that there are many common areas of focus, desired understanding, and – in certain cases – concern that regulators who review predictive models experience in their work. This paper strongly emphasizes that the best practices therein are guidance and that States’ regulatory autonomy takes precedence; this is exactly as it should be. At the same time, it remains each reviewer’s professional prerogative to seek a deeper understanding of the models he or she is reviewing, to ask questions, and to discern – even if only for his or her benefit (though often with other, more impactful implications as well) – whether a predictive model is appropriately supported, and whether the constituent elements of that predictive model make sense. Different States will apply this guidance differently. Prior-approval States, such as Nevada, already require answers in connection with many of the elements expressed in this paper. Other States may use the guidance in this paper to choose which model elements to focus on and/or to train new reviewers or to gain a superior understanding of how predictive models are developed, supported, and deployed in their markets. The above context for this paper renders it essential that some of the more conceptual and qualitative guidance in it remain in place, notwithstanding potential (and most likely actual) philosophical objections from certain interested parties. Most regulatory actuaries and other filing reviewers are aware that, for a long time, certain elements within the property/casualty


4/6/19


2

insurance industry have made what I will term the “correlation-only argument” – which asserts that to demonstrate a lack of unfair discrimination for a variable, it is enough to show a statistical correlation between insurance losses and/or expenses, and that even the very discussion of causal possibilities should not be attempted. Both I personally and the Nevada Division of Insurance as a regulatory agency disagree strongly with this “correlation-only” position. While, indeed, it is difficult to prove causation, and such a proof is not a standard against which rate filings are evaluated in any jurisdiction to my knowledge, there is an immense difference of both degree and kind between proving causation and discussing an intuitive or logical connection between a particular attribute and the risk of insurance loss. It is a non sequitur to assert that the lack of requirement for the former (proof) confers immunity upon insurers in regard to the latter (discussion and expression of plausibility).What a State does with the results of such a discussion is, of course, subject to the framework of statutes, regulations, precedents, and processes that comprise the insurance regulatory framework in that State; these will necessarily vary by jurisdiction. However, the very act of discussion of the intuitive, logical, or plausible relationships of individual risk attributes to the risk of insurance loss – and consideration of all related and relevant implications (such as perception by consumers, legislators, and media; philosophical considerations of fairness; interactions with public policy as determined by the relevant policymaking bodies; and relevance to the evolution of the insurance industry, consumer products, and overall impacts on the incentives and opportunities available to consumers) – is crucial to engage in and continue to do so for as long as new predictive models are being developed, new variables are being introduced, and consumer premiums as well as insurer underwriting decisions are being affected. In other words, the discussion needs to continue indefinitely in a variety of venues and evolve along with the industry and the broader society; we as insurance professionals cannot viably insulate ourselves from participation in the conceptual discourse. Furthermore, as actuaries, if we are indeed to practice the discipline called actuarial science, then it is incumbent upon us to adopt the proper scientific mindset of open inquiry – where no questions are off limits and continued systematic exploration and progress are the hallmarks of the scientific approach. Any insistence that certain questions must not be asked, or certain concepts must not be explored, that certain discussions must simply be cut off, entails a departure from the realm of science into the realm of dogma; if widely acquiesced to, this mentality would ossify the profession and quickly deprive it of broader relevance. Regulatory actuaries and other trained staff, especially when they review predictive models, are in a prime position to be the torchbearers for the scientific approach by maintaining the commitment to open but rigorous, systematic, and principled inquiry. It is important to emphasize that the white paper, as currently drafted, does not prescribe any specific answers regarding which particular treatments are to be considered logical or intuitive. Such answers cannot be arrived at without considering the context of a given jurisdiction’s laws, marketplace, and the specific nature of insurers’ proposals. Therefore, to preempt any arguments by some interested parties that the paper may be attempting to prescribe specific solutions or restrictions – it clearly is not. The purpose of the paper is to provide guidance to enable regulatory reviewers to be aware of possible questions to ask and issues to consider so as to add value to the review process and improve the robustness of the regulators’ consumer-protection role. It is entirely possible that different individuals and different jurisdictions will ultimately arrive at different answers to these questions, just as actuarial judgment may be deployed by


4/6/19


3

different actuaries in different ways and with different opinions and recommendations. All this is part of the continuing discussion and exploration of these essential questions. A common counterargument may be to present certain circumstantial characteristics that have been in use for a long time in most jurisdictions, such as age or gender, and suggest that these do not have a readily apparent intuitive relationship to the risk of loss for certain lines of insurance – such as automobile insurance. Such a counterargument, however, falls short in that it does not recognize that members of the general public are at least capable of positing comprehensible hypotheses as to why these attributes may (or may not) be predictive of risk. Opinions on such matters will certainly vary, and my own anecdotal experiences with laypersons’ perspectives have put me into contact with a wide-ranging diversity of thought. However, the key regarding older and more established rating attributes is that intelligent individuals can at least have discussions and provide plausible narratives regarding such characteristics. (Here I do not take any position as to which, if any, of those narratives is actually correct in explaining the data that insurers observe and present.) This, however, is not the case when it comes to esoteric rating variables that appear to have come into being solely because a predictive model correlated insurance losses with a particular characteristic that happened to be found in a dataset which an insurer purchased from a third party that was eager to market the data as having some insurance applications. While some innovative, reasonable, and truly predictive new attributes could indeed constitute improvements to a rating plan, this is not a license to use just any attribute merely because a correlation was found in some instances for some data sets. Many regulatory reviewers have encountered attributes, proposed to be used in rating, which could not possibly be related to actual consumer behaviors that affect the risk of insurance loss – or, if they are related, the relationship is directionally contrary to the proposed treatment of the attribute. (An example of the latter would be a behavior indicative of financial responsibility – such as refraining from taking out an installment loan or paying off an installment loan – being used adversely, to surcharge a consumer in a credit-based insurance scoring model. The Nevada Division of Insurance has, over the past decade, made efforts to surgically excise such treatments from credit-based insurance scoring models with no loss of the models’ predictive ability.) As ever-accumulating volumes of consumer data are being generated and sold, it is incumbent upon all decision-makers to thoughtfully consider the purposes to which such data are being put – not only whether such uses respect the essential values of privacy and consent, but also whether tying individual consumer decisions (such as purchasing decisions, social-media habits, or other lifestyle choices) to external and unanticipated financial consequences (such as insurance premiums) could trigger situations where a single innocuous (and entirely lawful) or even clearly laudable choice could result in a disproportionate cascade of unforeseen and unforeseeable (by the consumer) impacts that disrupt an individual’s financial situation and prospects. Again, the above is an expression of concerns and important areas of consideration – not a prescription for a solution, which can only emerge in the course of a thoughtful and multifaceted review process which upholds the autonomy of individual States and is consistent with their legal environments. For the above reasons, it is essential in my view that the white paper preserve the existing references to seeking an understanding of the logical or intuitive connections of specific model attributes (and not just the model as a whole) to the risk of loss (or future expense) that any given model is seeking to predict.


4/6/19


4

Editorial Suggestions The following are suggestions for minor editorial revisions for which I discerned the need upon reading through the exposed draft. I understand that the paper is still in a preliminary draft form, and that there will be various opportunities in the future to make these editorial changes and any others deemed necessary. Section VI (Page 5) The 5th-to-last bullet point should read as follows (revise punctuation): “● Obtain a clear understanding of how the predictive model was integrated into the insurer’s state rating plan and how it improves the state rating plan. (This latter element is only applicable when a new or revised model is introduced into an existing rating plan.)” The 2nd-to-last bullet point should read as follows (revise “each risk characteristics” to “each risk characteristic”): “● Obtain a clear understanding how often each risk characteristic used as input to the model is updated and whether the model is periodically rerun to reflect changes to non-static characteristics.” The last bullet point should read as follows (add “that” after “rating plan”): “● Given an insurer’s rating plan that relies on a predictive model and knowing all characteristics of a risk, a regulator should be able to audit/calculate the risk’s premium without consultation with the insurer.” Section VII (Page 5) First paragraph: In the sentence “Nor is every item on the list intended to be a required for every filing”, change “required” to “requirement”, so that the sentence reads as follows: “Nor is every item on the list intended to be a requirement for every filing.” B. Building the Model - Item B.1.d. – Change “analyses was performed” to “analyses were performed”. - Item B.2.f. – Change “Were there data situations GLM weights were used?” to “Were there data situations in which GLM weights were used?” Thank you again for the opportunity to provide these remarks for the Casualty Actuarial and Statistical Task Force’s consideration. Sincerely,

Mr. Gennady Stolyarov II, FSA, ACAS, MAAA, CPCU, ARe, ARC, API, AIS, AIE, AIAF Lead Actuary, Property and Casualty Insurance, Nevada Division of Insurance


4/6/19



4/6/19


Casualty Actuarial and Statistical (C) Task Force Regulatory Review of Predictive Models

White Paper - Exposure Draft

Table of Contents – Given the overall length of Section VII, a refinement to the table of contents to add subsections to Section VII would be helpful.

Section I:

Paragraph 2: o Reading of sentence, “When that back-and-forth learning is history, filing requirements…” was

unclear. o The word “even” is redundant in the sentence “Hopefully, this paper helps bring more

consistency and even uniformity to the art of reviewing predictive models…”.

Section VI:

I found some confusion and lack of clarity around the inclusion and juxtaposition of two guidance’s: o “Determine that individual input characteristics to a predictive model (and its sub-models) are

not unfairly discriminatory” o “Determine that individual outputs from a predictive model and their associated selected

relativities are not unfairly discriminatory”

The unfairly discriminatory language, from an actuarial context, is often used to convey concern that the prices used do not accurately reflect a reasonable relationship to cost for certain elements of the classification system. There are other considerations actuaries must make to comply with law or regulation in specific jurisdictions, where certain pricing practices or individual policy characteristics are forbidden for use in rating (e.g. use of education level, or use of gender for class rating in auto). I was unclear as to whether these statements were meant as statistical guidance, compliance with statute and regulation, or both?

Fourth to last bullet – change “Determine the extent the model…” to “Determine to what extent the model…” Second to last bullet – change “characteristics” to be “characteristic”.

Section VII:

General Statement – At times the guidance provided in this section appears to be very heavily focused on the statistical process of modeling. For example, review of the sections listed below indicate them being an essential element of the Regulator’s review:

o B.1.i o B.2.d o B.2.e o B.3.b o B.3.d o B.4.c


4/6/19


o B.4.d o B.4.e o B.4.f o B.4.g o B.4.h o B.4.l

While these items are well articulated and would drive a thorough statistical review of the modeling process, my concern would be over the level of requirement put on them. The review, interpretation, and drawing of conclusions from these items would often require an individual who is well trained in actuarial science, statistics, or data science. Is there potential for jurisdictions who do not retain or hire individuals with proper backgrounds in these areas to be overwhelmed or be presented with information that is not valuable to their market regulation process?

Is it reasonable to have more variation in the level of review? For instance,

o “Essential for Regulatory Review”: Review of input variables for compliance with statutes and regulations. That is to say, is the Company including any variables that might be prohibited in the jurisdiction and the model should be withdrawn. Review the target variable of the model. Reviewed to understand whether the model is predicting frequency, pure premium, loss ratio, or some other cost related metric. This would be important for winnowing out loss based models versus life-time value based models for instance. Review of combined output of the model in comparison to performance of current pricing plan. Does the final GLM (in the context of the white paper) perform better at predicting the target variable than the current pricing practice? Review the implementation of the model within the rate plan. Is the rating plan that is actually proposed in line with what the model says, or were other constraints, modifications, or judgments applied post-modeling.

o “Required for Actuarial Opinion to be rendered”: B.1.i B.2.d B.2.e Etc.

o “May be Requested”. A.2. – Consider including a list item on the impact of bias from overlapping data or variables in both sub and primary models in all phases of the model development process (similar to C.1.d) B.1.e – After this section, we believe a clearer call to document how the raw data was divided into the modeling/testing/validation datasets would be helpful. B.1.f – Additional exposition of this element within the comments section would prove valuable. B.3.a –

o This section is the only area where control or offset variables are discussed. Is there any intended review of control or offset variables in the GLM that is implied in the other sections? Similarly, should there be any inquisition into why it makes sense for a variable to be a control or offset within the model. For instance, we sometimes see geodemographic data used as a control variable for territorial rating effects. This process might make sense depending on how the territorial system was developed and how interaction between these two classification systems was established.


4/6/19


o Additionally, this section and others make no specific mention of the use of interaction terms as input variables within the model. Is the investigation of interaction terms intended to be implied by the other items of Section VII?

B.3.d - Within the charge, it should also be requested for the company to document how the results of the PCA process were used within the GLM. B.4.a – It would be helpful to add something in the comments like “it is useful to understand the rationale for dividing the datasets and why the selected approach was deemed most appropriate.” B.4.k – This statement was unclear upon my reading. Is it asking for a descriptive statement for how the concern around overfitting was addressed, and a descriptive statement for how correlation test were performed or considered? Or is it requesting specific results of correlation tests be documented in the filing (Correlation Matrices for instance). C.1.a & C.7.g – Their appeared to be lack of clarity in these sections between the “Importance to Regulator’s Review” and the “comments” section. This comes from the identification that it is “essential” while the comments suggest that it becomes “essential” upon certain conditions? Is this a “may be requested” with the comments dictating when it changes status? C.1 , C.2, & A.2 – There appears to be unclear requirements as to how the regulator should go about reviewing ensemble models. The section on sub-models largely appears to be considering the use of other commercial models as input variables into the GLM (Credit scores, cat models, etc.) While sections do address questions like whether the GLM was performed by-peril or Frequency/Severity vs pure premium, it is unclear whether anything in the guidance would suggest the regulator question how a model ensemble was performed to derive rate factors proposed. To be clear, I am speaking of a simple situation (which we see commonly), where a GLM might be built on a frequency and severity basis, but they are combined through some process to derive a single indicated rate factor. Similarly, we have seen for homeowners where a separate GLM is created for each peril (fire, wind, theft, liability, etc.) which are then combined to generate a single indicated rate factor applicable to all perils. These processes for combining various GLMs are becoming increasingly common.


4/6/19


From: Tomasz Serbinowski <[email protected]> Sent: Thursday, November 1, 2018 4:30 PM To: DeFrain, Kris <[email protected]> Cc: Klausmeier, Tracy <[email protected]> Subject: Preliminary comments on the Predictive Modeling white paper I would like to offer some preliminary comments on the CASTF's white paper "Regulatory Review of Predictive Models". These comments are made on my behalf and are not meant to represent the opinions of the State of Utah. 1. Several places in the draft suggest that the regulator should require an intuitive or logical support for the selected explanatory variables. Why this sounds reasonable, it probably is not. Historically, auto insurers were allowed the use of gender as a rating variable. I don't know of any acceptable logical or intuitive explanation of that. I would venture a guess that an insurer could not offer any explanation other than the data showing that losses vary by gender. 2. Several bullets in Section VI (Best Practices) require that a characteristic used have "an intuitive or demonstrable actual relationship to expected loss or expense". Would the "demonstrable" part be satisfied by showing a correlation between the characteristic and the loss or expense? 3. Would the requirement of "intuitive or demonstrable" relationship of characteristics to the loss or expense extend to sub-models? This could create problems. Number of open accounts may be an input to a credit score model. Credit score may have a demonstrable impact on expected loss. However, the insurer may not the data showing that the number of open accounts have demonstrable impact on the loss or expense. 4. Some bullets in Section VI (Best Practices) require determination that individual characteristics not be unfairly discriminatory. Would that determination entail more than showing "intuitive or demonstrable" impact on the loss or expense? Is the idea to make sure that the impact the characteristic has on the rate commensurates with the loss/expense differential? 5. Eight bullet in Section VI (Best Practices) requires understanding "why the insurer believes this type of model works". Would this be conveyed through goodness of fit statistics, lift charts, etc.? 6. Item A.1.g.in Section VII and 14th bullet in Section VI might need a clarification that the insured's data that might have been included in building the model is not subject to the consumer audit. Only the data used in rating is. Any problems with individual insured's data that was included in building the model would fall under date quality and would have impact on all insureds (because it would impact the final version of the model). 7. Item A.5.a stipulates that raw data could be provided if it is in a format that can be made available to the regulator. What does that mean? What would be the format that could not be provided to the regulator? Is this about the size of data file? 8. Item B.2.e requires an explanation of "the link function distribution". How does that relate to Item B.4.l that requires demonstration that "the GLM assumptions are appropriate"?


4/6/19


Sincerely, -- Tomasz Serbinowski, Actuary Utah Insurance Department State Office Building, Room 3110 | 350 North State Street | Salt Lake City, UT 84114 P: 801-537-9289 | [email protected]


4/6/19


Attachment Four Casualty Actuarial and Statistical (C) Task Force

4/6/19


Draft: 2/11/19

Casualty Actuarial and Statistical (C) Task Force Conference Call January 29, 2019

The Casualty Actuarial and Statistical (C) Task Force met via conference call Jan. 29, 2019. The following Task Force members participated: Steve Kelley, Chair, represented by Phillip Vigliaturo and Connor Meyer (MN); James J. Donelon, Vice Chair, represented by Rich Piazza and Lawrence Stewart (LA); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis (AL); Ricardo Lara represented by Lynne Wehmueller (CA); Michael Conway represented by Rolf Kaumann and Sydney Sloan (CO); Paul Lombardo represented by Qing He (CT); David Altmaier represented by Howard Eagelfeld and Robert Lee (FL); Colin M. Hayashida represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Kevin Fry represented by Judy Mottar (IL); Vicki Schmidt represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Anita G. Fox represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Julie Lederer and Anthony Senevey (MO); Marlene Caride represented by Mark McGill and Carl Sornson (NJ); John G. Franchini represented by Mark Hendrick and Anna Krylova (NM); Barbara D. Richardson represented by Gennady Stolyarov II (NV); Jillian Froment represented by Thomas Botsko (OH); Glen Mulready represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl (OR); Jessica Altman represented by Kevin Clark and Michael McKenney (PA); Raymond G. Farmer represented by Will Davis (SC); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Discussed a Comment Letter on the Statement of Actuarial Opinion Instructions

Mr. Vigliaturo said the Task Force and the Executive (EX) Committee’s ad hoc group exposed revisions to the Property/Casualty (P/C) Statement of Actuarial Opinion instructions on Dec. 15, 2018. The Task Force met Jan. 8, 2019, and Dec. 18, 2018, via conference call to discuss the proposed changes to the instructions and decided to draft a comment letter on the changes proposed. Mr. Vigliaturo said the comments would be made only on the changes the ad hoc group proposed and not on the changes the Task Force proposed. Volunteers from Alabama, Louisiana, Michigan, Minnesota and Nevada, assisted by NAIC staff, drafted a potential comment letter. He said the draft comment letter has two main sections. A short section describes four consensus items. A much longer section includes documentation of the reasons for and against adding American Academy of Actuaries (Academy) membership as a requirement to become an Appointed Actuary. He emphasized that the purpose of the second section is not to debate whether membership should be a requirement but rather to do the best job possible to present both sides of the argument to the ad hoc group. The Task Force discussed the first section. Ms. Lederer said it would be helpful to provide some explanation about why the Task Force would propose to add “member of the Academy” when discussing the Academy’s Casualty Practice Council. She said membership in the Academy is a current requirement for those actuaries who need to be evaluated by the Academy’s Casualty Practice Council. The Task Force agreed. Mary Miller (Academy) agreed, saying the Academy has no authority to evaluate the qualifications of someone who is not a member. If the restrictions on a designation end up including the need to pass a specific exam, Ms. Lederer said the exams and exam content can change over time. Thus, it would be difficult to determine if the restriction would be met when exams were taken many years ago. After the Task Force discussed the intent of the grandfathering provision, Ms. Lederer agreed to propose revised wording to better reflect the discussed intent that actuaries who qualified under the current definition would remain qualified under the new definition. Ms. Miller suggested changing the wording from “four states” supported membership in the Academy as a requirement to “some states.” She said there was no formal vote. Mr. Piazza said he asked twice during the Jan. 8 conference call for any state to express support for explicitly recognizing Academy membership. He said only four states did so. He said that the wording that “four states” supported membership is factual and consistent with the Jan. 8 conference call minutes. Mr. Dyke said there is no formal vote and then not a formal act. He said it would not be accurate to assume the remainder of states are against requiring Academy membership. Mr. Will Davis agreed. Mr. Daniel Davis said some might have still been deciding. The Task Force decided to change “four” to “some.”

Attachment Four Casualty Actuarial and Statistical (C) Task Force

4/6/19


The Task Force discussed the remainder of the comment letter. Mr. Dyke drafted the section to include Academy membership, and Mr. Stolyarov drafted the section not to include Academy membership. Both summarized the content in each section. Mr. Smith said it might be informative to identify the organizations that would meet the requirements in item (iv). Mr. Gennady and Mr. Daniel Davis agreed. Ms. Miller said the descriptor of the Academy being a “lobbying/advocacy” group should be revised. Mr. Stolyarov said he would change the description to “policy advisory” group and would evaluate other language to remove any reference to the Academy taking any position. Mr. Vigliaturo asked Ms. Miller to suggest any additional revisions prior to the Feb. 12 conference call. Mr. Vigliaturo asked the drafters to make revisions based on the discussion. He said the Task Force will consider adoption of the revised comment letter on its Feb. 12 conference call. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\1-29 CASTF min.docx

Attachment Five Casualty Actuarial and Statistical (C) Task Force

4/6/19


Draft: 1/29/19

Casualty Actuarial and Statistical (C) Task Force Conference Call January 8, 2019

The Casualty Actuarial and Statistical (C) Task Force met via conference call Jan. 8, 2019. The following Task Force members participated: James J. Donelon, Chair, represented by Rich Piazza (LA); Steve Kelly, Vice Chair, represented by Phillip Vigliaturo (MN); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Michael Conway represented by Deborah Batista, Mitchell Bronson and Sydney Sloan (CO); Paul Lombardo represented by Susan Andrews and Qing He (CT); David Altmaier represented by Howard Eagelfeld and Robert Lee (FL); Gordon I. Ito represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Karin Zosel represented by Reid McClintock (IL); Ken Selzer represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Patrick M. McPharlin represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Gina Clark and Julie Lederer (MO); Marlene Caride represented by Carl Sornson (NJ); Barbara D. Richardson represented by Gennady Stolyarov (NV); Maria T. Vullo represented by Sakman Luk (NY); Jillian Froment represented by Brad Schroer (OH); John D. Doak represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Jessica Altman represented by Kevin Clark and Michael McKenney (PA); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka, Brian Ryder and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Adopted the Statistical Reports

Via e-votes, the Task Force adopted all four statistical reports: 1) the Report on Profitability by Line by State (Profitability Report); 2) the Competition Database Report (Competition Report); 3) the Dwelling Fire, Homeowners Owner-Occupied, and Homeowners Tenant and Condominium/Cooperative Unit Owner’s Insurance Report (Homeowners Report); and 4) the Auto Insurance Database Report (Auto Report).

2. Discussed Revised Statement of Actuarial Opinion Instructions

Mr. Piazza said the Task Force should continue discussion from its Dec. 18, 2018, conference call regarding the revised Statement of Actuarial Opinion instructions jointly exposed by the Executive (EX) Committee’s ad hoc group and the Task Force. He said comments are due Feb. 15. He said the Task Force would decide whether to communicate to the ad hoc group on its proposed changes. He said a few of the items discussed on the prior call include: 1) whether to change the items in the list in the definition of qualified actuary, including whether to change item (i) to change the “or” to “and;” whether to note that the NAIC Accepted Actuarial Designations and the associate grandfathering clause should say “including restrictions”; and 3) whether to require membership in the American Academy of Actuaries (Academy). Mr. Piazza said the purpose of the conference call is for Task Force members to discuss the proposal. Mr. Piazza said membership in the Academy is not required to be a qualified actuary under current rules, except those actuaries who must be approved by the Academy’s Casualty Practice Council must be members of the Academy. He said that there is still the ability to be evaluated by the Academy’s Casualty Practice Council in the proposed instructions, but that wording is missing the requirement to be an Academy member. He suggested that membership requirement be reinserted into the proposed instructions when referring to the Casualty Practice Council’s process. Mr. Piazza asked if any Task Force member objected. No Task Force member objected. Mr. Stolyarov said Nevada maintains a list of commissioner-approved qualified actuaries for captives. He said there are some optional ways an actuary can qualify. He said in addition to other items in the review, the actuary must be a Fellow of the Casualty Actuarial Society (FCAS), a Fellow of the Society of Actuaries (FSA), a member in good standing of the Academy (MAAA) or a person who has otherwise demonstrated competence in the evaluation of loss reserves to the commissioner. Mr. Stolyarov said if membership in the Academy is specifically in the instructions, it should be an option. He said he supports the current wording in (iv) because it is consistent with the status quo and the law in Nevada. Mr. Dyke said he supports requiring membership in the Academy in the definition of a qualified actuary. He said that could be added to the wording in (iv) or completely replace that wording in (iv). He said not having membership in the Academy deviates from what is typically in model laws and what is in the qualified actuary definition for life and health actuarial opinions. He said the current instructions deviate from almost every other standard for development of model laws. Mr. Dyke said the joint

Attachment Five Casualty Actuarial and Statistical (C) Task Force

4/6/19


qualified actuary task force tried to harmonize the qualified actuary definitions across annual statement blank types. He said membership in the Academy would harmonize the requirements. He said the federal government uses membership in the Academy in laws (e.g., the Affordable Care Act [ACA]). He said at least six states specifically require Academy membership by statute. He said the NAIC benefits from the Academy’s support, including issues regarding professionalism that occurred a few years ago, qualifications standards, etc. Mr. Dyke said membership in the Academy needs to be required. Mr. Piazza said he understands that state laws might have membership requirements, but the aim is to address what the instructions should say now. He said the aim is not to harmonize with the life and health instructions, although the actuarial groups tried to do that years ago and got nowhere. Mr. Piazza said he is not sure that membership in the Academy for the Appointed Actuary adds value to the instructions for purposes of the definition. Mr. Stolyarov agreed. He said the definition of a qualified actuary in the NAIC accreditation standards does not require membership in the Academy. He said differences or distinctions in the model laws might reflect differences in practice. He said the actuarial opinion instructions should not be more stringent than the model law. Mr. Dyke said the basic education pathway is different from being a licensing concept, just like getting education to be a doctor is separate from becoming licensed. He said membership in the Academy would not change the basic education concepts. Mr. Stolyarov said state law does not address general qualification standards for an actuary. Mr. Eagelfeld said the Academy performs a vital function and does good work, but he views the organization more like the American Medical Association (AMA) than a licensing board to practice medicine. He said if a requirement is to be a member of the Academy, then the Academy membership is enshrined as a necessity for a fully functioning Appointed Actuary. He said some people choose not to join the Academy for various reasons. Mr. Piazza asked if any Task Force member would like to see membership in the Academy as a requirement to be an Appointed Actuary. Alabama, Maine, Michigan, and Oregon supported the requirement. Ms. Lederer recused herself because the Missouri law requires membership. Mr. Piazza said another proposed change is for the item (i) to be an “and” instead of an “or.” Mr. Stolyarov and Ms. Lederer disagreed with making the proposed change. Mr. Piazza asked if any Task Force member objected. No Task Force member objected. Mr. Piazza said another proposed change is to add wording about appropriate restrictions in reference to the grandfathering provision. Mr. Stolyarov said that is consistent with the plan for the definition of NAIC Accepted Actuarial Designation to have noted restrictions. Mr. Piazza said the proposed change would affect both the item (ii) in the qualified actuary definition and the grandfathering wording in the NAIC Accepted Actuarial Designation definition. Mr. Piazza asked if any Task Force member objected. No Task Force member objected. Mr. Piazza said the actual restrictions will be insert upon completion of the NAIC’s Educational Standards and Assessment Project. No additional changes were proposed for discussion. Mr. Piazza asked if the Task Force wants to draft a letter to the Executive (EX) Committee’s ad hoc group. Mr. Gennady said a letter should address consensus changes. He suggested the areas of disagreement should be handled in individual or small group letters. Ms. Mottar said the letter could address consensus items and then explain what else was discussed. She said she supports a letter because the instructions are typically drafted by the Task Force and, therefore, the Task Force should weigh in on another group’s proposal. Ms. Elliott agreed with Ms. Mottar and said there could be explanation of both sides to an issue with a statement that consensus was not reached. The Task Force decided to include consensus items and both sides of the issue in the letter. Mr. Piazza asked if any Task Force member objected to a Task Force letter containing consensus items and both sides of the item where the Task Force could not reach consensus. No Task Force member objected. Volunteers agreed to draft a letter for Task Force consideration prior to the Feb. 15 comment deadline. Mary Miller (Academy) said she finds value in Academy membership as a qualification for signing opinions. She said that qualification standards provide for gaps in learning and that they do not have to be put in instructions. She asked for the Academy’s membership application to be shared with the Task Force. She said the Casualty Actuarial Society (CAS) and Society of Actuaries (SOA) are international organizations. If the Academy is not referenced in item (iv), then there is no way to know whether a member of the CAS in another country has knowledge of U.S. laws and regulations. She said the Academy membership form addresses that. She said she would send some of that information to the ad hoc group. She said there is confusion for a company to know whether the items in (iv) are met but very easy for the company and a board of directors to understand whether an actuary is a member of the Academy. She suggested the actuaries on the Task Force with only ratemaking responsibilities and experience might want to discuss the issues with members who have experience in financial regulation and do actuarial opinion analysis work. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\1-8 CASTF min.docx

Attachment Six Casualty Actuarial and Statistical (C) Task Force

4/6/19


Draft: 1/10/19

Casualty Actuarial and Statistical (C) Task Force Conference Call

December 18, 2018 The Casualty Actuarial and Statistical (C) Task Force met via conference call Dec. 18, 2018. The following Task Force members participated: James J. Donelon, Chair, represented by Rich Piazza (LA); Jessica Looman, Vice Chair, represented by Phillip Vigliaturo (MN); Lori K. Wing-Heier represented by Mike Ricker (AK); Jim L. Ridling represented by Daniel J. Davis and Jerry Workman (AL); Michael Conway represented by Deborah Batista, Mitchell Bronson and Sydney Sloan (CO); Paul Lombardo represented by Susan Gozzo Andrews and Qing He (CT); David Altmaier represented by Robert Lee (FL); Gordon I. Ito represented by Randy Jacobson (HI); Doug Ommen represented by Travis Grassel and Andria Seip (IA); Karin Zosel represented by Reid McClintock and Shannon Whalen (IL); Ken Selzer represented by Nicole Boyd (KS); Eric A. Cioppa represented by Sandra Darby (ME); Patrick M. McPharlin represented by Kevin Dyke (MI); Chlora Lindley-Myers represented by Gina Clark and Julie Lederer (MO); Marlene Caride represented by Carl Sornson (NJ); Barbara D. Richardson represented by Gennady Stolyarov (NV); Maria T. Vullo represented by Sakman Luk (NY); Jillian Froment represented by Brad Schroer (OH); John D. Doak represented by Andrew Schallhorn (OK); Andrew Stolfi represented by David Dahl and Ying Liu (OR); Jessica Altman represented by Kevin Clark and Michael McKenney (PA); Kent Sullivan represented by J’ne Byckovski, Brock Childs, Nicole Elliott, Miriam Fisk, Eric Hintikka, Brian Ryder and Jennifer Wu (TX); and Mike Kreidler represented by Eric Slavich (WA). 1. Discussed Revised Statement of Actuarial Opinion Instructions

Mr. Piazza said the Executive (EX) Committee’s ad hoc group and the Task Force jointly exposed revised Statement of Actuarial Opinion instructions on Dec. 15 for a 60-day public comment period ending Feb. 15 (Attachment ___). Mr. Piazza said the ad hoc group working on the definition of a qualified actuary is comprised of insurance commissioners from Alabama, Louisiana and Maine. Kris DeFrain (NAIC) presented the proposed changes and explained which changes were proposed by which group. Mr. Slavich asked about the definition of NAIC-Accepted Actuarial Designation. He said it appears that the organizations must meet all requirements to be accepted, with whatever restrictions are listed. Mr. Piazza said the Executive (EX) Committee ad hoc group is conducting a study and will determine which organizations are approved and if there are any restrictions. Mr. Stolyarov said he supports the proposed wording, including the new definition of the NAIC-Accepted Actuarial Designation and the proposed grandfathering approach. Mr. Dyke said the NAIC-Accepted Actuarial Designation seems to be adding complexity to the definition. He suggested identifying the procedure and explaining how other organizations can be accepted. He suggested the Task Force submit a comment letter as a group. He said there are different perspectives and that it might be helpful to see what a collective group of actuaries would say. He said it might add credibility and insight to the process. Mr. Piazza said it might be easier if individuals submit their comments separately. He said if there is a Task Force response, he would not draft the letter given his insurance commissioner is on the ad hoc group. Mr. Chou agreed with Mr. Dyke. Mr. Stolyarov said the exposure contains proposed wording from the Task Force, so it would seem unusual for the Task Force to draft a comment letter. He suggested individuals submit comment letters. He said differences in views should be addressed. He said he prefers the current wording of the definition of NAIC-Accepted Actuarial Designation and would not want that changed to describe the process to become an NAIC-Accepted Actuarial Designation. He said the process itself can be kept outside of the definitions. Mr. Dyke said the instructions can be changed annually, so it would not seem to be a problem to update the process as needed. Ms. Lederer asked whether the actuarial designations in the grandfathering statement are still under consideration. Mr. Piazza said the ad hoc group is currently reviewing the designations to determine which will be accepted. He said the designations listed in the grandfathering part would be the same as the accepted designations. Mr. Davis questioned the grandfathering of old exams that have been changed in order to meet the requirements. Mr. Stolyarov said the requirements of the standards will be known and that the actuaries will be held to having the combined experience and education that is needed. Mr. Davis suggested part 3 should be an “or” statement instead of an “and” statement when listing the ways someone would obtain knowledge. Mr. Piazza said more than half of the knowledge is probably learned outside of basic education. He said basic education will never be enough, alone, to provide enough education and knowledge for an actuary to provide a Statement of


4/6/19


Actuarial Opinion in the annual statement. Mr. Davis agreed and said it would be more clear to use an “and” statement. Mr. Dyke said with basic education and experience working together, the new instructions might be more complex than needed. Mr. Dyke asked for an example of restrictions on the actuarial designations. Mr. Piazza said restrictions might be that an Associate of the Casualty Actuarial Society (CAS) must also have passed the CAS Exam 7, which covers advanced reserving topics. Ms. DeFrain said the Fellow of the Society of Actuaries (SOA) might have a restriction that the general insurance track must be passed. Mr. Dyke said that given the complexity of the definition, especially with the restrictions on the actuarial designations, it might be worthwhile to consider making the whole definition simpler by requiring membership in the American Academy of Actuaries (Academy). He said that the Academy designation is a generally accepted designation in NAIC model laws and other laws, generally accepted by the federal government, and that five or six states require membership in the Academy. He said it is a simple objective measure of membership. Mr. Stoylarov said he disagrees with requiring membership in the Academy. He said a non-Academy member must meet all the requirements in Item 4 within the Qualified Actuary definition, essentially addressing the substance of what an Academy membership would achieve. He said not being an Academy member does not exempt someone from those requirements. The only things missing is paying dues to the Academy and getting services such as webinars, publications and networking activities. He said they are good services, but they are not essential to the work of the Appointed Actuary. Mr. Slavich asked about the restrictions to the designations and whether those same restrictions would apply in the grandfathering provisions. Mr. Piazza said it would be the same in both places. Mr. Slavich suggested that the grandfathering sentence should include “with noted restrictions” or similar wording. Mary Miller (Risk & Regulatory Consulting) said that the qualified actuary discussion is not a Task Force definition and that the Task Force has not had a full discussion of the issues that surround the definition. She said it was previously an NAIC position that the education provided by the SOA general insurance (GI) track was not sufficient to be included, yet now the NAIC would be grandfathering those. Ms. Miller said the qualification standards provide for a mechanism for someone who had deficiencies in basic education to get that basic education through experience and to have that documented. She said the need for grandfathering can get into all sorts of permutations that are not necessary because the qualifications already provide a mechanism to address the deficiencies. On behalf of the Academy, she said there are people who do not understand the history and that the CAS and SOA created the Academy more than 50 years ago to address public policy issues and be a single place for state insurance regulators to go. She said the original draft of property/casualty (P/C) instructions, when they were first being created, required the Academy membership. She said that at the time, there were issues with non-casualty actuaries signing California workers’ compensation opinions, so they decided to require the CAS membership instead of the Academy membership. She said 30 years ago, the CAS was not as international as it is today. As well, when the SOA announced their general insurance track, the SOA said it was for their international students to have a full array of topics in their examinations. She said she does not want to see the definition as a passport to international actuaries who do not understand U.S. laws and regulations and did not pass an examination on U.S. laws and regulations. She said the Academy is the only organization that requires someone requesting membership to demonstrate familiarity with U.S. laws and regulations if they are a non-resident or a resident alien of fewer than three years. She said she is not sure who made the recommendations to the Executive (EX) Committee, but she hopes they looked in the mirror and made sure they met the qualification standards for giving that opinion to the ad hoc group of insurance commissioners. She said they were being relied on in their role as actuaries. She said the initial exposure of the definition included a requirement to be a member of the Academy, and neither the CAS nor SOA objected to such. She said it is a disservice to not include membership of the Academy. As a former state insurance regulator, Ms. Miller said insurance commissioners have always had the power to allow someone who is not a qualified actuary to sign an opinion. She said the revised instructions make that more prominent and could encourage that. She said it does not serve the public good and creates a lower standard. Ralph Blanchard (Travelers) said he cannot find “experience period” anywhere other than in the qualification standards and asked whether the NAIC would be asking those standards to be modified. Mr. Piazza said the experience period has been discussed. He said the NAIC would not be asking the Academy to change the qualification standards. He said the combined basic education, experience and continuing education for an appointed actuary would need to be sufficient for a given line of business or company structure. He said at one point, the drafts went beyond relying on the qualification standard, but that was taken out. Mr. Blanchard asked whether there is modification to the U.S. qualifications experience period requirement. Mr. Piazza said there is no suggestion for the qualification standard to be revised in that regard.


4/6/19


Mr. Blanchard said that the wording of “being subject to the ABCD” needs to be tweaked given the Actuarial Board for Counseling and Discipline (ABCD) only provides recommendations to the other actuarial organization to decide the action to take, if any. Craig Hanna (Academy) asked if there was formal action by the Executive (EX) Committee. Mr. Piazza said the Executive (EX) Committee’s ad hoc group has exposed the document for a public comment period. He said the ad hoc group will make a recommendation to the Executive (EX) Committee. Ms. DeFrain said there would be a public hearing March 22 via conference call. Mr. Dyke asked if the Executive (EX) Committee would have any exposure of the proposal. Ms. DeFrain said the current project timeline does not include another exposure period. Mr. Dyke asked about impediments to the Task Force issuing a comment letter. Mr. Piazza said Task Force members could draft a letter, although he would not be the main drafter given his role with the ad hoc group. Mr. Dyke asked Task Force members to contact him to draft a group comment letter. Mr. Piazza said there would be time on the Task Force’s Jan. 8, 2019, call to discuss a draft comment letter or to attempt to reach some consensus for a Task Force comment letter. Having no further business, the Casualty Actuarial and Statistical (C) Task Force adjourned. W:\National Meetings\2019\Spring\TF\CasAct\12-18 CASTF min.docx

casualty actuarial and statistical (c) task force · the motion passed unanimously. 2. adopted the...

Documents