Upcoming Caveon Events

• Caveon Webinar Series: Next session, September 18

Improving Testing with Key Strength Analysis

• EATP – September 25-27 in St. Julian’s, Malta.

– Caveon’s John Fremer and David Foster presenting the session:

What are we Accountable For? Security Standards and Resources for High

Stakes Testing Programs

• Handbook of Test Security – Now Available. We will share a discount code

at the end of this session.

Caveon Online

• Caveon Security Insights Blog

– http://www.caveon.com/blog/

• twitter

– Follow @Caveon

• LinkedIn

– Caveon Company Page

– “Caveon Test Security” Group

• Please contribute!

• Facebook

– Will you be our “friend?”

– “Like” us!

www.caveon.com

I Think I’m A Clone Now.

Item Cloning Strategies for Secure Testing

Lisa Hirtzinger, CIA, CRMA, CCSA, MBA, M.A

Director, Exam Development and Analysis

The Institute of Internal Auditors (IIA)

August 21, 2013

Caveon Webinar Series:

Patrick Martin

Assessment Project Manager

Test Development Services

Caveon

Our Story: The IIA

(Institute of Internal Auditors)

• A flagship examination: Certified Internal Auditor (CIA)

• Three-part multiple choice exam

• Computer based testing offered year round

• 105 countries; 17 languages

• Approximately 110,000 certified over 40 years

• Also four, one-part specialty exam (in fewer languages)

IIA Timeline

• In 2008 The IIA transitioned our exams to CBT

• In 2011 we renewed our focus on exam maintenance

post CBT transition

• In 2012 we researched cloning as a strategy

• In 2013 our efforts are focus on item writer training

Why Cloning Makes Cent$

The decision to try cloning was based on:

• Cost of Development

• Content Security

• Exam Relevancy and Validity

• Pre-Testing Strategies

• Leveraging Technology

• Use of Staff / Resources

The Process

• Partnered with Caveon professional item

writers

• Shared our core reference materials and

our objectives for cloning/refurbishing

• Provided the specific items we wanted

cloned

• Had Caveon writer attend in person review

with SMEs for comments and feedback

• Going forward assigned items to

professional writer; and had virtual SME

review/approval

Tips for Buy In

• Focus on augmenting the work of

your current valuable SMEs

• Allows focus on other

strategic priorities

• Host a working session with a

professional item writer and your

SMEs

• Share writing tips from the

professional writer with SMEs

• Ensure two-way communication

The Results

• Our SMEs found the process extremely

valuable and learned from it

• We increased our inventory of items for

le$$ than new development

• This changed my personal bias that

only SMEs from the profession can

write items

The Resulting Test Items

• Items have higher level of consistency and

clarity for exam candidates

• Enhances the overall professionalism

of the exam

• SMEs are great SMEs; but rarely

excellent technical writers

• In 2013 we are also partnering with

Caveon for additional item writer

training so we have an even better

pipeline of items to clone

• The overall objective is an item writing

maturity model (from trash to treasure

to treasure x 2)

Cloning Strategies

New Correct Answer

A trainer notices that during a heavy

barbell squat, the client’s upper back rolls

over as the client comes out of the bottom

of the movement.

Which verbal cue should the trainer deliver

to help correct the problem?

A: “Chest Up!”

B: “Eyes Up!”

C: “Shoulders Up!”

D: “Head Up!”

New Correct Answer

A trainer notices that during a heavy

barbell squat, the client’s upper back

rolls over as the client comes out of the

bottom of the movement.

Which verbal cue should the trainer

deliver to help correct the problem?

A: “Chest Up!”

B: “Eyes Up!”

C: “Shoulders Up!”

D: “Head Up!”

A trainer notices that during a heavy

goblet squat, the client tends to get

momentarily stuck at the bottom of

the movement.

Which verbal cue should the trainer

deliver to help correct the problem?

A: “Chest Up!”

B: “Eyes Up!”

C: “Shoulders Up!”

D: “Head Up!”

New Correct Answer

A woodworker is building an adjustable-

shelf bookcase for a client who has

requested that the frame be assembled

with miter joints and that no screw holes or

nail holes be visible when the cabinet is

finished.

Which joinery method should the

woodworker use to assemble the cabinet

as requested?

A: biscuits

B: pocket screws

C: dadoes

D: dovetails

New Correct Answer

A woodworker is building an adjustable-

shelf bookcase for a client who has

requested that the frame be assembled

with miter joints and that no screw holes or

nail holes be visible when the cabinet is

finished.

Which joinery method should the

woodworker use to assemble the cabinet

as requested?

A: biscuits

B: pocket screws

C: dadoes

D: dovetails

A woodworker is building an adjustable-

shelf bookcase for a client who has

requested that the frame be assembled

with simple butt joints and that no screw

holes or nail holes be visible when the

cabinet is finished.

Which joinery method should the

woodworker use to assemble the cabinet

as requested?

A: dowels

B: brads

C: tenons

D: rabbets

Flipping the Answer into the Stem

An analyst is evaluating a stock for possible inclusion in a portfolio designed to provide stable income.

Which of the following metrics should the analyst ascertain in order to ensure that the stock will meet the requirement of providing stable income?

A: payout ratio

B: price to earnings

C: free cash flow

D: book value

Flipping the Answer into the Stem

An analyst is evaluating a stock for

possible inclusion in a portfolio

designed to provide stable income.

Which of the following metrics should

the analyst ascertain in order to

ensure that the stock will meet the

requirement of providing stable

income?

A: payout ratio

B: price to earnings

C: free cash flow

D: book value

An analyst is evaluating a stock for possible inclusion in a portfolio designed to provide stable income.

For what reason should the analyst determine the payout ratio?

A: It is an indicator of the company’s ability to maintain the dividend payment.

B: It is a measure of the expected return over a 5 to 10 year time period.

C: It is a gauge for determining whether the stock is currently over or undervalued.

D: It is a benchmark for measuring the performance of the stock going forward.

Changing the Shape of the Stem

Which two statements are true about brewing green tea? (Choose two.)

A: It should be brewed at a maximum temperature of 175°B: It should be brewed at a maximum temperature of 195°C: It should be allowed to steep for 1-3 minutes.

D: It should be allowed to steep for 3-5 minutes.

Changing the Shape of the Item

Which two statements are true

about brewing green tea?

(Choose two.)

A: It should be brewed at a

maximum temperature of 175°

B: It should be brewed at a

maximum temperature of 195°

C: It should be allowed to steep

for 1-3 minutes.

D: It should be allowed to steep

for 3-5 minutes.

Simple tea uses electric kettles to brew a variety of teas to order. A customer has requested a large plain green tea.

Which brewing instructions should the barista follow?

A: Steep for 1-3 minutes at a temperature of 175°B: Steep for 3-5 minutes at a temperature of 195.

C: Steep for 6-8 minutes at a temperature of 205°D: Steep for 8-10 minutes at a temperature of 215°

Rewording or Keyword Swapping

Read the following passage

The film was criticized for its dull writing,

wooden acting, and for being too long.

Which revision corrects the error in the

sentence without changing the original

meaning?

A. The film was criticized for its dull

writing, wooden acting, and

excessive length.

B. The film was criticized for its dull

writing, wooden acting and for being

too long.

C. Having dull writing, wooden acting,

and being too long, the film was

criticized.

D. Audiences criticized the film for its

dull writing, wooden acting, and for

being too long.

Rewording or Keyword Swapping

Read the following passage.

The film was criticized for its dull writing,

wooden acting, and for being too long.

Which revision corrects the error in the

sentence without changing the original

meaning?

A. The film was criticized for its dull

writing, wooden acting, and

excessive length.

B. The film was criticized for its dull writing,

wooden acting and for being too long.

C. Having dull writing, wooden acting, and

being too long, the film was criticized.

D. Audiences criticized the film for its dull

writing, wooden acting, and for being

too long.

Read the following passage.

The employee was reprimanded for missing

deadlines, ignoring emails, and late arrivals.

Which revision corrects the error in the

sentence without changing the original

meaning?

A. The employee was reprimanded for

missing deadlines, ignoring emails,

and arriving late.

B. The employee was reprimanded for

missing deadlines, ignoring emails and

late arrivals.

C. After missing deadlines, ignoring emails,

and late arrivals, the employee was

reprimanded.

D. Management reprimanded the employee

for missing deadlines, ignoring emails,

and late arrivals.

Tips for Getting Stronger Clones from non-SMEs

• Give items with the potential for new correct answers

the priority.

• If applicable, provide a reference with each item.

• Require item writers to provide rationale statements

for every option.

Example Rationale

An analyst is evaluating a consumer goods stock for possible inclusion in a portfolio designed to provide stable income.

Which of the following metrics should the analyst ascertain in order to ensure that the stock will meet the requirement of providing stable income?

A: payout ratio

B: price to earnings

C: free cash flow

D: book value

Answer Rationale

A: This answer is 100% correct because the

payout ratio measures the amount of earnings

paid to shareholders as dividends. If the ratio

is too high, the dividend may be unsustainable.

B: This answer is 100% incorrect because the

price to earnings ratio measures how

expensive a stock is relative to its earnings

and has no bearing on dividend payments.

C: This answer is 100% incorrect because free

cash flow measures the cash available after

capital expenses. In some cases analysts will

look at the FCF payout ratio to evaluate the

dividend stability of a utility company, but alone

it doesn’t provide enough information.

D: This answer is 100% incorrect because

book value is the total assets minus total

liabilities, which is more useful when

evaluating whether the stock is expensive.

HANDBOOK OF TEST SECURITY

• Editors - James Wollack & John Fremer

• Published March 2013

• Preventing, Detecting, and Investigating Cheating

• Testing in Many Domains

– Certification/Licensure

– Clinical

– Educational

– Industrial/Organizational

• Don’t forget to order your copy at www.routledge.com

– http://bit.ly/HandbookTS (Case Sensitive)

– Save 20% - Enter discount code: HYJ82

THANK YOU!

- Follow Caveon on twitter @caveon

- Check out our blog…www.caveon.com/blog

- LinkedIn Group – “Caveon Test Security”

Lisa Hirtzinger, CIA, CRMA, CCSA, MBA, M.A

Director, Exam Development and Analysis

The Institute of Internal Auditors (IIA)

lisa.hirtzinger@theiia.org

Patrick Martin

Assessment Project Manager

Test Development Services

Caveon

patrick.martin@caveon.com