thinking strategically about and practically about ... · 6 things any program need to know… 1....

Thinking Strategically About and Practically About BioSafety Training

2

Today

1. Think of evaluation and planning together2. Gathering “process use” insights3. Understanding how to set an evaluation

focus

3

Why We Care! The Continuous Quality Improvement Cycle

Planning—What actions will best reach our goals and objectives.

Performance measurement— How are we doing?

Evaluation—Why are we doing well or poorly?

What do we do?

Why are we

doing well or poorly?

How are we

doing?

How do we do it?

4

6 Things Any Program Need to Know…1. Big “need” to which it is contributing2. “Accountable” outcome3. ST outcomes that accountable

outcome4. Outputs: “Strong” activities that ST

outcomes, AND what “strong” means5. Moderators: Contextual factors that

help/hobble

55

A (Simple) Roadmap/Logic Model Helps!

Activities Intermediate Outcomes

Short-term Outcomes

Long-term Outcomes/

Impacts

What the program does…

Who or what will changebecause of the program…

6

Payoff for Evaluation

1. Accountable outcome is key one to measure

2. What ST/MT drive it, IF we don’t think we’ll get there?

3. Outputs=Process measures for our activities

4. Moderators=Ability to explain (lack of) progress

7

BUT, Immediate Payoff in Process Use…What do we need to do NOW to:1. Improve ability of activities to drive

accountable outcomes (Outputs)2. ‘Own” factors in the larger environment

and address them so they don’t get in our way (Moderators)

3. OR, ACCOMMODATE to factors in the environment that are “laws of gravity” (Moderators)

88

(Simple) Logic Models are Roadmaps for ANY Effort

Activities Intermediate Outcomes

Short-term Outcomes

Long-term Outcomes/

Impacts

What the program does…

Who or what will changebecause of the program…

Big “need” to which program is contributing

99

(Simple) Logic Models are Roadmaps for ANY Effort

Lab Safety Training

3: Behaviors sustained on the job

1: Happy trainees

4: Org change—Culture of lab safety

What the program does… Who or what will change

because of the program…

2: KAB and skills

Big “need” to which Lab Safety Training effort is contributing

1010

(Simple) Logic Models are Roadmaps for ANY Effort

Activities Intermediate Outcomes

Short-term Outcomes

Long-term Outcomes/

Impacts

What the program does…

Who or what will changebecause of the program…

“Accountable” outcome

1111

(Simple) Logic Models are Roadmaps for ANY Effort

Lab Safety Training

3: Behaviors sustained on the job

1: Happy trainees

4: Org change—Culture of lab safety

What the program does… Who or what will change

because of the program…

2: KAB and skills

“Accountable” outcome for Lab Safety Training

effort

12

(Selected) Behaviors Sustained on Job Proper use of PPE Proper use of BSC Conduct formal risk assessments Ability to recognize hazards in any

situation Complete appropriate documentation Proper waste management

1313

Simple Logic Models as Roadmaps

Activities Inputs Outputs Intermediate Outcomes

Short-term Outcomes

Long-term Outcomes/

Impacts

What the program does…

Who or what will changebecause of the program…

What the program needs…

Outputs: What “strong” activities means

1414

(Simple) Logic Models are Roadmaps for ANY Effort

Lab Safety Training

3: Behaviors sustained on the job

1: Happy trainees

4: Org change—Culture of lab safety

What the program does… Who or what will change

because of the program…

2: KAB and skills

Outputs: What “strong” activities means for Lab Safety Training efforts

15

(Selected) Key Attributes of Activities to Ensure Outcomes are Reached Content of training should explain why it

should be done Focus on actual work practices/evidence-

based work practices Real-time experience during the training Include reinforcing materials Personalizing content to the situation of

the person as opposed to general regs

1616

Simple Logic Models as Roadmaps

Activities Inputs Outputs Intermediate Outcomes

Short-term Outcomes

Long-term Outcomes/

Impacts

What the program does…

Who or what will changebecause of the program…

What the program needs…

Context and Assumptions

External factors that influence getting to outcomes

5. What “strong” activities means

Killer assumptions: Outside factors that can

help or hobble

17

(Selected) Moderating Factors, Even if Our Training is Great! Supervisors’ attitudes and expectations Lack of resources for quality and safety Lack of reinforcement (decay of KAB) Time/safety trade-off, particularly during

public health emergency (time constraint) Competing priorities

18

(Selected) Actions We Can Take to Mitigate Moderating Factors Supervisors’ attitudes and expectationsAdd consistent messages in supervisor

trainingPush others to create SOPs or policies to

induce supervisor cooperation. Lack of reinforcement (decay of KAB)Refresher training; written materials

19

(Selected) Actions We Can Take to Mitigate Moderating Factors (cont’d) Time/safety trade-off, particularly during

public health emergency (time constraint) Competing prioritiesHighlight consequences of not doing things

right Lack of resources for quality and safetyArray solutions in class from

cheapexpensivePush others for more budget for these efforts

20

Payoff for Evaluation1. Accountable outcome is key one to

measure1. % of trainees maintaining (selected)

behaviors on the job 3/6/12 months post-training?

2. What ST/MT drive it, IF we don’t think we’ll get there?

1. % trainees exhibiting appropriate knowledge and skills on key competencies post-training

3. Outputs=Process measures for our activities

21

Payoff for EvaluationOutputs=Process measures for our activities1. Degree to which the trainings:Focus content on why it should be doneFocus on actual work practices/evidence-

based work practicesProvide real-time experience during the

training Include reinforcing materialsAre personalized to the situation of the person

as opposed to general regs

22

Payoff for EvaluationModerators=Ability to explain (lack of) progress Did supervisors’ exhibit consistent

attitudes and expectations? Were there sufficient resources for quality

and safety improvements? Did KAB decay over time? Did trainees see safety as important even

in the face of time constraints and competing priorities?

23

Questions?