Download - OR-george schmid
-
7/29/2019 OR-george schmid
1/50
How to Do Operational
Research
George Schmid, M.D., M.Sc.
World Health Organization/Centers for
Disease Control and Prevention
-
7/29/2019 OR-george schmid
2/50
Format Informal
Practicalif you want to hear about modeling (alegitimate, important part of OR), run out the door now
The session is for you, and us
You will forget 90% of what you hear today. So, we've
given you materials on four CD-ROMs: Designing HIV/AIDS Intervention Studies (and other materials)
(Pop Council);
Training course materials in operations research (Pop Council)
Framework for Operations and Implementation Research in Health
and Disease Control Programmes (and other materials) (GlobalFund and WHO)
Other materials (note the FHI guide to qualitative research)
-
7/29/2019 OR-george schmid
3/50
Doing Operations andImplementation Research
-
7/29/2019 OR-george schmid
4/50
Research Techniques
Systematic data collection
Qualitative techniques
Quantitative techniques Surveys, experiments, focus groups.
-
7/29/2019 OR-george schmid
5/50
The OR Process
Problem identification
Solution generation
Solution testing Results dissemination
Results utilization
-
7/29/2019 OR-george schmid
6/50
OR Requires Collaboration
Managers:
responsible for decision parameters anddesired outcome
Researchers:
responsible for recommending andimplementing research techniques
-
7/29/2019 OR-george schmid
7/50
Taking My Problem Into
Operational Research
-
7/29/2019 OR-george schmid
8/50
Q1.What Proportion of Individuals
Potentially Helped by Bed Nets are
Using Them?1. 2%
2. 7%
3. 14%
4. 24%
Morel CM et al. BMJ 2005
-
7/29/2019 OR-george schmid
9/50
Taking My Problem Into
Operational Research
-
7/29/2019 OR-george schmid
10/50
First, You Must Recognize a Problem
Wanting to know
Wanting to act
Having the ability to act
-
7/29/2019 OR-george schmid
11/50
Q2. Who Typically Identifies Problems
for OR? (more than one answer
possible)1. The patient
2. The programme staff
3. The programme manager
4. The District manager
5. National staff
-
7/29/2019 OR-george schmid
12/50
Zambia Example
Problem: TB patients are not being
tested for HIV
Observation: "TB Corners" where TBpatients are seen are in open hallways
Intervention: Provide counselling with
privacySome issues simply need some common
sense, and initiative
-
7/29/2019 OR-george schmid
13/50
Identifying Problems
The role of anyone who works in the
programme!
But, primarily, the role of the manager Identifying problems requires a team
approach
Managers must be present in their programmes Managers must have good relationships with
their staff
-
7/29/2019 OR-george schmid
14/50
You've Identified a Potential Problem
Patients may not be taking their ART
People may not be using condoms
Patients may not be returning forfollow-up visits
Your health care workers seem to
have high rates of absenteeism .
-
7/29/2019 OR-george schmid
15/50
The OR Process
1. Problem identification
2. Solution generation
3. Solution testing4. Results dissemination
5. Results utilization
Diagnostic step
Intervention step
-
7/29/2019 OR-george schmid
16/50
There are Lots of Ways to Approach
OR
Some are simple, others more formal
-
7/29/2019 OR-george schmid
17/50
Simple? Complex?
Attendance at the ART clinic is low
Patients do not have transport
Injection drug users are not usingclean needles
Clean needles are not available
-
7/29/2019 OR-george schmid
18/50
There are Lots of Ways to Approach
OR
Some are simple, others more formal
Flexibility needed and desired, to
meet the objectives of what you needto do
But, let's talk about more formal
approaches now.
-
7/29/2019 OR-george schmid
19/50
The Menu of the WHO/GF OR Book
1. Research team
2. Determine Issues
3. Develop proposal
4. Ethical clearance
5. Funding
6. Budget
7. Capacity building
8. Monitor project
9. Pre-test
10. Quality control
11. Stakeholder
discussions12. Dissemination plan
13. Disseminate results
14. Document changes
15. Monitor changes
16. Consider furtherimprovement
WHO (TDR)/GF (Jane Kengeya-Kayondo, George Shakarishvili, Serge Xueref)
and Bill Brieger and Amy Ellis of Johns Hopkins School of Public Health
-
7/29/2019 OR-george schmid
20/50
Three Phases to OR
1. Planning
2. Implementation
3. Follow Through
-
7/29/2019 OR-george schmid
21/50
Planning
1. Organize the Research Group
What disciplines do you need? Epidemiology, behavioural science?
Statistics?
Economics (It is difficult to retroactively do the economicspart of a study)?
Structure
Assign tasks, with deadlines
Meet regularly
Give credit!!!!!!
Who had the research idea? Builds teamwork among departments/persons and avoids
hard feelings
Pays off in the future
Pl i
-
7/29/2019 OR-george schmid
22/50
Planning
2. Determine the research questions
(and objectives) Be clear, be specific
Research questionWhat do you want to know?
Observation
"Our health care workers seem to have high rates of
absenteeism"
Research question
"What is the rate of absenteeism among our nurses in the
Kiev city hospital, and the reasons for it?"
Objective
Not: To study adherence
But: To determine rates of adherence among attendees at the
XX hospital and reasons for adherence and nonadherence
Pl i
-
7/29/2019 OR-george schmid
23/50
Planning
3. Develop a research
proposal/protocol
A document that details:
1. What the research is about and why it
is important
2. How the study will be conducted
3. How the results will be used
Planning
-
7/29/2019 OR-george schmid
24/50
Planning
3. Develop a research
proposal/protocol"standard parts" Research objectives and questions
Background Literature review
Local context
Research team
Methods (study design, study population,sampling details, how data will be collected, datamanagement, quality assurance)
Data collection instruments Plans for use of data
Budget
-
7/29/2019 OR-george schmid
25/50
Planning
3. Develop a research protocol
Does every "formal" OR project need
a research protocol?
Yes
Makes you clearly state what you will do
(and helps to improve your thinking)
Gives you the plan that you will follow, so
everyone knows what will be done, andshould be done
-
7/29/2019 OR-george schmid
26/50
Q3. Does Every OR Proposal need
to go through Ethical Review?
1. Yes
2. No
-
7/29/2019 OR-george schmid
27/50
Planning
4. Obtain ethical clearance
-
7/29/2019 OR-george schmid
28/50
Q4. What is the most common thing
wrong with consent forms?
1. Do not contain the contact
information for the local ethics board
2. Cannot be understood by potentialparticipants
3. Do not mention the word "research"
4. All risks are not revealed5. Give the patient too much information
-
7/29/2019 OR-george schmid
29/50
Planning
5. Identify funding
A variety of sources
Requests for small amounts of money arealmost always preferred
Follow closely the guidelines of fundingagencies
Ensure projects are well-written
Having money from one agency for aproject enhances the ability to get furtherfunds from any agency, i.e., "a proven trackrecord"
-
7/29/2019 OR-george schmid
30/50
Planning
6. Establish a budget
Should be comprehensive, even if nota large budgetensure you havecosted everything, including
dissemination costs A good budget is a sign of careful
planning
? Include a financial officer
-
7/29/2019 OR-george schmid
31/50
Planning
7. Technical capability/capacity
building
Ensure that staff know their roles and
responsibilities
Ensure that staff have the training and
capability to perform their jobs
Short term training (courses, TA)
Long term training (building capacity)
Help staff gain skills to advance
-
7/29/2019 OR-george schmid
32/50
Implementation
8. Quality Assurance of the project
There must be a protocol!
There almost always is a work plan, that includes: All activities to be implemented, before and during the
study
Starting date, completion date, intermediate deadlines
Persons who are responsible for each activity
Milestones for each activity
An advisory group that meets regularly may assistin maintaining quality and the work plan
Monitoring should be part of regular team meetings
A quality assurance plan should be part of theprotocol
If you do not perform the study exactly as you
said you would, you cannot trust the results
-
7/29/2019 OR-george schmid
33/50
Implementation
9. Pre-test all research materials
Valid? (Do data collection instrumentscapture the desired information reliably?)
Reliable? (Do data collection instrumentscapture the desired information
consistently?)
-
7/29/2019 OR-george schmid
34/50
Implementation
9. Pre-test all research materials
Questionnaires Is the wording clear? Do people understand what
you want? Do the questions ask what they aresupposed to?
Test in 5-20 persons
Focus groups Are the guides clear?
Hold one or two focus groups.
Consent forms
Do people understand them? Language level?(have your child read it) (sometimes, to checkunderstanding during study, develop a series of 5or so questionspatient must answer 4 or 5 toenter study)
Dissemination
-
7/29/2019 OR-george schmid
35/50
Dissemination
10. Establish and maintain data
management and QA
Forms should be easy to use
Observe data collectors regularly,supervisors review all data collection
instruments daily for completeness andaccuracy, have regular meetings of the datacollectors (daily, likely)
Data entry into computers preferably asstudy is ongoing so errors by datacollectors can be quickly caught and fixed
Double-entry of data?
Dissemination
-
7/29/2019 OR-george schmid
36/50
Dissemination
11. Explore with stakeholders
interpretation of results
Review preliminary results with theteam early, e.g., within a week afterdata entry is complete, to identifyreasons for unusual findings
Subsequently, review data with theteam and stakeholders, e.g.,
community leaders or project advisorygroup, to identify key findings andinterpretations
Follow-through
-
7/29/2019 OR-george schmid
37/50
Follow-through
12 & 13. Develop dissemination plan
and disseminate the results
Audience #1 ("internal"). Those who can benefitfrom, and act on, the information locally Local meetings
Meetings with decision makers
Audience #2 ("external"). Those who can adaptthe information for their own context, or use it forfurther research National or international meetings
Journals Create the right documents/strategy/medium for
the right audience
-
7/29/2019 OR-george schmid
38/50
Q5. Who Need Not Be Part of the
Pre-Test Evaluation of Materials?
1. Potential participants
2. Staff who use the materials, e.g.,interviewers, focus group leaders
3. Data management staff4. The members of the research team
developing the materials
5. All must be
Dissemination
-
7/29/2019 OR-george schmid
39/50
Dissemination
14. Document changes as a result of
the research
Were the results implemented? If not,
why not?
Dissemination
-
7/29/2019 OR-george schmid
40/50
Dissemination
15. Monitor changes in the revised
program
If the findings were valid, andappropriately implemented, thenprogram changes should occur whenthe results are implemented
More of a monitoring (and evaluation)function than OR, but gets into the
area of Translation Research.
Dissemination
-
7/29/2019 OR-george schmid
41/50
Dissemination
16. Consider ways of further
improving the program
Q6 Which Format of Information
-
7/29/2019 OR-george schmid
42/50
Q6. Which Format of Information
Sharing Do Decision-makers Like
Least (Industrialized Country)1. Internet/e-mail
2. Meetings/conferences
3. Colleagues
4. Journal articles
5. Short summaries
Dobbins M et al. Implementation Sci 2007
-
7/29/2019 OR-george schmid
43/50
Q7. Who Typically Acts on Identified
Problems?
1. The patient
2. The programme staff
3. The programme manager
4. The District manager
5. The National staff
6. Too few people act
Q1 What Proportion of Individuals
-
7/29/2019 OR-george schmid
44/50
Q1.What Proportion of Individuals
Potentially Helped by Bed Nets are
Using Them?1. 2%
2. 7%
3. 14%
4. 24%
Morel CM et al. BMJ 2005
Q2 Who Typically Identifies Problems
-
7/29/2019 OR-george schmid
45/50
Q2. Who Typically Identifies Problems
for OR? (more than one answer
possible)1. The patient
2. The programme staff
3. The programme manager
4. The District manager
5. National staff
Q3 D E OR P l d
-
7/29/2019 OR-george schmid
46/50
Q3. Does Every OR Proposal need
to go through Ethical Review?
1. Yes
2. No
Q4 Wh t i th t thi
-
7/29/2019 OR-george schmid
47/50
Q4. What is the most common thing
wrong with consent forms?
1. Do not contain the contact
information for the local ethics board
2. Cannot be understood by potentialparticipants
3. Do not mention the word "research"
4. All risks are not revealed5. Give the patient too much information
Q5 Wh N d N t B P t f th
-
7/29/2019 OR-george schmid
48/50
Q5. Who Need Not Be Part of the
Pre-Test Evaluation of Materials?
1. Potential participants
2. Staff who use the materials, e.g.,interviewers, focus group leaders
3. Data management staff4. The members of the research team
developing the materials
5. All must be
Q6 Which Format of Information
-
7/29/2019 OR-george schmid
49/50
Q6. Which Format of Information
Sharing Do Decision-makers Like
Least (Industrialized Country)1. Internet/e-mail
2. Meetings/conferences
3. Colleagues
4. Journal articles
5. Short summaries
Dobbins M et al. Implementation Sci 2007
Q7 Wh T i ll A t Id tifi d
-
7/29/2019 OR-george schmid
50/50
Q7. Who Typically Acts on Identified
Problems?
1. The patient
2. The programme staff
3. The programme manager
4. The District manager
5. The National staff
6. Too few people act