1

Foundations for RCR

Talked presented at the Conference on:

Mentoring and Supervision for the Responsible Conduct of ResearchWashington University School of Medicine

July 24, 2006

Nicholas H. Steneck, PhD

© Nicholas H. Steneck

For personal use; may not be quoted or used without permission of the author

2

Two approaches to RCR education

Formal ~ set curriculum / learning experience Courses, seminars, retreats, workshops Web-base programs

Informal ~ learn from others, on the job Journal clubs and laboratory meetings Mentor/PI sessions

Conference focus: Informal RCR education Objectives:

Key areas that should be covered Different approaches to RCR instruction for mentors Major challenges

3

1990s, shift to formal RCR education

Key events: 1989, IOM Report 1990, NIH Training Grant requirement

On Being a Scientist (2nd edition) In the past, young scientists learned the ethics of

research largely through informal means–by working with senior scientists and watching how they dealt with ethical questions. That tradition is still vitally important. But science has become so complex and so closely intertwined with society's needs that a more formal introduction to research ethics and the responsibilities that these commitments imply is also needed-an introduction that can supplement the informal lessons provided by research supervisors and mentors. (Preface)

4

Core areas

Developed over time: 1994 Training Grant Requirement

Conflict of Interest Responsible authorship Policies for handling misconduct Data management Human & animal subjects

2000 HHS RCR requirement 9 areas

Other areas could be added: Grant and financial management Laboratory safety & special rules (biological materials…) Workplace rules

Nine Core Areas1. Data acquisition,

management, sharing, and ownership

2. Mentor/trainee responsibilities

3. Publication practices and responsible authorship

4. Peer review

5. Collaborative science

6. Human subjects

7. Research involving animals

8. Research misconduct

9. Conflict of interest and commitment

Nine Core Areas1. Data acquisition,

management, sharing, and ownership

2. Mentor/trainee responsibilities

3. Publication practices and responsible authorship

4. Peer review

5. Collaborative science

6. Human subjects

7. Research involving animals

8. Research misconduct

9. Conflict of interest and commitment

5

Informal difficult to characterize

Wide range of informal instruction & settings: Mentor (mentoring) & supervisor (supervising) Research director / thesis advisor / major professor Administrative staff (IRB staff, department administrators…) Colleagues & fellow students

Mentor/thesis advisor may not be major influence Graduate students learn from postdocs Undergraduates learn from graduate students Coleagues may be more important than thesis advisors

Little is known (empirically) about how RCR is shaped by informal “instruction” Will focus on mentors, but there are other influences

6

Grows in importance over time

Plays greatestrole in shapingRCR

Poorly planned, organized, and assessed

In some cases, may do as much harm as good

Hypotheses ~ informal RCR instruction

early late

impo

rtan

ceinformal

formal

7

I. What should be taught? Key areas?

QRP ~ 10% - 50%QRP ~ 10% - 50%

High or highest standards ??High or highest standards ??

Could be as high as 1 in 100

Could be as high as 1 in 100

Research behavior

Option #1. Professional Standards Focus on what researchers do not appear to know

8

Research misconduct (FFP)

More common than generally assumed

Definitions & procedures not complicated

Accepting responsibility is a problem Reporting suspected misconduct Protecting whistleblowers Granting due process

Challenge Mentors in ideal position to promote RCR re. FFP Mentors are sometimes poor role models

9

Misconduct studies

JM Ranstam, Control Clin Trials (2000) Survey, 442 biostatisticians, 37% response 51% knew about fraud in medical research

26% involved FF 31% directly involved in projects with misconduct

Estimates of rate, .69% –> .80% (.25% standard)

Geggie, J Med Ethics (2001) Survey, 305 new medical consultants, 64% response

55.7% observed misconduct (FF lower) 5.7% committed misconduct in the past 18% would commit in future 17% had research ethics training

10

Misconduct studies continued…

Gardner, Contemporary Clinical Trials (2005) Sample ~ authors (549) who reported results of

pharmaceutical clinical trials (64% response) 1% reported target article misrepresented the research 5% reported fabrication in study they had participated in 17% knew personally of fabrication in a study

What should be taught; key areas? Definition of research misconduct! How to report research misconduct and to whom! There are no justifications for engaging in misconduct! Research misconduct is not tolerated in MY research

group! Misconduct is everyone’s responsibility

11

Questionable Research Practices

Definition Less than responsible conduct Falls short of good research practice At odds with “commonly accepted

practices”

Examples Misrepresentation of credential Conflicts of interest that impact

results/actions Undeserving/improper authorship Sloppy and bias publication

practices

What can/should mentors teach?

12

Authorship & publication Relevant authorship & publication rules Practical experience learning how to deal with:

Who should be an author? One publication or several? Necessary background research? What information is essential? Elements of a complete, accurate citation? Abstract? What constitutes a conflict of interest?

QRPs common (rampant) in research publications Mentors…

May not provide enough practical publication experience Mentors can be poor role models and set poor examples

13

Citation errors Inaccurate information/claims in notes

Grouped as major and minor (or) Citational vs. quotational Rates range from 10% to 30%

Recent findings:

Field All Major

Otolaryngology (2000) 37.5 11.9

Primary Care/AIDS (2003) 3% / 8%

Manuel Therapy (2004) 20 to 59% citational

Rate appears high & constant

14

Data management & interpretation Relevant data management & interpretation rules Practical experience learning how to deal with:

Necessary elements of a rigorous design? How data should be collected and stored? Who owns the data and with whom can it be shared? Appropriate and inappropriate statistical analyses? How to handle data that do not fit? What conclusions do the data support? Not support?

Data in publications is often biases and/or unreliable Mentors…

May not provide enough practical experience discussing data Mentors can be poor role models and set poor examples

15

Data practices that concern researchers Al-Marsouki, Contemporary Clinical Trials 26(2005)

Design %

Failure to use random allocation 92

Failure to specify in the protocol the main outcome measure 88

Inadequate allocation concealment 84

Different follow-up schedules in arms 80

Conduct %

Tampering with treatment packs so as to un-blind allocation 95

Selective withdrawals on basis of knowledge of allocation 92

Data falsification 92

Data fabrication 92

= Federal definition of misconduct

16

Al-Marsouki, continued….Analysis %

Altering analysis methods until finding a significant result 100

Use of battery of methods of comparison to get the right answer 100

Altering results in knowledge of allocation 100

Excluding patients to exaggerate effects or remove adverse events 99

Use of primary outcome measure that was not pre-specified 96

Reporting %

Failure to report unfavorable results 100

Selective reporting of positive results or omission of adverse events 96

Selective reporting based on p-values 92

Report of subgroup without reference to wide study 92

17

Al-Marsouki, Impact & likely to occur

• Over-interpretation of “significant" findings in small trials 83• Selective reporting based on p-values 80• Selective reporting of outcomes in the abstract 76• Subgroup analyses done without interaction tests 75• Negative or detrimental studies not published 68• Putting undue stress on results from subgroup analysis 68• Inappropriate subgroup analyses 64• Selective reporting of (i) subgroups (ii) outcomes (iii) time points 64• Selective reporting of positive results/omission of adverse events data 60• Failure to report results or long delay in reporting 60• Post-hoc analysis not admitted 59• Giving incomplete information about analyses with non significant results 56• Analysis conducted by the sponsor of the trial 54

• Over-interpretation of “significant" findings in small trials 83• Selective reporting based on p-values 80• Selective reporting of outcomes in the abstract 76• Subgroup analyses done without interaction tests 75• Negative or detrimental studies not published 68• Putting undue stress on results from subgroup analysis 68• Inappropriate subgroup analyses 64• Selective reporting of (i) subgroups (ii) outcomes (iii) time points 64• Selective reporting of positive results/omission of adverse events data 60• Failure to report results or long delay in reporting 60• Post-hoc analysis not admitted 59• Giving incomplete information about analyses with non significant results 56• Analysis conducted by the sponsor of the trial 54

18

Impact of COI on results Bekelman (2003), JAMA

Meta-analysis of 37 COI studies (1,000s of trials) Positive correlation (3.60 OR) , industry sponsorship &

positive outcomes

Lexchin (2003), BMJ Meta-analysis of 30 COI studies Positive correlation (4.05 OR), industry sponsorship &

positive outcomes

Friedman (2004) 398 publications, NEJM and JAMA Correlation (2.35-2.64 OR), industry/positive outcomes

19

What needs to be taught?

Ten Top Behaviors (Martinson, Nature, June 05) All Mid Early

1. Falsifying or ‘cooking’ research data 0.3 0.2 0.5

2. Ignoring major aspects of human-subject requirements 0.3 0.3 0.4

3. Not properly disclosing involvement in firms whose products are based on one‘s own research

0.3 0.4 0.3

4. Relationships with students, research subjects or clients that may be interpreted as questionable

1.4 1.3 1.4

5. Using another’s ideas without obtaining permission or giving due credit 1.4 1.7 1.0

6. Unauthorized use of confidential information 1.7 2.4 0.8

7. Failing to present data that contradict one’s own previous research 6.0 6.5 5.3

8. Circumventing certain minor aspects of human-subject requirements 7.6 9.0 6.0

9. Overlooking others' use of flawed data or questionable interpretation 12.5 12.2 12.8

10. Changing the design, methodology or results of a study in response to pressure from a funding source

15.5 20.6 9.5

= Federal definition of misconduct

20

Martinson, continued….

Other behaviors All Mid Early11. Publishing the same data or results in two or more publications

4.7 5.9 3.4

12. Inappropriately assigning authorship credit 10.0 12.3 7.4

13. Withholding details of methodology or results in papers or proposals

10.8 12.4 8.9

14. Using inadequate or inappropriate research designs 13.5 14.6 12.2

15. Dropping observations or data points from analyses based on a gut feeling that they were inaccurate

15.3 14.3 16.5

16. Inadequate record keeping related to research projects 27.5 27.7 27.3

Why do researchers engage in these behaviors? Do not know they are wrong? Have been “taught” that there are justifications for bending

the rules?

21

Other key areas for RCR education?

Option #2. Principles and professional responsibility

Essential foundation for RCR Strong sense of professional responsibility

Awareness of social responsibility

Moral reasoning skills

Mentors ideally situated to develop all three Know and interact with trainees on a personal level

Can lead by example – role model

Generally oversee trainees entry into professional practice

22

II. Teaching methods Options open to mentors

Role model; set a good example Include RCR in:

Regular and informal meetings with trainees Laboratory meetings / journal clubs Seminars and advanced courses Lab postings & information updates

Other ways to influence: Encourage trainees to attend RCR lectures & programs Encourage & support attendance at professional meetings With permission, involve in reviews & other professional

activities

Are mentors up to delivering the task at hand?

23

III. Major challenges Mentors are not trained to be RCR mentors

10-20% of researchers have some RCR education Graduate education focused on research not teaching

Mentoring seldom is planned & organized No lesson plan or course of study No common content or curriculum

Mentoring skills are not evaluated / rewarded Hiring & promotion based on research not teaching

Outcomes of mentoring are not assessed Evaluate research skills, not professional knowledge Professional knowledge of trainees is not assessed

24

How can RCR mentoring be improved?

Research institutions should rigorously assess mentoring skills and reward accordingly

NIH & NSF should encourage (require?) training programs to introduce RCR across the curriculum and as part of mentoring

RCR educators should pay more attention to mentoring Currently focus on courses, seminars, web pages and

other formal instruction Need better resources to help mentors provide RCR

education More information on the strengths and weaknesses

of the role mentoring plays in foster RCR

25