the future of scientific communication · the case for transparency • ―every day we read of...

Richard Smith

Former editor, BMJ; editor, Cases Journal

The future of scientific

communication

Outline of talk

• Problems with looking to the future

• Why the present methods of sharing medical and scientific information are not fit for purpose

• An aside on transparency and trust

• Drivers of change

• Four futures for scientific publishing

• A sketch of the future

• How transparency might save us

Lord Kelvin, president of the Royal Society, 1890-95

Lord Kelvin’s predictions

• "Radio has no future"

• "Heavier than air flying machines are impossible"

• "X rays will prove to be a hoax‖

What was predicted

• Paperless office

• Leisure society

• Death of the book

What wasn’t predicted

• Explosion of the internet (future

of medical journals, 1990)

• Berlin wall coming down

• September 11

• Credit crunch

Looking to the future:

common mistakes

• Making predictions rather than

attaching probabilities to possibilities

• Simply extrapolating current trends

• Thinking of only one future

Looking to the future:

common mistakes

• People consistently overestimate the

effect of short term change and

underestimate the effect of long term

change.

• Ian Morrison, former president of the Institute for the

Future

Why bother with the future?

• "If you think that you can run an

organisation in the next 10 years as

you've run it in the past 10 years you're

out of your mind.‖

• CEO, Coca Cola

Why bother with the future?

• ―The future belongs to the

unreasonable ones, the ones who look

forward not backward, who are certain

only of uncertainty, and who have the

ability and the confidence to think

completely differently.‖

• Charles Handy quoting Bernard Shaw

Why bother with the future?

• The point is not to predict the future but

to prepare for it and to shape it

1980: two choices for a global network

• Choice one

– Everybody has access

– Everybody can forward material

– Only trusted sources can put on information

• Choice two

– Anybody can put up anything

1990: two choices to build the world’s best encyclopaedia

• Choice one

– A global corporation with state of the art governance

– First class contributors and editors

– Elaborate fact checking

– Scrupulous copy editing

• Choice two

– A website where anybody can contribute and correct anything that’s there

How doctors feel about information

One man’s view

Current information problems

• Our current information policy resembles the worst aspects of our old agricultural policy, which left grain rotting in thousands of storage files while people were starving. We have warehouses of unused information rotting while critical questions are left unanswered and critical problems are left unresolved.

• Al Gore

Current problems

• Think of all the information that you might read to help you do your job better.

• How much of it do you read?

Answers from a sample of doctors

Current problems

• Do you feel guilty about how much or how little you read?

Do you feel guilty about how much or

little you read?

Yes

No

Answers from a sample of doctors

Words used by 41 doctors to describe their information supply

• Impossible Impossible Impossible Impossible Impossible Impossible

• Overwhelming Overwhelming Overwhelming Overwhelming Overwhelming Overwhelming

• Difficult Difficult Difficult Difficult

• Daunting Daunting Daunting

• Pissed off

• Choked

• Depressed

• Despairing

• Worrisome

• Saturation

• Vast

• Help

• Exhausted

• Frustrated

• Time consuming

• Dreadful

• Awesome

• Struggle

• Mindboggling

• Unrealistic

• Stress

• Challenging

• Challenging

• Challenging

• Excited

• Vital importance

Information paradox

• ―Water, water everywhere,

• Nor any drop to drink.‖

• The Rime of the Ancient Mariner

Information problems of scientists: Science Commons

• The World Wide Web was created by scientists

• It has transformed much, but we haven’t yet realised the benefits of the web for science

• Video: http://blip.tv/file/1319678

What’s wrong with medical journals

• Don’t meet information needs

• Too many of them

• Too much rubbish

• Too hard work

• Not relevant

• Too boring

• Too expensive

What’s wrong with medical journals

• Don’t add value

• Slow every thing down

• Too biased

• Anti-innovatory

• Too awful to look at

• Too pompous

• Too establishment

What’s wrong with medical journals

• Don’t reach the developing world

• Can’t cope with fraud

• Nobody reads them

• Too much duplication

• Too concerned with authors rather thanreaders

The importance of transparency

―What isn’t transparent is assumed to

be biased, corrupt, or incompetent

until proved otherwise.‖

―What have they got to hide?‖

That's how the world is—like it or not

The doctor patient relationship 1871

• ―Your patient has no more right to all the truth you know than he has to all the medicines in your saddlebags...He should get only just so much as is good for him.‖

• Oliver Wendell Holmes

The doctor patient relationship 2007

• ―The whole structure of medicine has been based on the assumption that physicians have the current information and patients do not. The bottom line is, the consumer will have virtually all the information the professionals have. This is comparable to the way communism fell. Once people start getting in good communication you won’t be able to play the game in the same way.‖

• Tom Ferguson 2004

Onora O’Neill: a Question of Trust

The case for transparency

• ―Every day we read of untrustworthy action by politicians and officials, by hospitals and exam boards, by companies and schools.‖

• ―Mistrust and suspicion have spread across all areas of life…‖

• ―We believe that increased accountability will help--and accountability depends in large part on information and transparency‖

• ―The efforts to prevent abuse of trust are gigantic, relentless, and expensive; their results are always less than perfect.‖

• Increased transparency is much easier in the age of the internet

• It’s increasingly difficult to hide anything anyway

• Plus people and patients are fed up with being patronised

What is the relationship between transparency and trust?

• There can be no such thing as complete transparency

• ―At some point we just have to trust‖

• As transparency has advanced trust seems to have receded

• ―If we want to restore trust we need to reduce deception and lies rather than secrecy‖

What is the relationship between transparency and trust?

• Increased transparency may lead to increased deception because people may be careful in what they write or say if they know everything is to be made public--using evasions, hypocrisies, and half-truths

• They may also resort to spin

• ―Well placed trust grows out of active inquiry rather than blind acceptance‖

• People need information they can check and assess its accuracy for themselves

• This is demanding

Ingredients of trust in 2009

• If you start from a position of trust, then an absence of evidence of being deliberately deceived or misinformed

• Accurate, understandable, interpretable, unspun, checkable information

• Capacity to understand, interpret, and check the information

• Repeated checking of the information without any discovery of deliberate deception

• Prompt admission of error by the trusted source

Problems in science communication that transparency might solve

• Public and professional distrust of science: different story every week, BSE in Britain, MMR

• Extravagant claims made on limited data at conferences and in the mass media

• Pervasive bias in sponsored research

• Research agenda not focused on what matters most

• Publication bias

• Unfairness in the publication process

• Ineffectiveness of peer review

• Research misconduct

• Inability to access research

What are the drivers of a new form of publishing?

• Failures of the present system

• A vision of something better

• Money

• Balkanisation of the literature

• Slowness

A vision of something better

• "It's easy to say what would be the ideal online resource for scholars and scientists: all papers in all fields, systematically interconnected, effortlessly accessible and rationally navigable, from any researcher's desk, worldwide for free.‖

• Stevan Harnad

A vision of something better

• If you have an apple and I have an apple and if we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas.

• George Bernard Shaw

Money: What does the research community do?

• Do the research, often funded by public money, often costing millions

• Hand over the copyright to the journals

• Do the editing, often unpaid

• Do the peer review, almost always unpaid

• Often do the technical editing, often unpaid

• Buy the journals, often at inflated prices, some cost $20 000

• Read the journals

• Store the journals

Money: What do the publishers do?

• May own the journals, although often they don’t

• Manage the process

• Lend the money to keep the process going

• Design - usually minimal

• Typeset, print, and distribute the journal

• Market the journal - but often to libraries that have to have them

• Sell reprints - sometimes for $1m a time (nothing to authors or funders of the research); can almost sell themselves

• Sell advertising - often none

Balkanisation

• If you are a gastroenterologist the research that might matter to you may be in 30 different journals

• The difficulty of doing systematic reviews

• Important research articles are all over the place, some in Pubmed, many not

• Even if you can find the stuff, it costs a fortune to gather it all together (systematic review on research misconduct - £2000 to get photocopies)

Slowness

• For many journals the time betweensubmission and publication is still overa year--unacceptable

Four possible futures: Simpson scenarios

Homer: fat, lazy, rather gormless

• ―Medical publishing ain’t broke so need to fix it‖

• Traditional journals remain

• Peer review closed

• Publishers invest in summarising material and point of care information

Marge: the wise mother

• Almost all material is open access and published on databases rather than in journals

• Open peer review

• A few journals remain but have become magazines

• Point of care information

• Researchers linked electronically in clubs

• Academic credit comes from hits, citations in magazines, and evidence of making a difference in the real world

Lisa: smart, sassy, well informed daughter

• Published material replaced by electronic conversations: blogs, social networking sites

• Everybody, including researchers, are in clubs where data are shared

• Powerful search engines

• Information a side product of work and leisure activities

• Wikis for everything: ―the wisdom of the many‖

• Traditional publishers largely gone

Bart: the streetwise son

• A world where information comes mainly from large organisations—governments, pharma companies, Google, WHO

• Editors work for these organisations which also sponsor research

• Money and idea markets intertwined

A sketch of the future

• Scientific papers published not on paper but posted on the web in databases—using the full possibilities of the web:

– all raw data

– software used to manipulate the software

– links to all relevant material

– multimedia

• Peer review—not a black box but an open scientific discourse

– World is moving from ―filter then publish‖ to ―publish then filter‖—as with Wikipedia

• Everything open access

• A few paper journals remain, finding research that matters to their readers and presenting it to them in a readable, actionable way

Creating the future: injecting transparency into every stage of the process of communicating science

Saved by transparency: the question

• The best research starts with a question

• Why not share that question and your method for answering it? With your crowd? (Facebook? A social network for researchers/doctors?) With the whole world?

• ―Somebody will steal my idea‖

• ―But it's there recorded for all the world to see‖

• Sharing the question and your method for answering it will refine both the question and your methods

• CLOSER EXAMINATION OF THE QUESTION MAY BE VITAL BECAUSE THAT IS HOW SPONSORING GET THE ANSWERS THEY WANT

Saved by transparency: share the protocol

• Probably many people do anyway—but not with the world

• Routine with Cochrane

• Lancet review of protocols, but 12 in 1997, 2 only in 07—not catching on

Saved by transparency: Register the trial

Saved by transparency:Issues around registering trials

• Big increase in number of trials since ICMJE required it

• But how much information should be given?

• Should there be let outs?

• ―We recognize that requiring public registration of trials whose prespecified goal is to investigate the biology of disease or to direct further research might slow the forces that drive innovation. Therefore, each journal editor will decide on a case-by-case basis about reviewing unregistered trials in this category.‖ ICMJE

• ―There is no good commercial reason for refusing to register trials.‖ Richard Sykes, former CEO of Glaxo Wellcome

Saved by transparency: posting your results, full data set, and software used to manipulate data?

• Why not? Should be the default position

• Eprint server for physics, math, computer science, quantitative biology, and statistics since 1996

• 300 eprints posted a month for physics; just starting for statistics

• Very slow in medicine. BMJ and Lancet tried 10 years ago—flopped

• Fear of ―worrying the public with material that has not been peer reviewed‖

• But

– Little evidence for the benefit of peer review

– Loads of rubbish published anyway

– Extravagant claims made at conferences without full data

– Claims made directly to mass media without full data

• It would be much better if full data and methods were available

• The real reason is cultural resistance to change

Saved by transparency: make all raw data available

• Possible in the age of the internet

• Guards against fraud (but not completely)

• Reviewers can check results

• Do data on my blood pressure belong to me? I think so. I want my data to be available to all for public use.

• Can make multiple uses of data sets

• Why not?

– I collected the data, why should others benefit?

– May be misused

Saved by transparency:

opening up peer review

• Was unstudied but now extensively studied—very little evidence of benefit, much of problems

• Slow, expensive

• Wasteful—studies work their way down the foodchain

• Something of a lottery—concordance only slightly higher than chance

• Doesn’t detect errors. Inserted 8 errors into a 600 word paper; around 300 reviewers; nobody spotted more than 5, median number spotted 2; 20% spotted none

• Biased—Peters and Ceci study: 12 papers resubmitted; 3 journal spotted published before; 8 rejected

• Anti-innovatory

• Easily abused—steal ideas, harshly review work of rivals

• ―If it was a drug it would never get onto the market.‖ Drummond Rennie

• Central to science, and yet belief in it is based on faith not science

Saved by transparency: opening up peer review

• Blind reviewing—several studies, no overall evidence of improvement in quality of review; blinding fails in around a fifth of studies

• Let authors know the names of the reviewers

• Let readers and authors know the names of reviewers on publication

– Reviewers may object initially but will comply

– No evidence of improvement in the quality of the opinion

– Does disclose conflicts of interest

• Training reviewers, little evidence of benefit

• Let peer review become an open scientific discourse conducted online; the wisdom of many not the few

• Returning to the original way of communicating science—as in the 17th century

Saved by transparency: Declaring conflicts of interest

• Of course

• Unresolved questions

– Disclose amount of money in financial conflicts?

– Disclosing non-financial conflicts

– When is disclosure not enough?

Saved by transparency: “Postpublication” comment

• ―Anything goes‖ as with many comments on blogs?

• Let correspondents be anonymous? (This would be against transparency)

• Select comments--with a higher or lower threshold

• Publish only selected (? biased) comments

• Let respondents comment in the text—as with Wikipedia

Saved by transparency: Opening up

the whole scientific body of research

Of course

John Milton on “transparency”

• "Give me the liberty to know, to utter, and to argue freely according to conscience, above all liberties. Truth was never put to the worse in a free and open encounter ... It is not impossible that she [truth] may have more shapes than one ... If it come to prohibiting, there is not ought more likely to be prohibited than truth itself, whose first appearance to our eyes bleared and dimmed with prejudice and custom is more unsightly and implausible than many errors.‖