open science monitor study on open science: … · trends, and understanding the drivers (and...
TRANSCRIPT
OPENSCIENCEMONITOR
STUDYONOPENSCIENCE:MONITORINGTRENDSAND
DRIVERSReference:PP-05622-2017
D.2.4Finalreport
13thofDecember2019
Consortiumpartners:
TheLisbonCouncil
ESADEBusinessSchool
CentreforScienceandTechnologyStudies(CWTS)atLeidenUniversity
Subcontractor: Elsevier
2
TableofContents1 Introduction...........................................................................................................................................................5
1.1 Objectives....................................................................................................................................................5
1.2 Scope..............................................................................................................................................................7
1.3 Methodologicalapproach....................................................................................................................8
2 Overviewoftrendsandindicators............................................................................................................9
2.1 Openaccesstopublications...............................................................................................................9
2.2 Openresearchdata..............................................................................................................................13
2.2.1 Researchers’behaviour..........................................................................................................14
2.2.2 Howdataareshared................................................................................................................14
2.2.3 Dataownership..........................................................................................................................16
2.2.4 Attitudesacrossdisciplines.................................................................................................17
2.2.5 Datarepositories.......................................................................................................................19
2.3 Opencollaboration..............................................................................................................................20
3 Driversandbarriers.......................................................................................................................................23
3.1 Drivers........................................................................................................................................................23
3.2 Barriers......................................................................................................................................................30
4 Impact....................................................................................................................................................................34
4.1 Scientificimpact....................................................................................................................................35
4.1.1 Crossanalysisofcases............................................................................................................36
4.1.2 Qualitativeinsightfromcases.............................................................................................37
4.2 Industrialimpact...................................................................................................................................40
4.2.1 Crossanalysisofcases............................................................................................................42
4.2.2 Qualitativeinsights...................................................................................................................44
4.3 Societalimpact.......................................................................................................................................49
4.3.1 Crossanalysisofcases............................................................................................................50
5 Policyanalysis...................................................................................................................................................53
5.1 Statisticaloverview.............................................................................................................................53
5.1.1 Openaccesspolicies.................................................................................................................53
5.1.2 Opendatapolicies.....................................................................................................................55
5.1.3 Otherpolicies...............................................................................................................................57
5.2 Indepthcross-analysisofpolicycases......................................................................................58
5.2.1 Objectives......................................................................................................................................58
5.2.2 Provisions......................................................................................................................................59
5.2.3 Lessonslearnt.............................................................................................................................61
6 Policyconclusions...........................................................................................................................................62
3
1.1. Conclusionsonthefutureofopenscience..............................................................................62
1.2. ConclusionsonthefutureoftheOpenScienceMonitor..................................................63
7 References...........................................................................................................................................................67
ListofFiguresFigure1:Aconceptualmodel:aninterventionlogicapproach.............................................................6
Figure2:Percentageofopenaccesspublications........................................................................................9
Figure3:Percentageofopenaccesspublicationsbycountry.............................................................11
Figure4:PercentageofopenaccesspublicationsbyFieldofScienceandTechnology........13
Figure5:Haveyoupublishedtheresearchdatathatyouusedorcreatedaspartofyourlastresearchprojectinanyofthefollowingways?.......................................................................14
Figure6:Doyoutakestepstomanageyourresearchdataand/orarchiveitforpotentialre-usebyyourselfand/orothers?..........................................................................................................15
Figure7:Haveyoudoneanyofthefollowingwithanyoralloftheresearchdatathatyouusedorcreatedaspartofyourlastresearchproject?.................................................................15
Figure8:Categoriesofresearchdatausedinyourlastresearchproject.....................................15
Figure9:Researchdataformatsusedorcreatedaspartofyourlastresearchproject........16
Figure10:Whatwastheprimarysourceoffundingforyourlastresearchproject?.............16
Figure11:Whatwastheprimarysourceoffundingforyourlastresearchproject?.............17
Figure12:Attitudestodatamanagementbysubject(I/II)..................................................................17
Figure13:Attitudestodatamanagementbysubject(II/II)................................................................18
Figure14:Attitudestodatamanagementbyregion/age(I/II).........................................................18
Figure15:Attitudestodatamanagementbyregion/age(II/II)........................................................19
Figure16:Datarepositoriesbysubject..........................................................................................................19
Figure17:Datarepositoriesbytypeofaccess............................................................................................19
Figure18:Datarepositoriesbycountry,bluebarindicateEuropeancountries......................20
Figure19:NumberofscientificAPIs................................................................................................................21
Figure20&Figure21:Openhardwareprojects........................................................................................21
Figure22:ProjectsinScistarterbydiscipline.............................................................................................22
Figure23:projectsinZooniversebydiscipline..........................................................................................22
Figure24:Driversofsharingresearchdata.................................................................................................25
Figure25:Howimportantwerethefollowingfactorswhendecidingtouseother’sresearchdata?...................................................................................................................................................26
Figure26:Howwouldyoudescribetheefforttypicallyrequiredtomakeyourresearchdatare-useablebyothers?..........................................................................................................................26
Figure27:Barriersacrossthecasestudies..................................................................................................32
Figure28:Numberofbarriersacrosscasestudies...................................................................................32
Figure29:Barrieracrossopensciencecategories....................................................................................33
4
Figure30:Thinkingaboutthemostrecentresearchprojectonwhichyoushareddata,didindividualsoutsideoftheresearchteamcontactyouconcerningthedatathatyoushared?Iwascontactedbyresearchersfrom..................................................................................34
Figure31:Stillthinkingaboutthemostrecentresearchprojectonwhichyoushareddata,doyoubelieveanyofthefollowinghappenedasaconsequenceofsharingthedata?...................................................................................................................................................................................34
Figure32:Scientificimpactsaslistedinthecrossanalysisofpolicies..........................................36
Figure33:Topfivedriversfoundincasestudieswithhighindustryimpact.............................44
Figure34:Impactonsocietyoftheanalysedcasestudies...................................................................50
Figure35:Casestudieswithhighandmediumimpactonsociety,byresearchphases.......51
Figure36:Funder’spoliciesonopenacesspublishingbycountry..................................................53
Figure37:Funders’policiesonopenaccessarchiving...........................................................................54
Figure38:Archivingpolicesbytypeofmandate.......................................................................................55
Figure39:Opendatapoliciesbytypeofmandate....................................................................................56
Figure40:Opendatapoliciesbycountry......................................................................................................56
Figure41:Journalsbydatasharingmodality..............................................................................................57
Figure42:Journalsbycodesharingmodality.............................................................................................57
ListofTablesTable1:Articulationofthetrendstobemonitored....................................................................................7
Table2:overviewofthemethodology...............................................................................................................8
Table3:Openaccesspublications(gold,green,hybridandbronze)byyear.............................10
Table4:Openaccesspublications(gold,green,hybridandbronze)bycountry.....................12
Table5:Codingdriversanddescription.........................................................................................................24
Table6:Analyticalsummaryofmechanismsenablingthepositiveimpactofopensciencetoindustry...........................................................................................................................................................48
Table7:Casestudieswithhighandmediumimpactonsociety,byopensciencecategory...................................................................................................................................................................................51
Table8:Overviewofscientific,societalandindustrialdriversacrossNLandFIOpenScienceandUKOpenResearchPolicies..............................................................................................58
Table9:ProvisionsaddressedintheNLandFIOpenScienceandUKOpenResearchPolicies..................................................................................................................................................................60
Table10:Highlevelmethodologicalapproach,past(x)andfuture(greyshade)....................65
5
1 IntroductionOpen science has emerged as a powerful trend in research policy. To be clear,opennesshasalwaysbeenacorevalueofscience,butitmeantpublishingtheresultsor research in a journal article. Today, there is consensus that, by ensuring thewidestpossibleaccessandreusetopublications,data,codeandotherintermediateoutputs, scientific productivity grows, scientific misconduct becomes rarer,discoveriesareaccelerated.Yetitisalsoclearthatprogresstowardsopenscienceisslow,because ithas to fit ina system thatprovidesappropriate incentives toallparties.TheEuropeanCommissionhasrecognizedthischallengeandmovedforwardwithstronginitiativesfromtheinitial2012recommendationonscientificinformation(C(2012)4890), suchas theOpenSciencePolicyPlatformand theEuropeanOpenScienceCloud.1OpenaccessandopendataarenowthedefaultoptionforgranteesofH2020.The Open Science Monitor (OSM) aims to provide data and insight needed tosupporttheimplementationofthesepolicies.Itgathersthebestavailableevidenceon the evolution of open science, its drivers and impacts, drawing on multipleindicatorsaswellasonarichsetofcasestudies.2This monitoring exercise is challenging. Open science is a fast evolving,multidimensional phenomenon. According to the OECD (2015), “open scienceencompasses unhindered access to scientific articles, access to data frompublicresearch,andcollaborativeresearchenabledbyICTtoolsandincentives”.Thisverydefinition confirms the relative fuzzinessof the conceptand theneed fora cleardefinitionofthe"trends"thatcomposeopenscience.Preciselybecauseofthefastevolutionandnoveltyofthesetrends,inmanycasesitis not possible to find consolidated, widely recognized indicators. For moreestablished trends, such as open access to publications, robust indicators areavailable through bibliometric analysis. Formost others, such as open code andopenhardware,therearenostandardisedmetricsordatagatheringtechniquesandthereistheneedtoidentifythebestavailableindicatorthatallowsonetocapturetheevolutionandshowtheimportanceofthetrend.
Today, especially at European levelwhere competences on research are limited,data and indicator play a powerful role in executing policies. Conversely, theabsenceofrobustdatacanhindertheimplementationofthepolicy.
ThisispreciselytheobjectiveoftheOpenScienceMonitor.
1.1 ObjectivesTheOSMcoversfourtasks:
1. Toprovidemetricsontheopensciencetrendsandtheirdevelopment.2. Toassessthedrivers(andbarriers)toopenscienceadoption.
1https://ec.europa.eu/research/openscience/index.cfm?pg=open-science-policy-platform;https://ec.europa.eu/research/openscience/index.cfm?pg=open-science-cloud2TheOSMhasbeenpublishedin2017asapilotandre-launchedbytheEuropeanCommissionin2018throughacontractwithaconsortiumcomposedbytheLisbonCouncil,ESADEBusinessSchoolandCWTSofLeidenUniversity(plusElsevierassubcontractor).Seehttps://ec.europa.eu/research/openscience/index.cfm?pg=home§ion=monitor
6
3. Toidentifytheimpacts(bothpositiveandnegative)ofopenscience.4. Tosupportevidence-basedpolicyactions.
The indicators presented here focusmainly on the first two tasks:mapping thetrends, and understanding the drivers (and barriers) for open scienceimplementation.Thechartbelowprovidesanoverviewoftheunderlyingconceptualmodel.
Figure1:Aconceptualmodel:aninterventionlogicapproach
Thecentralaspectofthemodelreferstotheanalysisoftheopensciencetrendsandis articulated alongside three dimensions: supply, uptake and reuse of scientificoutputs.In the OSM framework, supply refers to the emergence of services such as datarepositories.Thenumberofdatarepositories(oneof theexisting indicators) isasupplyindicatorofthedevelopmentofopenscience.Onthedemandside,indicatorsinclude,forexample,theamountofdatastoredintherepositories,thepercentageofscientistssharingdata.Finally,becauseofthenatureofopenscience,theanalysiswillgobeyondusage,sincethereusedimension isparticularlyimportant. In thiscase,relevantindicatorsincludethenumberofscientistsreusingdatapublishedbyotherscientists,orthenumberofpapersusingthesedata.On the left side of the chart, themodel identifies the key factors influencing thetrends,bothpositivelyandnegatively(i.e.driversandbarriers).Bothdriversandbarriers are particularly relevant for policy-makers as this is the areawhere anaction canmake greatest difference, and are therefore strongly related topolicyrecommendations.Theseinclude“policydrivers”,suchasfunders’mandates.Itisimportanttoassessnotonlypolicydriversdedicatedtoopenscience,butalsomoregeneralpolicydriversthatcouldhaveanimpactontheuptakeofopenscience.Forinstance,theincreasingrelianceonperformance-basedfundingortheemphasison
7
marketexploitationofresearcharegeneralpolicydriversthatcouldactuallyslowdowntheuptakeofopenscience.Therightsideof thechart in themodel illustrates the impactsofopensciencetoresearch or the scientific process itself; to industry or the capacity to translateresearchintomarketableproductsandservices;tosocietyorthecapacitytoaddresssocietalchallenges.
1.2 ScopeBydefinition,openscienceconcernstheentirecycleofthescientificprocess,notonly open access to publications. Hence the macro-trends covered by the studyinclude:openaccess topublications,open researchdata andopencollaboration.Whilethefirsttwoareself-explanatory,openscientificcollaborationisanumbrellaconcepttoincludeformsofcollaborationinthecourseofthescientificprocessthatdonotfitunderopendataandopenpublications.Table1:Articulationofthetrendstobemonitored
Categories TrendsOpenaccesstopublications
• Openaccesspolicies(fundersandjournals),• Greenandgoldopenaccessadoption(bibliometrics).3
Openresearchdata
• Opendatapolicies(fundersandjournals)• Opendatarepositories• Opendataadoptionandresearchers’attitudes.
Opencollaboration
• Opencode,• Altmetrics,• Openhardware,• Citizenscience.
Newtrendswithintheopenscienceframeworkwillbeidentifiedthroughinteractionwiththestakeholder’scommunitybymonitoringdiscussiongroups,associations(suchasResearchDataAlliance-RDA),mailinglists,andconferencessuchasthoseorganisedbyForce11(www.force11.org).
Thestudycoversallresearchdisciplines,andaimstoidentifythedifferencesinopenscienceadoptionanddynamicsbetweendiversedisciplines.Currentevidenceshowsdiversityinopensciencepracticesindifferentresearchfields,particularlyindata-intensive research domains (e.g. life sciences) compared to others (e.g.humanities)
Thegeographiccoverageofthestudyis28MemberStates(MS)andG8countries,includingthemaininternationalpartners,withdifferentdegreesofgranularityforthedifferentvariables.Asfaraspossible,datahastobepresentedatcountrylevel.
3AccordingtotheEC,“‘Goldopenaccess’meansthatopenaccessisprovidedimmediatelyviathepublisherwhenanarticleispublished,i.e.whereitispublishedinopenaccessjournalsorin‘hybrid’journalscombiningsubscriptionaccessandopenaccesstoindividualarticles.Ingoldopenaccess,thepaymentofpublicationcosts(‘articleprocessingcharges’)isshiftedfromreaders’subscriptionsto(generallyone-off)paymentsbytheauthor.[…]‘Green.openaccess’meansthatthepublishedarticleorthefinalpeer-reviewedmanuscriptisarchivedbytheresearcher(orarepresentative)inanonlinerepository.”(Source:H2020ModelGrantAgreement)
8
1.3 MethodologicalapproachTheOpenScienceMonitorgathersdatafromapluralityofsources.Whilethedetailsarefullyexplainedinthemethodologicalnote,itisworthsummarizingthemainaspectshere.Thestudyproduceskeyindicatorsandmetricsrelatedtotheabove-mentionedopensciencetrends.Theseindicatorsaregeneratedusingmultipledatasourceswhichcanbecategorizedinthreemaintypes.First,ananalysisofexistingdataandmetrics,forinstanceasprovidedbytheSherpaorRe3datadatabase.Indicatorshereareproducedbyanalysingtheunderlyingdataprovidedbytheseservices.Secondly,thestudygeneratesitsownmetrics,namelyonopenaccesstopublications,byanalysingpublicationsbasedontheScopusandUnpaywalldata.Thirdly,thestudyincludedasurveyofaboutathousandresearchersworldwide,performedbyElsevier.Thestudyalsoprovidesananalysisofpolicies,drivers,barriersandimpacts.Inthiscase,theevidenceisprovidedmostlyby28casestudiesandtherelatedmeta-analysis.
Table2:overviewofthemethodology
Existingmetrics
Ownmetrics
Survey Cases
Trends X X X Publication X X
Data X X Collaboration X Drivers X XBarriers X XPolicies X XImpact X X
9
2 OverviewoftrendsandindicatorsThissectionpresentsanoverviewofupdatedindicatorsofOpenScienceMonitor.Theinitialoverviewoftrendsisbasedontheinterventionlogicofthestructureofthemethodologyproposedintheinceptionreport:
• Openaccesstopublications,• Openresearchdata,• Opencollaboration.
AllthedataincludedinthissectionaretakenfromtheonlinedashboardoftheOpenScience Monitor, available at: https://ec.europa.eu/info/research-and-innovation/strategy/goals-research-and-innovation-policy/open-science/open-science-monitor_en.Fulldetailsonthemethodologyareavailableonthewebsiteandinannextothisreport.
2.1 OpenaccesstopublicationsThe indicators ofOpen access to publications aremeasuring towhat extent it ispossible to freelyaccess researchpublications.The indicators coverbibliometricdataonpublications,aswellasdataonfunders'andjournals'policies.Concerningtheavailabilityofopenaccesspublications,thetotalnumberofopenaccess publications (gold, green, hybrid and bronze) had been growing between2009and2018(theamountofopenaccesspublicationshasalmostdoubledinthisperiodfrom361thousandtoover684thousand),though,asignificantslowdowningrowth since 2016was registered (8% growth as compared to growth rangingbetween12%to16%inthepreviousyears).Datafor2017and2018showadecline,however inourconsiderationthis issimplyduetotheembargoperiod forgreenopenaccesssothattheywillbecomeavailableoverthenextmonths–asshownbythe fact that gold open access, where embargo does not apply, is continuouslygrowing.Figure2:Percentageofopenaccesspublications
10
Table3:Openaccesspublications(gold,green,hybridandbronze)byyear
Year 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
NotOA 807672 824427 876706 917504 971428 1010275 1017187 1044540 1086564 1208464
NotOA%ofchange 2.1% 6.3% 4.7% 5.9% 4.0% 0.7% 2.7% 4.0% 11.2%
OpenAccess 361932 389100 445341 503875 569939 625503 677841 731812 736656 684636
OA%change 7.5% 14.5% 13.1% 13.1% 9.7% 8.4% 8.0% 0.7% -7.1%
GoldOA 54910 72608 101740 140147 175309 212999 242785 269360 309593 349589
%change 32.2% 40.1% 37.8% 25.1% 21.5% 14.0% 10.9% 14.9% 12.9%
GreenOA 239013 264567 303047 344491 395195 444121 478279 514907 490630 289461
%change 10.7% 14.5% 13.7% 14.7% 12.4% 7.7% 7.7% -4.7% -41.0%
HybridOA 42571 43850 57033 62363 70992 84624 102384 119348 117541 119019
%change 3.0% 30.1% 9.3% 13.8% 19.2% 21.0% 16.6% -1.5% 1.3%
BronzeOA 134029 134547 139945 147194 151145 144268 143379 138292 134500 93216
%change 0.4% 4.0% 5.2% 2.7% -4.5% -0.6% -3.5% -2.7% -30.7%
Thecloserlookintothedatarevealsthatthegrowthwasmainlydrivenbytherapidgrowthofthenumberandshareofopenaccessgoldpublications.In2018,theopenaccess gold publications outnumbered for the first-time open access greenpublications(green–289thousandtogold–349thousand).
At the country level, top five countries with the biggest share of open accesspublications(totalofgold,green,hybridandbronze)are:UnitedKingdom(52.3%),Switzerland (51.8%), Croatia (50.8%), Luxembourg (50%) and Netherlands(49.9%).Ontheoppositespectrumbottomfivecountrieswiththelowestshareofopen access publications are: Russian Federation (23.9%), China (27.8%), India(29.9%),Greece(34.7%),Canada(37.1%).
However, if we look at the absolute numbers of open access publications, thesituation changes significantly with US clearly leading the way to open access(27.6% of all open access publications), followed by China (10%), UK (9.9%),Germany(6.6%)andJapan(5.4%).Togetheropenaccesspublicationsofthesefivecountriesrepresentnearly60%ofallopenaccesspublications.
11
Figure3:Percentageofopenaccesspublicationsbycountry.
12
Table4:Openaccesspublications(gold,green,hybridandbronze)bycountry
Intermsofthescienceandtechnologyfield,Multidisciplinaryfieldhasthebiggestshareofopendatapublications–86.2%,followedbyOthermedicalsciences55.7%,Agriculture,forestry,fisheries(51.9%),andlifesciences:Biologicalsciences(49.2%),Basicmedicalresearch(48.5%)andClinicalmedicine(44%).
Themost closed fieldswith the lowest number of open access publications are:Mechanicalengineering(17.3%),Art(19%),Chemicalengineering(19%),Chemicalsciences(19.4%)andMaterialsengineering(19.8%).
13
Figure4:PercentageofopenaccesspublicationsbyFieldofScienceandTechnology
It isworthnoticing, however, that in termsof absolute numbers the situation isslightly different. Clinical medicine, Biological sciences, Basic medical research,PhysicalsciencesandastronomyandEarthandrelatedenvironmentalscienceshavethe highest aggregated number of open access publications, whereas Otheragriculturalsciences,Art,Mediaandcommunication,LawandPoliticalsciencearethefieldswiththelowestnumberofopenaccesspublications.
2.2 OpenresearchdataTheOpenresearchdataindicatorsmeasuretowhatextentdataunderpinningscientificresearchresultshasnorestrictionsonitsaccess.Themainsourcefor
14
theseindicatorsisthesurveyrunbyElsevierin2018,butsomeindicatorsaredevelopedthroughexistingmetrics.
2.2.1 Researchers’behaviourDataareafundamentalinputinallresearchactivities.Observationaldataarethemostcommondatacategory,usedby48%ofresearchers,followedbyexperimentaldata,usedby35%ofresearchers.Intermsofsize,thevastmajorityofresearchers(70%)generateslessthan10GBofdataintheirlastresearchproject.Researcherssharedataindifferentways.Themostcommonlyusedmethodisasanappendixtoanarticle(35%),whileonly14%ofresearchersshareitthroughdatarepositories.Overall,athirdofresearchersdeclarenottosharetheirdata.This ismostlygoodnews: thevastmajorityofresearchersdeclaretosharedata.However,acloserlookmakesthesedatalessgoodthantheylook:
1. Thisisdatasharingasself-reportedbyresearchers.Notonlyitispronetotheusuallimitationsofthemethodcomparedtoobservation,butalsoitdoesnotreflectthearticleasaunitofanalysis.Othersourcesshowthatthevastmajorityofarticlesdonotincluderesearchdata.
2. Repositoriesareconsideredthemostsuitablewaytomakedatagenuinelyaccessibleandreusableforotherresearchers,buttheyareyetusedonlybyasmallminorityofresearchers(roughlyoneinseven).
3. These results are practically identical to the 2016 survey, both in theaggregateand in the individualoption, showing that limitedprogresshasbeenachieved.
Figure5:Haveyoupublishedtheresearchdatathatyouusedorcreatedaspartofyourlastresearchprojectinanyofthefollowingways?
Ontheotherhand,researchersneeddatasharing:42%declarethattheyrelyondatafromotherresearchers,and74%declaretheywouldbenefitfromaccesstoothers’researchdata.
2.2.2 HowdataaresharedRegardingapproachestoresearchdatamanagement,themajorityofresearchersdotakestepstomanagetheirresearchdataforpotentialfuturereuse(89%),however,
15
this isnotalwayscarriedout inaconsistent fashionwith49%statingthat thesestepsweretaken“sometimes”.Again,thereisnoprogressvisiblefrom2016.Figure6:Doyoutakestepstomanageyourresearchdataand/orarchiveitforpotentialre-usebyyourselfand/orothers?
When focussing on data sharing practices, research data sharing is carried outmostlybetweencollaboratorsonthesameprojects(80%).Only38%ofthosewhoshare, do it with researchers outside of their own project. This suggests thatresearcher do not adopt fully open data approaches, but rather discriminatoryapproach,sharingwithselectedpartnersonacasebycasebasis.Figure7:Haveyoudoneanyofthefollowingwithanyoralloftheresearchdatathatyouusedorcreatedaspartofyourlastresearchproject?
Thenumberofresearcherssharingdataintheirlastprojecthasremainedstable,withnogrowthshownoverthepasttwoyears.Observationalandexperimentaldatacontinue to be the main type of data used by researchers (48% and 35%respectively).Figure8:Categoriesofresearchdatausedinyourlastresearchproject.
16
Whenanalysingtheresearchdataformatsusedinresearchprojects,numericalandtext formats are the most popular (60% and 42% respectively) amongstresearchers.Figure9:Researchdataformatsusedorcreatedaspartofyourlastresearchproject
2.2.3 DataownershipRegardingresearchfunding,acountry/subjectspecificfunderremainstheprimarysourceoffunding.Whenobservingdataownership,thedatawasdividedintotwosections:1)beforepublicationand2)afterpublication.Themajorityofresearcherssaytheyownthedatapre-publication(62%).Figure10:Whatwastheprimarysourceoffundingforyourlastresearchproject?
Afterpublication,theperceptionthattheresearcherownsthedatadecreasesandthereisastronggrowthintheperceptionthatthepublisherownsthedatawithanincreasefrom4%(priortopublication)to35%(afterpublication).
17
Figure11:Whatwastheprimarysourceoffundingforyourlastresearchproject?
2.2.4 AttitudesacrossdisciplinesRegardingresearcherattitudestodatamanagementandsharing,Chemists(87%)andComputerScientists(86%)aremostlikelytobenefitfromdatasharingandhavethestrongestrelianceonresearchdatafromoutsideoftheirresearchteam(54%and58%respectively).Mathematiciansstandoutasthegroupwhoaremostwillingtoallowotherstoaccesstheirresearchdata(82%)andwhohavesharedtheirdatawithothersinthepast(79%).Figure12:Attitudestodatamanagementbysubject(I/II)
ComputerScientistsaremostlikelytoberewardedforsharing(60%)andastronglinkbetweendatasharingandrewardcanalsobeseenamongstChemists(58%)andLifeScientists(52%).Meanwhileonly30%ofresearchersinthefieldofSocialSciences,ArtsandHumanitiesandEconomicsstatedthatsharingresearchdataisassociatedwithcreditandrewardintheirfield.Intermsoftraininginresearchdata,
18
Chemists believe they have received the most experience in this area (51%).However,theperceptionofagenerallackoftrainingcanbeseenacrosstheboardintheothersubjects.Chemistsalsostandoutasthegroupthatmakestheirresearchdataavailabletopublisherstomakeitaccessiblewiththeirresearcharticles(79%).Those researchers in Medicine & Allied Health most believe there is a role forresearch data management specialists in research data sharing. Finally, bothMathematicians and Computer Scientists are the groups that most reuse otherresearcher’sdata(76%and66%respectively).Figure13:Attitudestodatamanagementbysubject(II/II)
Whenanalysingtheattitudestodatamanagementbyregionandage,itcanbeseenthatthoseinAsiaPacific(APAC)areleastlikelytosharetheirdata(69%)andthoseinWesternEuropeare finding itmoredifficult toaccessother researcher’sdata(44%).Thoseresearchersundertheageof36arethegroupthatmostagreethataccesstootherresearcher’sdatawouldbenefittheirownresearch(82%).However,thestrongwillingnesstoallowaccesstoresearchdatabytheover55sshouldalsobehighlighted(76%).
Figure14:Attitudestodatamanagementbyregion/age(I/II)
19
In Asia Pacific there is a stronger link between sharing research data and theperceivedcreditandrewardthatitbringstoresearchers(60%).Africa(90%),theMiddleEast(71%)andLatinAmerica(69%)standoutasthoseregionswherethereisaneedforresearchdatamanagementspecialistsinresearchdatasharing.Figure15:Attitudestodatamanagementbyregion/age(II/II)
2.2.5 DatarepositoriesRegarding the availability of data repositories, there were no less than 3449repositorieslistedonRe3data.orginOctober2019,upfrom2986in2018,outofwhichrepositoriesoflifescienceconstituted36%,naturalscience33%,humanitiesand socialscience21%andengineeringsciences10%.Thevastmajorityofdatarepositoriesprovideopenaccesstothedata–over94%ofallrepositoriesforwhichthisinformationisavailable.Figure16:Datarepositoriesbysubject
Figure17:Datarepositoriesbytypeofaccess
ThevastmajorityofdatarepositoriesislocatedinUS–39,2%(1048repositories)–almostthreetimesthenumberofdatarepositoriesofGermany–asecondcountryon the list with 14% of data repositories and the EU country with the highestnumberofdatarepositories.
Lookingattop10countrieswiththehighestnumberofdatarepositoriesonlyfourEU countries are prominent: Germany (381 repositories), United Kingdom (282repositories),France(103repositories)andNetherlands(56repositories).
20
Fromnon-EUcountriesCanada(252repositories),Australia(90),Switzerland(68),Japan(56)andIndia(50)areplacedintop10countries.
RomaniaandCroatiaarethecountrieswiththelowestnumberofdatarepositories.Figure18:Datarepositoriesbycountry,bluebarindicateEuropeancountries
2.3 OpencollaborationOpenscientificcollaborationreferstotheformsofcollaborationinthecourseofthescientificprocessthatgobeyondopendataandopenpublications.Measuringopenscientificcollaborationincludesmeasuringofdifferenttypeofoutputssuchasopencode,openhardware,theuseofcollaborativeplatformsbetweenscientistsandthe"citizen-science"phenomenon.Regarding the availability of scientific Application Programming Interfaces(APIs),thenumberofscientificAPIsisgrowingoverthecourseoflasttenyears.In2012therewasasuddenincreaseinanumberofAPIs(it´snumberedquadrupled)andsincethenanumberofAPIsgrewsteadilyto625scientificAPIsin2019.WhileAPIsarenotnew,theyareshowingcontinuousgrowthandbecomingcommonplacealsointhescientificcontext.
21
Figure19:NumberofscientificAPIs
JustasthenumberAPIs,numberofprojectsonOpenHardwareRepositoryisgrowingsteadily,amountingto323projectsin2019.
Figure20&Figure21:Openhardwareprojects
Anothergrowingphenomenonintherealmofopencollaborationisopencitizen-science. Open Science Monitor is using data from to science crowdsourcingplatformsSciStarterandZooniverse.ByOctober2019theSciStarterrecorded4506projects,upfrom4095in2018,outofwhichthemajorityreferredtolifesciencesandenvironment.Similarly,outof217projectsrecordedbyZooniversetillOctober2019,morethanhalfofthemconcernedlifesciencesandenvironment.
22
Figure22:ProjectsinScistarterbydiscipline
Figure23:projectsinZooniversebydiscipline
23
3 Driversandbarriers3.1 DriversTherearemanydifferentdriversatplay,whichcanfacilitateorhindertheadoptionofopenscience.Their identification is crucial to thedevelopmentof appropriatepolicyrecommendations,whicharetypicallybuiltaroundthese“levers”:theyaimtoenhancethedriverstoaccelerateadoption.Whiletherearegeneraltrends,externaltoscience,thatareimportantbecausetheyreinforce or damage open science, in the framework of the 28 case studiesperformed,theresearchteamattemptedtoidentifythespecificdriversorenablingfactorsthatwerebehindtheopenscienceinitiativesselected.Someofthefactorsmostcommonlymentionedinpriorresearchandstrictlyrelatedtoopenscience(non-includinggeneraltrendsaffectingopenscienceadoption)aresummarisedinTable5.Acrosstheanalysisofcaseswehave identifiedninemaindrivers thathavebeenclusteredinthreeoverarchingcategories:
1. Supporter: Refers to factors that describe the support of certainstakeholdersintheecosystemtotheinitiative.Suchsupportfacilitatesandencourages the adoption of open science behaviours by the researchcommunitythroughthecases.Underthesecategories, the factors include:Institutional support (i.e. when an institution is behind the initiative);publishers support (i.e.when a publisher or publishers are supporting insomeformtheinitiative);funderssupport(i.e.theinitiativehasthesupportofa fundingagency); industrysupport(i.e.whenthe initiativehasprivatesponsors);andcommunitysupport(i.e.thecaseissupportedbythescientificcommunity,citizensorfinalusersingeneral).
2. Intention: Includes factors where different design and implementationmeasuresweretakenintoaccount,thatis:whethertheinitiativewaseasilyto plug in already existing measures, processes and practices (i.e.commodity);andwhetheritimpliesmajororminorchangesintheexistingworkflows(i.e.changes).
3. Focus:Thethirdcategoryreferstohowtheinitiativetargetedthedifferent
collectives and includes:whether the initiativewas designed arounduserneeds,understandingtheirpracticespatternsandcommonbehaviour(i.e.user-centricity); or whether the initiative involved suppliers capable ofprovidetheproductorserviceandsupportadoptionorwasdesignedtakingintoaccountorinvolvingfuturesuppliers(i.e.supplier)
The description of all these factors, most commonly mentioned and uncoveredthroughtheanalysisof24cases,aresummarizedinTable5below.Wehavealsoincludedhowtheyaffectopenscienceadoptionthroughthecases.
24
Table5:Codingdriversanddescription
Category Drivers Description EffectsinopenscienceadoptionSupporter
Institutionalsupport Institutionalsupportsdesignswhenaninstitutionisbehindtheinitiative
Supportfactorsenablethegestationofthecasesinthefirstplace,butalsotheadoptionoftheinitiativewhensuchsupportcomesfromoneofthetargetgroupsoraffectstargetgroupsbehaviour.
Publisherssupport Publisherssupportdescribeswhenapublisherorpublishersaresupportinginsomeformtheinitiative
Funderssupport Funderssupportdesignswhentheinitiativehasthesupportofafundingagency
Industrysupport Industrysupportdesignswhentheinitiativehasprivatesponsors
Communitysupport Communitysupportreferstothesituationwhentherewasasupportbythescientificcommunity,citizensorfinalusersingeneralbehindtheinitiative
Intention Commodity Commodityreferstowhethertheinitiativewaseasilytopluginalreadyexistingmeasures,processesandpractices
Intentionfactorsenhanceadoptionbyeasingtheprocessofincorporatingsuchopensciencepracticetowardalreadyexistingbehavioursandprocesses.
Changes Changereferstowhethertheinitiativeimpliedmajorchangesintheworkprocessesofthetargetgroups
Focus Usercentricity Usercentricityreferstowhethertheinitiativewasdesignedarounduserneeds,understandingtheirpracticespatternsandcommonbehaviour
Focusfactorsfacilitateadoptionbytakingintoaccountbeneficiariesandstakeholdersviewinthedesignandthedevelopmentprocessoftheinitiative.
Supplier Supplierreferstowhethertheinitiativeinvolvedsupplierscapableofprovidingtheproductorserviceandsupportadoptionorwasdesignedtakingintoaccountorinvolvingfuturesuppliers
25
OverviewThe coding of the 24 cases developed in the framework of the Open Science Monitoruncoverednotonlythefactorsdescribedabovebutalsoitrevealsthattheyareunequallyimportanttotheadoptionofopensciencepractice.Acrossthethematicanalysisofcases,andcodingexercisedevelopedacrossthedatasources,threemajorgroupsoffactors,accordingtotheirrelevance,wereidentified.First,communitysupportandasupplierapproachappeartobethemostimportantfactorsexplainingthecaseadoptionandtakeup.Secondly,usercentricity and institutional support appears almost as much as relevant to explain thesuccessoftheopenscienceinitiativesselected.Finally,funderssupportandtheremainingfactorswouldexplainthemainresidualvarianceforthecaseadoption.Astheanalysisshowallfactorsuncoveredhadasignificantroleinthecases.However,wewillzoominnowinthedifferentopensciencetrendstounderstandhowdriverschangeandimpactdifferentlydependingonthetrend.
AnalysisperopensciencetrendTheanalysisofcasesreflectedsimilarpatternsacrossthediverseopensciencetrends(i.e.open access,open data, open collaboration and citizen science).Whenwezoom into thedifferenttrends,weobservethefollowing:OpenscientificdataLooking at the factors driving research data sharing, those benefits related to advancingsciencethroughcollaboration(66%)orreproducibility(57%)areconsideredtobemoreimportant toresearchersthanmeeting journal/publisherrequirements(28%)or fundingbodymandates(21%).Thefactthatsharingresearchdataalsoencouragesotherresearcherstomaketheirdatapubliclyavailablewasalsodeemedtobeofsubstantialimportance(55%).Figure24:Driversofsharingresearchdata
26
Gooddocumentationandcompliancewithbestpracticearethemostimportantfactorsfortrustingandusinganotherresearcher’sdata.Citationsinapublicationinformalliteratureorarepositorywithagoodreputationwerealsoconsideredtobeimportantfactors.Ontheotherhand,personalconnectionsarelessimportant.44%ofresearchersuseddatathathadbeensharedbyothersontheirlastresearchproject.Figure25:Howimportantwerethefollowingfactorswhendecidingtouseother’sresearchdata?
Formorethanhalfofresearchers,“someeffort”isrequiredtomakeresearchreusablebyotherssuggestingthatamoderateamountofworkhastobedonetomakedatareusableforothers. 27% of the surveyed researchers stated that “a lot of effort” was required,highlightingthecomplexityofdataprocessingandformattingasapotentialbarriertodatasharing.Figure26:Howwouldyoudescribetheefforttypicallyrequiredtomakeyourresearchdatare-useablebyothers?
Themajordriverspresentinthecasesrelatedtoopenscientificdatashowtherelevanceofinstitutional,communityorfundersupport,user-centricdesignoftheinitiativeandasupplierapproachwherethefutureexploitationisclear.Thequalitative analysis and fine-grainedanalysis through the cases reflect the followingdriverswithinopenscientificdata:
27
Drivers1. Contextoflowresearchanddevelopment(R&D)productivity:Opensciencemodelis
astrategythatcontributestofightlengthy,costly,lowsuccessrate,highattritionrates,andcomplexityinR&Dprocesses,inparticularindrugdiscoveryprocesses.
2. Newcomputationalapproaches:Theincreasingvalueofaggregatingdatarelyonnewcomputationalapproachesthatrequiredifferentgovernancemodels
3. Responsiblesharingofscientificdataisessentialtoacceleratescientificdiscovery4. Socialormoralobligation:Transparencyinclinicalresearchdatacanimprovepublic
healthandsafety;itisalsoconsideredasocialobligationforthecitizens'thatvolunteerthemselvestothosetrials
5. Pressurefromshareholdersabouttheliabilityofunpublisheddata
1. About fighting lengthy, costly, low success rate: The main drivers seen in OpenTargets,PistoiaAllianceandYODAaretoeffectivelylowerthebarrierstoR&DinnovationandincreaseR&Dproductivityinbiomedicalresearch.AsGSKdeclares:"Webelievethatharnessingthepotentialof"bigdata"andgenomesequencingthroughthiscollaborationcould helpus dramatically improve our success rate for discovering newmedicines,"(GSK,2017).
2. New computational approaches and the increasing valueof aggregatingdata inbiomedical research: The explosion of data availability in biomedical research hasincreasingly rendered it a data-driven science. As a result, drug discovery effortsaccumulatetodayfarmoreinformationthanforpreviousdrugs.Therearevastvolumesofdataassociatedwithdrugdiscoverythatisbeingaccumulatedinpharmaceuticalfirms.Throughdataaggregation,anintegrationofalltheheterogeneousandcomplementaryevidenceavailableanda combinationof computational approaches, theOpenTargetscaseshowhowopenresearchdataiseffectivelyfacilitatingscientiststosystematicallyidentifyandassessalltheevidenceavailablethatassociatesadrugwithdiversediseases.
3. Aboutsocialandmoralobligation:Transparencyinclinicalresearchdatacanimprovepublichealthandsafety.AsthecaseofYODAreflectsresearchdatasharingisconsideredasocialobligationforthecitizens'thatvolunteerthemselvestoclinicaltrials.YODAisalsodrivenbytheviewthatpatients,healthcareproviders,andthegreaterlifesciencesindustrywillbenefitifacademiacanprovideindependentreviewsofdatarelevanttothepotential benefits and harms of industry products, and that these analyses allowphysiciansandpatientstoinformtheirdecisionsonthemostcomprehensiveandrecentevidenceavailable.
4. Aboutthepressurefromshareholdersabouttheliabilityofunpublisheddata:Onereason to believe that companies agreed to join YODA, but also other frameworks todisseminateclinicaltrialdata,isthattheirshareholdersareincreasinglyworriedaboutthe liability of unpublished data. As reported by different sources, owners of drug-companiessupportthebroaddisseminationofclinicaltrialsresultmakingsurethattheirresultsarefullyreported(TheEconomist,2015).Thereasonisthatlong-terminvestors
28
prefertoreducethelevelofriskintheirportfoliobyhavingalltheinformationabouttrialsfullydisclosed.
OpencollaborationOpencollaborationisanumbrellaconceptthatembraces:opensourcehardware,opencodeandreproducibilitycases.Themajordriverspresentinthecasesrelatedtoopencollaborationshowtherelevanceofauser-centricandsupplierapproachbehindtheinitiativeandthesupportfromcommunityandinstitutions.Regardingopensciencehardware,themore‘granular’driversthatwerefoundedacrosstheanalysisofWhiteRabbit(WR)casesandthecross-analysisofexistingopensourcehardwarelicencesare(includeheresourcestothecases):
Drivers1. To maximise the value of the technology for society by avoiding duplicative
investmentsonthesamehardware
2. Toallowtechnologyflexibility
3. Toallowextensivepeer-reviewofthetechnologydesigntoincreasetechnologyqualityandreliability
4. Toallowreusabilityofhardwaredesigns
5. Strongandhistoricalcommitmentwithopensciencebytheinitiativeleader
1. Tomaximise societal impact: When CERN decided to developWR as open source
hardware to solve a problem in their accelerators, they had the intention towidelydisseminate the technology tomaximise theirusebyotherorganisations inresearch.Opensourceproductsinsoftwarearerecognisedforofferingdecreaseddependencyonmonopoly suppliers (Bruns, 2000; Kogut and Metiu, 2001). This is analogous tohardware,anditisespeciallyvaluableforscientists.ByadoptingasimilarapproachforWR development, CERN intended to develop a platform to integrate a variety ofcontributionsfromdispersingstakeholders,whilemovingawayfromavendorlock-insituation,wherescientificinfrastructuresbuilddependencieswithtechnologyprovidersforhighlyspecifictechnologies.
2. To allow flexibility: The Open source hardware approach provides the requiredflexibility forscientificorganisations,whichisanessentialcharacteristic forscientistswhoneedcustomising,never-before-seenequipmentfortheirexperimentalendeavoursin uncertain andmodular environments. Such flexibility arguably leads to better andfaster progress of science (Pearce, 2014). By developingWR in the open, CERNwasseeking topave theway for thehigh customisationpotentialof the technology,whilefosteringeffectivepeer-reviewbyacommunityofexpertswhocanprovideusefulinput,testcases,andfeedbackinanopenspace.
29
3. Toallowextensivepeer-reviewof the technologydesign to increase technologyquality and reliability: By developing open hardware, the research infrastructure(CERN)was seeking to a dynamic peer-review of the technology by a community ofexpertswhocouldprovideusefulinput,testcases,andfeedbackinanopenspace.
4. To allow reusability: By adopting an open hardware approach, CERN was alsoexpandingthepossibilityofreusingthedesignsofotherelectronicsengineersworkingin experimental physics laboratories. The idea was to increase the efficacy of thetechnologybyreducingthenumberofdifferentteamsworkingindependentlyonsimilarproblemsandfocusdisperseyetcomplementaryknowledgeonacentralgoal.
5. Strongandhistoricalcommitmentwithopensciencebytheinitiativeleader:The
principleofopennessisstatedinCERN’sfoundingconvention,andtheorganisationhasbeenarelentlesspioneerinthisregardsincethereleaseoftheWorldWideWebunderanopensourcemodelbackin1994.CERNhascontinuouslyembracedtheprinciplesofopen science, such as open access with the Sponsoring Consortium for Open AccessPublishinginParticlePhysics-SAP3andallLHCpublicationshavebeenpublishedunderOpen Access conditions; open data, setting up the Open Data Portal for the LHCexperimentsorZenodo,a freeOpenDatarepository, launchedbytheorganisation foruse beyond the high-energy physics community; Invenio, an open source librarymanagementsoftwarepackageandtheiruseofopensourcelicences(NilsenandAnelli,2016; Murillo and Kauttu, 2018). Developing WR as an open source hardware wasalreadyinlinewiththephilosophicalmottoofCERNresearchactivity.
Reproducibility(includingpre-registration)
Drivers1. Awarenessabout‘reproducibilitycrisis'
2. Socialimpactmaximization
3. Strongcommitmentbyanorganisationworkingtofosteropenscience
1. Awareness about ‘reproducibility crisis’: Nature published in 2012 a study that
reviewed ten years of research, where scholars found that 47 out of 53 biomedicalresearchpapersoncancerresearchwasirreproducible(Prinzetal.,2011).Fouryearslater, in 2016, Nature surveyed 1576 scientists, finding that more than 70% ofresearchers failed to reproduce another scientist’s experiments andmore than 50%failed to reproduce the results of their experiments. Also, a famous study wasimplementedin2015inthefieldofPsychologywhenanopen,registeredempiricalstudyofreproducibilitywas launchedand270researchersaroundtheworldworktogetherandtrytoreplicate100empiricalstudiespublishedinthethreetopPsychologyjournals(theReproducibilityProject).Asa resultof thisgrowingawareness, reproducible researchhasbecomeapervasiveobjective in the research policy agenda at different governmental levels, includingfundersand journalpolicies. In thestruggle to improvethereproducibilityofscience,
30
there have been different initiatives from the scientific community to fight thereproducibility crisis, which include our case: REANA, a reusable and reproducibleresearchdataanalysisplatform.
2. Socialimpactmaximization:BydoingREANA,CERNknewnotonlytheparticle-physicscommunitywouldbenefit,butotherswouldalsodo,andthattheprojectwouldbenefitthevastscientificcommunity.
3. Strongcommitmentbyanorganisationworkingtofosteropenscience:REANAwasbornwiththegoaltofightthereproducibilitycrisisandasanaturalnextstepofCERN’sefforttomoveforwardtheiropenscienceeffortstothenextstep.
CitizenscienceFinally,themajordriverspresentinthecasesrelatedtocitizenscienceshowtherelevanceof community, institutional or publisher support and user centricity. The cases assessedshowed a successful adoption thanks to the support of the scientific community ororganization behind it and the user-centric design that facilitated the contributions by adistributedbaseofcitizensaroundtheglobe.
Drivers1. Increasedavailabilityofdatasetsinmanyresearchdisciplines
2. Motivationbyprojectownerstoproduceannotateddatasets
3. Usabilityofthetechnologyinterface
ThecasessuchasZooniverseshowthatthegrowthofcitizenscienceprojectsisbeingdrivenbytheincreasedavailabilityofdatasetsinmanyresearchdisciplinesaswellastheuseofweb-connectedcomputerandmobiletechnology.Ontheotherside,projectownersarealsomotivatedbytheneedtoproduceannotateddatasets forresearchpurposes.Finally, it isimportantthatthecitizenscienceprojectsmakeiteasytostartnewcitizenscienceprojectsandcontributetothemwithoutanysophisticatedtechnologicalexpertise.
3.2 BarriersThere aremany different drivers and barriers at play,which can facilitate orhinder theadoptionofopenscience.Theiridentificationiscrucialtothedevelopmentofappropriatepolicyrecommendations,whicharetypicallybuiltaroundthese“levers”:theyaimtoremovethebarriersandtoenhancethedrivers.Whiletherearegeneraltrends,externaltoscience,thatareimportantbecausetheyreinforceordamageopenscience,intheframeworkofthecasestudiesperformed,theresearchteamattempted to identify the specific drivers or enabling factors thatwere behind the openscienceinitiativesselectedandthebarriersorbottlenecksthatthedifferentprojectstudiedneededtoface.
31
Fromthecasestudies,anumberofrecurrentbarrierswereanticipated.Theyarelistedbelow:Category Barrier DescriptionMicro Lackofuserorientation Whenatool,method,approach,orcaseasawhole
wasdesignedwithouttakingintoaccountorinvolvingthefutureusergroups.
LackofAdoption Whenatool,method,approach,orcaseasawholeisonlyusedinaselectedsetoffields,usergroupsorinstitutes.
Technologicalbarrier Whenatool,method,approach,orcaseasawhole,showsthatthetechnologicalaspectsaretoocomplicatedforusers.
Lackofskills Whenatool,method,approach,orcaseasawholerequirestoospecificortoocomplicatedskillsofusers.
Cultural&behaviouralbarrier
Whenatool,method,approach,orcaseasawhole,doesnotfitinthewaythattheresearchisusuallycarriedout.
Lackofawareness Whentheexistenceofatool,method,approach,orcaseasawhole,isonlyknowninasmallusergroup.
Meso Lackofinstitutionalsupport
Whenatool,method,approach,orcaseasawholeisnotsupportedbyinstitutionalfunding.
Lackoffunderssupport Whenatool,method,approach,orcaseasawholeisnotsupportedbyfundingfromresearchfunders.
Lackofpublishers’support
Whenatool,method,approach,orcaseasawholeisnotsupportedbypublishers.
Macro Lackofbusinessmodels Whenatool,method,approach,orcaseasawholeisnotsupportedbyabusinessmodel.
Lackofstandards Whenitisnotclearwhetheratool,method,approach,orcaseasawholeisstandardized,andstandardsareeasilyavailable.
Difficultyinupscaling-fragmentation
Whenatool,method,approach,orcaseasawholeisnoteasytoupscaleinotherinstitutes/usergroupsandhencetheuseremainsfragmented.
Lackofdiversity Whenatool,method,approach,orcaseasawholeisnotsuitabletootherindividualswithdifferentskills,backgrounds,traditionsandbeliefs.
For each of the case studies, the research team has assessed whether the barrier wasapplicable.Ascorefrom1to3wasgiven,a1indicatingthatthisbarrierscoredhighinthecase,a2meaningthatthebarrierscoredmedium,anda3whenabarrierwasn’trelevantatall.Thescoreswereassignedinagroupdiscussion,basedonthecasestudystory.Inthefigurebelowisindicatedwhichofthebarrierswasgivena1(scoredhigh)mostoften.It isclear that ‘soft’barrierssuchasculturalandbehaviouralbarriersaremost frequent,occurring in 13 out 24 cases. The second highest barrier is difficulty in upscaling/
32
fragmentation(11/24),andthethirdoneisthelackofbusinessmodels(9/24),forwhichthereareahighnumberofscores2aswell.Thebarrierthathasthelowestscoreistechnologicalaspectsofatool,method,approachorcase;inonly2outof24casesthisbarrierisscoreda1.Also,thefinancialsupportbyfunders(2/24)orpublisher(5/24)israrelyscoredasastrongbarrier.Again,inthecasewheretherewasanequalnumberinscore1,thenumberofscores2wasdecisive.Thebarriersasscoredhereconfirmearlierstudiesonopendatapractice.Figure27:Barriersacrossthecasestudies
Thecrossanalysisofcasesalsoshowsthatthereisabroadrangeinthenumberofbarriersthatareexperiencedpercase.Inthefigurebelow,thecasestudiesarepresentedwiththetotalnumberofhighbarriers(score1).Ascanbeseenupto8outof13barriersapply,buttherearealsocaseswhichscorezerohighbarriers.Figure28:Numberofbarriersacrosscasestudies
33
The cases with the most barriers are Two use cases, As predicted, Mendeley and Openmetadata. They cover the research phases of data-integration, pre-registration andpublication.However,itseemsthatthebarriersinthecasesarenotnecessarilydependentontheresearchphase,becausethecaseswiththelowestnumberofhighbarriers(score1)arealso inpublication:YODAandORCIDwithzerobarriers,andZooniverseandGOFAIR(onemajorbarrier).Fromthenumberofbarriers,itseemsthatbroad‘technological’casessufferfromlessbarriersthan‘use’cases.Thisisconsistentwiththehighnumberofculturalandbehaviouralbarriers,andthefragmentation.Inordertofindpatternsinthebarriers,thescore1wasassessedacrossbarriercategoriesandopensciencetrends.Thescoresarenormalizedforthenumberofcasespertrend,andthenumberofsub-barrierswithinthemicro-,meso-ormacrobarriers.Theyarepresentedbelow:Figure29:Barrieracrossopensciencecategories
ThebarriersaremostprominentforOpencollaborations(micro-level)andCitizenscience(macro-level).ThisreflectsthefactthatthereisstrongpolicysupportforOpenaccess,alsoatmeso-level,butthattheactualbehaviouratthemicro-levelislaggingbehindinchange.Forcitizensciencethesituationisthereverse:thereislittlepolicysupportorinfrastructureforcitizenscience,whichshowstobetherelativelystrongestbarrier.The lowestbarrierscoresareforCitizenscienceandOpendata(meso-level),whichbothseemstobeodd,sincebothcitizenscienceandopendataareinaprematurestateinHEI’s.Otherwise,thebarriersaremoreorlessevenlyspread.
34
4 ImpactThissectionpresentsthekeyfindingsontheimpactofopenscienceasemergingfromthecasestudies.Inaddition,thesurveyprovidesadditionalinsightontheimpactofdatasharingpractices.Whenlookingattheconsequencesandimpactofdatasharing,overathirdofresearcherswerecontactedbyanotheruniversity/instituteaftersharingdata.10%werecontactedbyacompany,highlightingtheadditionalpositiveconsequencesofdatasharing.Figure30:Thinkingaboutthemostrecentresearchprojectonwhichyoushareddata,didindividualsoutside of the research teamcontact you concerning the data that you shared?Iwas contactedbyresearchersfrom
Furthermore, over a third of researchers believe that sharing data promoted newcollaborationswithresearchersintheirdiscipline.Figure31: Still thinking about themost recent researchproject onwhich you shareddata, do youbelieveanyofthefollowinghappenedasaconsequenceofsharingthedata?
Collaborationwith[academic]researchers…
35
4.1 ScientificimpactThereissubstantialevidencethatopenscienceiscorrelatedwithbetterscience,mainlybyremovingmistakesatanearlierstage,enhancingtheproductivityofsciencethroughdatareuseandreducingthecostsofsciencethroughcollaboration.Themaindifferentpositiveeffectsofimplementinganopenscienceapproachcapturedintheliteraturesofarinclude:
• Fasterdiscovery:Fasttracktodiscoverythankstoexternalcontributionsandcomments.Knowledge mining of open access literature can automatically classify, analyse andreason on existing literature,making newdiscoveries and connections that currentlyhappenveryseldomandunexpectedly.
• Increasedefficiency:Openscienceincreasestheefficiencyoftheresearchsystembecauseithelpstoreducetheduplicationofcosts,aswellascostsstemmingfromthecreation,transferandre-useofdata.Itallowsmoreresearchtobecreatedbythesamedataandthusboostsreturnonpubliclyfundedresearch(OECD,2015;Lyon,2009;Whyte&Pryor,2010).
• Greaterproductivity:Contributionsfromvolunteerstodatasets.
• Morerobustfindings:Willingnesstoshareresearchdataisrelatedtothestrengthoftheevidenceandthequalityofreportingofstatisticalresults(Wichertsetal.,2011).Intheabsence of open code, computational science as practiced today does not generatereliableknowledgebut“breezydemos”(Ioannidis2005).
• Increasedtransparencyandreplicationofstudies:Openscienceraisestransparencyandquality in the validation of research results and multiplies the opportunities forreplicabilityandvalidationofscientificfindings(Franceschet&Constantini,2010;OECD,2015;Fecheretal.2015).Bymakinginformationonmethods,protocols,anddataeasierto peer review, and by strengthening the scope of thematerial published (includingnegative results), open science offers greater scrutiny as it allows formore accurateverificationofresearchresults(Lyon,2009;Whyte&Pryor,2010).
• Increasedcollaborationandscientificinterdisciplinarity:Theimplementationofanopenapproachinscienceenhancescollaborationacrossinstitutional,nationalanddisciplinaryboundaries, and it fosters the exchange of information and expert knowledge (OECD,2015;Whyte&Pryor, 2010). Scientists are re-scaling the levelof contribution in thescientificprocessbycuttingitintosmallpiecesandenablingmicro-contributionsonamacro-scale.Hyper-specialisationisforeseenbyscientistsinmanycasesandtheylookfor micro-contributions from ‘citizen scientists’, which could lead to a furthergamificationofScience(VonAhn&Dabbish,2008;Szkuta&Osimo,2016).
• Minimisepublicationbiases:Minimisepublicationbiasesifstudiesthatreportnegativeornoeffectsarenotpublishedintraditionalvenues,butaremorelikelytobepublishedonnovelpublicationplatforms(Boultonetal.,2012).
• Newresearchfieldsandenhancedopportunities:BigDataishelpingtocreatebrandnewfieldsofscience:computationalchemistry,biology,economics,engineering,mechanics,neuroscience, and geophysics. Open science enables sharing of complex models andsimulationsbasedonlarge-scaleopendataanalysis(Lyon,2009;Boultonetal.,2012).
36
Modern computers can identify highly complex, unperceived relationships. Thus,technology is supporting traditional ways of doing science, and enables enquiry byconstructinghypothesesafteridentifyingrelationshipsinthedatasets.
• Newemerginginstrumentstoevaluatescience:Overthelasttenyears,withtheadoptionofweb 2.0 tools by scientists, new instruments are emerging to assess the quality ofscience outputs (Herman et al., 2015). Today, reputation (which is key to scholarlyresearch) is intrinsically linked to the model of journal publication. The impact ofresearch ismeasuredbycitationsandpatents.Recently, therehavebeenexperimentswith new evaluation and reputationmodels, such as data citation and data journals,altmetricsandpost-publicationreviews(SzkutaandOsimo,2016).
Regardingthepotentialnegativeimpacts,theliteraturesuggeststhatopensciencemayalsoposevariousthreats.Mostoftheconcernsarerelatedtoresearchquality:Withtheopennessoftheresearchprocesstherearesomeconcernsabouttheeffectivenessandpracticalityofqualitychecks.Thereisawidespreadacknowledgementthatassessingthequalityofsuchvastvolumesandrangesofscientificmaterialsischallenging(Whyte&Pryor,2010).ThemainquestionsthatpersistaftertherevisionofthedifferentstudiesthatOpenScienceMonitorsoughttoaddressthroughtheselectionandanalysisofasetofcasestudiesacrossopensciencetrendsare:
• Howdoesopenscienceimpactthequalityofresearch?
• Underwhichmechanismsdoesopenscienceincreasetheefficiency,effectivenessandproductivityofresearch?
• Hownewformsofevaluationofscienceaffectscientificperformance?
• Howopenscienceimpactsmultidisciplinaryresearch?
4.1.1 CrossanalysisofcasesAccordingtotheassessmentofthecases,in19out24casesthescientificbenefitofthetool,method,approachorcaseisconsideredhighbyascore1.Onlyfivecasesscorea2-PistoiaAlliance,Twousecases,DataCite,RDAandOpenmetadata.Thesecasescoverabroadrangeof research phases so again there is no specific phasewhere open science is critical forscientificimpact.Figure32:Scientificimpactsaslistedinthecrossanalysisofpolicies
Fasterdiscovery
Increased
efficiency
Greater
productivity
Morerobust
findings
Increased
transparency
andreplication
ofstudies
Increased
collaboration
andscientific
interdisciplinari
ty
Minimise
pub lication
biases
Newresearch
fieldsand
enhanced
opportunities
Newemerging
instrumentsto
evaluatescience
GBIF x x x x x Hypothes.is x x x x x Mendeley x x xWhiteRabbit x x x xOSHlicences
37
Fasterdiscovery
Increased
efficiency
Greater
productivity
Morerobust
findings
Increased
transparency
andreplication
ofstudies
Increased
collaboration
andscientific
interdisciplinari
ty
Minimise
publication
biases
Newresearch
fieldsand
enhanced
opportunities
Newemerging
instrumentsto
evaluatescience
Opentargets x x x Zooniverse x x x x Mosquitoalert x x x Pistoiaalliance x x x x xGlobalAlliance Twousescases UtrechtOSP x x x xRDA x xELN Zenodo GOFAIR x x Aspredicted RSE F1000 YODA x x x OpenMetadata x xREANA ORCID x x xDataCite
4.1.2 QualitativeinsightfromcasesTheopendatacasestudieswereallbottom-upinitiativesinthebroaderbiomedicalfieldinwhich PistoiaAlliance andOpenTargetswere industry-driven; andYODAwas led by anacademicorganisation.Therefore,itisnosurprisethatthescientificimpactofthesecasesrelates to faster discovery and increased efficiency. Thismay provide an answer to thesecondkeyquestion.Allcasestudiesshowthattheyaimtobridgepracticesthatarenotnecessarilytakenupin‘normal’scientificpractices. It takestime,effortandfundingtosetup infrastructuresordeveloptoolsthathaveprimarilyabridgingcapacity.Inallcases,industryistakingpartandisbringinginfunding.LargesustainableinfrastructuressuchasCERNprovideenvironmentsinwhichopenscienceinitiativesflourishbenefittingtheoverallscientificcommunity.In the current cases, except from YODA, there seems to be low impact on minimisingpublicationbiases.Thismayhavetodowiththefactthattheyareallbridgingactivitiesthatdonotoperateinthecoreofthescientificpublicationarena,wheremostscienceistakingplace.Openscientifichardwareisanemergingformulabeingexploredbyresearchinfrastructuresthat produce experimental technologies. The example ofWhite Rabbit (WR) providesrelevantinsightsinhowthepublicationofevenafirstversionoftheWhiteRabbitdesigns,
38
resulted in the uptake by the scientific industry, and other scientific infrastructures andfacilities.Atpresent,accordingtolastupdateintheWRrepository,30organisationshaveimplemented WR and 15 others are current evaluating if the technology can fit theirpurposes. In addition, nineR&Dprojects have appliedWhiteRabbit technology for theirscientificpurposes.ExamplesofthisscientificapplicationincludeGSIwhereWRbecamethetimingsystemoftheFacility forAntiprotonandIonResearch(FAIR),an internationalacceleratorbased inGermany of an estimated investment of $2 billion. Also, the Cubic Kilometre NeutrinoTelescope (KM3Net), a European research infrastructure located at the bottom of theMediterraneanSea,usesWRinordertosynchronizethedetectorunits.OpenTargetsisaninnovative,large-scale,public-privatecollaborationonpre-competitiveresearchthatprovidescomprehensiveanduptodatedatafordrugtargetidentificationandprioritisation.OpenTargetsintegratespubliclyavailableinformationanddatarelevanttotargetsanddiseasesintheOpenTargetsPlatform.Itperformshighthroughputexperimentalprojectsthatgeneratetarget-centreddatainphysiologicallyrelevantsystemstounderstandcausal relationships between targets and diseases in three therapeutic areas: Oncology,Immunology, and Neurodegeneration. A cornerstone of this public-private collaborationfrom thebeginning is anagreementamong theorganisations that alldataand resourcesgeneratedwithinOpenTargetsshouldbemadeavailablerapidlyinthepublicdomaintotheentire scientific community. The Open Targets Platform is an open access “Google”-typesearchenginethatextensivelysearchesassessandintegratesthehugequantityofgeneticandbiologicaldataavailable.Theplatformsupportstwomainworkflows:thefirstistarget-centric;thesecondisadisease-centricworkflow.Theexplosionofdataavailabilityinbiomedicalresearchhasincreasinglyrendereditadata-drivenscience.Thereareextensivevolumesofdataassociatedwithdrugdiscoverythatisbeing accumulated in pharmaceutical firms. Drug repositioning is one strategy in drugdevelopmentthatseekstoexpandtheindicationspaceforasuccessfuldrugorfindanewindication foradrugthatwasnotsuccessful in theclinical trials.OpenTargets facilitatesidentificationofpotentialrepositioningopportunities.ThroughtheOpenTargetPlatform,scientists are able to systematically identify and assess all the evidence available thatassociatesadrugwithdiversediseases.Basically,theaggregationandintegrationofalltheheterogeneousandcomplementaryevidenceavailable in theplatformhelpprioritiseandanalysepotentialdrugrepositioning.AsreportedbyKhaldakaretal.(2017),OpenTargetshas been able to uncover 2,540 potential new indications for 791 existing drug targets.Among these 2,540 new indications, 1,366 are for rare diseases where the target isassociatedwithmorecommondiseases.RegardingtheuseoftheOpenTargetsPlatform,900uniqueIPaddressesaccesstheOpenTargetsPlatformeveryweek. Somemetricsavailable aboutOpenTargetplatformusage,fromApril2016-March2017,revealthattheplatformisusedsubstantially.ThemetricsarealsoalignedwiththequalitativefeedbackreportedbytheOpenTargetsteamfromplatformusers.Thescientific impactof theevidenceavailable in theplatformissubstantial.Associationsbetweendrugtargetanddiseasearethemainfocusforbothnewdrugdevelopmentanddrug
39
repositioning.Scientists seekevidence supporting target-diseaseassociations that canbestored in structured databases and integrated to obtain a comprehensive assessment intargetvalidationstudies(Kafkasetal.,2017).ThefundamentalconclusionfromtheOpenTargetscasestudyisthattheinherenttensionbetweenthegoalsofscientificopennessandcommercialexploitationdoesnotnecessarilyimplyincompatibility,butaneedtoidentifysophisticatedsolutionsthatadequatelybalancethedivergentinterestsatdifferentphasesofscientificprocesses.TheYaleUniversityOpenDataAccessProject(YODA)was launchedbytheCenter forOutcomesResearch andEvaluation (CORE) fromYaleUniversity ofMedicine in2011, toaddresstheproblemofunpublished,andselectivelypublished,clinicaltrialresearchdata.YODA has the goal to increase access and availability of clinical research data whilepromotingthere-useofsuchdatatocreatenewknowledge.YODAhasbeenactingasanindependent,unbiasedbridgebetweenresearchersandclinicaldatafrompharmaceuticalcompaniesandconsumerbusinesses.Todoso,YODAhasdevelopedamodeltomakedataavailabletoscientists,whichismediatedthroughtheYODAteam.YODAfacilitatesaccesstoclinical trial data made available by third party Data Holders to promote research thatsupports scientific endeavours and public health. YODA partnered with Medtronic andJohnson&Johnsontoshareitsdata.Metricsonalldatarequests, includingresponsetimesshowsthatcurrently, thereare90clinicaltrialslistedforarequest.DifferenttypesofstakeholdershaverequestedYODAforaccessingclinicaltrialsdata, includinggovernmental institutions, industry,andacademia.Themajority of requests for accessing clinical data trial come from academia (~90%ofrequests),andusuallytherequestsareaccepted.Despite governments and regulatory agencies efforts towards transparency of clinicalresearchdata,agreatdealofclinicaltrialdataispartiallyorcompletelyundisclosed.Thefirstlessonisthatregulatoryenforcementcanplayacriticalroleinincreasingdisclosure.TheframeworkfromYODAcanalsobeappliedforotherfieldsthatexpandmedicalresearch,wheredataisconsideredsensitiveandwherestakeholderssharesignificantconcernsthatpreventthemfromdisclosingrelevantdatatoacceleratescientificdiscovery.Tobere-usedfor scientific goals, clinical research data has to be disclosed with complementary andcomprehensivesourcesofclinicalresearch,whichhasalevelofevidencegranularitythatisexceedsjournalpublications;annotatedcasereportforms;datasetspecifications;protocolswithanyamendmentandreportingandanalysisplan;amongstothers.YODAhascarriedoutactivedisseminationabouttheprojectandpotentialitiesofre-usingclinical data research. Sharing the data is insufficient to accelerate its re-use; activedisseminationandengagementoftheresearchcommunityareneededtofosteranalysisofalreadyexistingdata.Pistoia Alliance is an industry-driven partnership created by large pharmaceuticalcompanies (GlaxoSmithKline (GSK), AstraZeneca, Pfizer, and Novartis) in 2009 withoutgovernmentinterventiontocooperateandshareresourcesinthepre-competitivephaseofdrugdevelopment.TheAlliancehasbeenexpandingsincethenacceptingapplicationsfromboth private and public who share the ideology of sharing data and resources of drugdiscovery research pipeline. Pistoia started to generate mutual standards in industry,
40
ontologiesandwebservicesthataremadeavailableunderanopen-source frameworktoacademicinstitutions,vendorsandcompanies,tofacilitatedatasharing,datarepresentation,text-mining activities and improve the R&D efficiency of the drug discovery anddevelopment process.Withmany projects underway since its foundation, covering fromInternetofThings (IoT) to ‘ontologiesmapping,' thealliancepools resources frommanycompanies,andopenlysharestheoutputofitsworkwiththeoutsideresearchcommunity.ThealliancehasdesigneddifferenttypesofmembershipinordertoengageadiversesetofinterestinenhancinglifescienceR&D.WhenPistoiaAlliancestarteditsactivityin2009,thepartnershipsetouttoworkaroundfour major technology pilots: Sequence services; SESL (Scientifically Enriched ScientificLiterature);ElectronicLabNotebook(ELN);VSI(VocabularyStandardsInitiative).PistoiaAlliancehasaprojectportfolio,whichiscontinuouslyevolvingwithprojectsenteringthepipelineperiodically.Inthecasestudy,oneusecase(project)wasanalysedtoshowtheimpact of the collaboration:Hierarchical Editing for LargeMolecules (HELM), which is aglobalstandardnotationthatismachine-readableforlargemoleculesoriginatedatPistoiaAlliance4 . HELM is now a FAIR standard. The project team includedmembers from 23differentorganisationsinthelifesciencespaceintegratingalargeanddiverseecosystem.TheHELMtechnologywasofficiallyreleasedintotheopensourcecommunityconsistingofaGitHubrepositorycontainingthesourcecodeofthetoolkitandeditor,alongwithawebsite (www.openhelm.org) containing the notation language specification, a free appletversion of the editor, training videos and user guides. HELM notation became an openstandard,publishedopenlyandfor free.HELMis increasinglybeingadoptedthroughthecompanies and academic organisations working on biopharma, including also solutionprovidersandregulatoryagents(PistoiaAlliance,2018).The uptake ofHELMhas been rapid and includes today a vast rangeof organisations ascontentproviders, informaticsvendors, and life science companies.ThebenefitsofusingHELM include facilitating the work of scientists willing to employ computationalmanipulations and calculations with complexmolecules. HELMmakes easier researchertasks.Bydoingthis,HELMfacilitatestheintegrationofprivateandpublicdataeasierusingthestandard.
4.2 IndustrialimpactMovinginnovationsfromscientificdiscoverytothecommercialisationofnewproductsandservicesinvolvesnumerousstakeholdersandchallengestoovercome(e.g.,CarayannisandCampbell,2009).Atoneendofthespectrumthereisaheavyconcentrationofgovernmentinvestment inbasicresearch; andat theother, in thecommercialmarketplace, there is amuchhigherlevelofindustryinvestmentindirectproductdevelopment(NationalScienceFoundation,2017).Betweenbothofthem,thereisthepathforpotentialinnovationbutfirmsand industry players need to invest resources to develop them to a stage where theircommercialpotentialcanbeexploited.Successfullygoingthisprocessrequirescomplextiesand relationships through the innovation spectrum, which usually involves contractualrelationships with non-disclosure-agreements (NDAs), Intellectual Property issues and
4AboutHELM:www.openhelm.org
41
otherformalapproachesforcollaborationthatguaranteethattheinterestsofthedifferentstakeholdersareprotected.Thesociologiesofscientificcommunitiesareoftenuniqueandthepursuingofopennessandfreesharinginthescientificprocesscanbeatoddswiththeinstitutionallogicofthebusinessworld. On the one hand, the norms of Science, which were famously described by thesociologist Robert K. Merton (1973), highlight the cooperative character of inquiry,emphasising that knowledge growth stems from a collaborative process where fulldisclosureof findingsandmethodsare fundamental.Ontheotherside, for-profitseekingfirmsfollowtherationaleofprivateinvestmentmodelsthatseektoprotectR&Dinvestmentstoextractrentsfromtheresultingnewproductsandservicesinthemarket.Forinstance,thetraditionalbusinessmodelformostcommercialhardwarecompaniesisbasedonkeepingthedetailsofthedesignsecrettobeabletomaximisethemargins(Pearce,2017).In the current context, this historical ‘tension’ coming from the sociology of scientificcommunityandbusinessworld,whichhasbeenextensivelystudiedbyscholarsespeciallysince the ’80safter theBayh-DoleAct,needs tobe re-examined in the lightof the recenteventsboostingopenscience,wherepolicy-makersandthescientificcommunityitselfhavestarted placing greater emphasis on the transparency and public scrutiny of scientificprocesses(OECD,2015).Opensciencemandatesacrossthescientificcommunitystresstheefforts by scientists and governments to make the primary outputs of publicly fundedresearchresults(bothdataandpublications)publiclyaccessibleinadigitalformat,withoutany (or minimal) restriction in order to accelerate knowledge growth, while enhancingtransparencyandcollaboration.Theacademicandgreyliteratureassessingtheimpactofapplyinganopenscienceapproachhighlights as positive impacts the increased collaboration and innovation. Theimplementationofanopenapproachinscienceenhancescollaborationacrossinstitutional,nationalanddisciplinaryboundaries,and it fostersthesharingof informationandexpertknowledge(OECD,2015;NESTA,2010).Byfacilitatingaccesstoresearchdataandoutputs,opensciencefostersknowledgespilloversandinnovationacrossdifferenteconomicsectors.It facilitates collaboration among research organisations and businesses, enhances firms’absorptivecapabilities(CohenandLevinthal,1990;Fabrizio,2009;SorensonandFleming,2004)andmakesforaswifterpathfromresearchtoinnovationbycuttingdelaysinthere-useofscientificresults,includingdatasets(OECD,2015;NESTA,2010).Also,theincreaseaccess to results by improving public access to research outcomes and data has beenhighlightedaspositive impactofopenscience towardsbusiness,which fostersspilloversfrom science and research (OECD, 2015). As an example, the use of data fromPubMedCentral, theUSNational InstitutesofHealth's repository, shows that 25%of theunique daily users are fromuniversities,while 17%are from companies and 40% fromindividual citizens (UNESCO, 2012). A recent study on R&D-intensive SMEs in Denmarkshowsthat48%ofthefirmsconsiderresearchresultsessentialtotheirbusinesses,andover65%reportbarriersinaccessingresearchoutcomes(Houghtonetal.,2011).Ontheotherhand,otherscholarssuggestthattheforcesofsharingscientificknowledgeandprotection of commercial interests run in opposition. The commercialisation of scientificoutputsusuallyrequiresasignificantinvestmentthatcompaniesarewillingtobeariftheycanprotecttheinnovationfromimitationorunfaircompetition.Somescholarshighlightthe
42
potentialrisksforfirmsengaginginopenknowledgesystems,withrisksarisingfromtheopposed institutionalrationalesof theworldsofscienceandtechnology(Jong&Slavova,2014). For instance, Perkman & Schildt (2015) show that participation in Open Datapartnershipswithuniversitiesmayjeopardiseacompany’sattemptstocapturevaluefromresearch.
4.2.1 CrossanalysisofcasesThediversecasestudiescarriedoutintheframeworkoftheOpenScienceMonitorprovideanunderstandingoftheeffectsofopensciencetrendsoninnovation,exploresthedriversofsucheffectsandinparticularassessunderwhichconditionsopenscientificpracticesarecompatiblewiththemarketlogic.Wediscussthethreeoftheminthefollowingsections.First,regardingthemajoreffectsthattheanalysisofthethirtycasesstudiesreflectarethefollowing.Wedescribe theeffectsbyemploying theexamplesof the cases that score thehighestintermsofindustryimpact(i.e.WhiteRabbit;ComparativeanalysisofOpensourcehardwarelicences;OpenTargets;PistoiaAlliance;Re-useofpublicdata;Openmetadata;andGOFAIR).However,allcaseshaveinformedthepresentassessment,thusweinvitereaderstoreadthemforfurtherinformation.1. Re-useofscientificoutputsbyfirmsinR&DprocessesOpeningupscientificdata(openscientificdata),thedesignsofscientificexperimentaltoolsthatlaterfindapplicationsindifferentindustrysettings(opensourcehardware),codethatcanbere-usedindataanalyticsprocesses(opencode)andmakingavailabletheprocessesthroughwhichscientistsreachconclusions(opennotebooks),amongstothers,showinthecasespositiveknowledgespilloverstowardsbusinesses.Forexample,inopenscientificdata,theOpenTargetscasereflectedthatnotonlypartnerswithinopentargetsconsortiumbenefittedofthedatapublishedinopentargetsplatformforbetteridentifyingandvalidatingtargets,butalsoexternalstakeholders(seeTwouse-casesofRe-useofopendata)takeadvantagefromtheopendata.OpenTargetsreceive1000visitsaverageperweek;butalso,weseeanindustryemergingofintermediateplayerssupportingpharmaceutical, biotech and companies in biomedical research to extract value byintegratingopenscientificdataintheirworkflowsandinternalinformationsystems.PistoiaAlliancecasealsoreflectedthebenefitsfromopenscientificdataandtheneedforindustrytoimplementFAIR-ificationofdataandadoptdatastandardssothatdatainteroperabilityacrosssystemsisnotabarriertograspsuchbenefits.Additionally,thecaseofWhiteRabbitshowforopensourcehardware,thepositiveeffectsofopeningupdesignsandschematicsaboutscientifichardware.WhiteRabbit,whichwasfirstconceived as a solution for the synchronisation of CERN’s distributed network, findapplicationslateroninawiderangeofscientificbutalsoindustrysettingssuchasfinancialservices(Frankfurtstockexchange),telecommunicationoperators(e.g.Vodafone)butalsoSmartgrid,air-trafficcontrolorautomationvehicles.Ifthesponsororganization,CERNinsuchcase,wouldhavechosenthealternativesituation,thatistooutsourcethedevelopmentofWhiteRabbittechnologytoavendorandprotectthedesignwithIPinsteadofdevelopingasanopensourcehardware,alltheotherscientificinfrastructuresthatwereinneedandre-usedsuchdesign,wouldhavehadtoincurthesameinvestmentfordevelopingsomething
43
that may be not as good as this one that leverage collective skills and expertise. Thecomparativeanalysisofopensourcehardwarelicencesshedslightinhowbusinessesmaybenefit from open source hardware licences by setting governance rules that allowproprietarydevelopmentsandimprovementstoemerge.2. AccelerationofindustrialR&Dprocesses:EfficiencyandEfficacygainsThegrowingcomplexityinR&Dprocessesisincreasinglyrequiringcollaborativemethodsacross different stakeholders (including competitors) to try to leverage complementaryknowledgeandskillstoaccelerateR&Dprocesses.ThisispreciselywhatthecaseofOpenTargetsreflectedwherecompanieshavebeenabletogainefficiencyintheearlystagesofdrug discovery process by sharing and openly disclosing pre-competitive data andknowledge regarding targets.Theyhavealsobeen able,byemploying suchcollaborativeresources,toidentifybettertargetsandthustoimproveefficacy.Other cases such asGOFAIR seeks precisely to accelerateR&Dprocesses bymaking thescientificdataavailabletobere-usedbydifferentplayersstartingbythemselves.CasessuchasPistoiaAllianceorGA4Healthshowedustheimportanceofsettingupandwidelyagreeingonstandardsinordertomakepossiblesuchre-useofdataandknowledgewithinandacrossorganizations,whichturnsinefficienciesinR&Dprocessesforpharmaandbiotechindustry.3. NewproductsandservicesemergingaroundopenscienceOpen science can also lead to new products and services commercialised by a set ofcompanies.TheexampleoftwousecasesofcompaniesemployingpublicscientificdatatobuildsemanticplatformsandanalyticsthattheyselltopharmaceuticalindustryandotherplayersinR&Dbiomedicalresearch;consultanciesthatselltheirservicestootherplayersintheecosystemtoworkaroundplatformsofpublicdatasuchasopentargetswhichhelptointegratepublicdataintheirworkflows.Beyonddata,thecaseofWhiteRabbitalsoreflectedhowcompanieswereabletore-useandsellWhiteRabbitswitchesandnodesandtheirservicestodifferentorganizationsinmultipleindustrialsettings.CurrentlyfourvendorsofferWhiteRabbittechnology,accordingtotheinformationintheopensourcehardwarerepository.Beyondthecasesselected,therearedifferent examples that showhow open science is generating an industrywith differentproductsandservicesaroundpublicresourcestoacceleratere-useand integratewhat isavailableininternalprocesses.4. IncreasequalityofR&DprocessesThecasessuchasYODAorREANAwhohavereproducibilityas theirraisond’êtremovedR&Dprocessestowardsincreasequalityandoverallreliability.REANAwhostartedasCERNprojectearly findanapplication inthe lifesciencesdomainandYODAprovidesgoodusecasesabouthowthecompleteandfullavailabilityofclinical trialshelpedpharmaceuticalcompanies to scientifically crowd source a better understanding of their results. OpenTargetsalsodescribeshowcollaborativeefforts inpre-competitive research improve thequalityoftheearlystagesofdrugdiscoveryprocess;andWhiteRabbit,clearlyreflectedhowtheopensourceapproachallowedtopooltogetherdifferenttypesofexpertiseandtocrowdsourcetheidentificationofbugsinthetechnologywhichleadtobetterqualityofWRoverall.
44
5. NewcooperativestrategiesinindustryR&DprocessesThe cases in themonitor describe emerging cooperative strategies and new governancemodels that are being built around open scientific outputs. Some examples include: thecreativegovernancemodelsputinplaceinYODAtogoverntheaccessandcontroltowardsdatabetweencompanieswhoaredataholdersandthescientificcommunity;opentargetsalsosetupa setof governanceboardsand rules toallowboth thedisclosureof relevantinformation to cooperate in a pre-competitive space while allowing protecting relevantassetsfromcompetition;theevolutionofthelicencesinWhiteRabbitwhichgovernedthecollaborationacrosscontributorswasalso insightful inseeinghowCERNsetupa licencewhichgothroughasetofdebatesaroundthecommunitywhofinallyagreeuponsomebasicrulesofthegame.Second, going through the thematic analysis carried out across the thirty cases and thecodingexerciseimplementedbytheOpenScienceMonitorresearchteam,weuncoverthedriversofthecasesthatscorethehighestintermsofindustryimpact.Notsurprisinglywesee that the caseswith high industry impact also score the highest in terms of supplierapproach.Usercentricityapproachappearsrelevant,whichislogicwhenthinkingaboutthere-useofdataandknowledge,whichmeanstakingseriouslyusers’workflowssothatthebenefitsofpublicdisclosurearegrasped.Finally,allthosecasesreceivedthesupportfrominstitutions,communityandfundersFigure33:Topfivedriversfoundincasestudieswithhighindustryimpact
4.2.2 QualitativeinsightsWhile companies increasingly seek to cooperate with other organisations outside firmboundaries to tap into ideas for new products and services (e.g. Chesbrough, 2003),mechanisms that motivate innovators to reveal the processes and the outcome of theirinvestmentsopenlyarenotevident.Theresultsofthecasestudiesarecoherentwithwhatpreviousliteraturedescribed.Opensciencebenefits innovationbyenablingaswifterpath fromscience tonewproductsandservicesinthemarket.However,asthecasestudiesshow,thiscanonlyhappenifdifferent
45
mechanismsareputinplaceinordertomakecompatiblebothgoalsofwidelysharingthescientific outputs while letting companies protect their private investments in the R&Dprocessthattheyarebeingpartof.Thosemechanismsvarydependingontheopensciencetrend:OpensourcehardwareTheexampleofWRprovidesrelevantinsightsonhowitispossibletogoonestepfurtherfromthemodelofopensourcesoftwaretoother,morecapital-intensivetechnologiesthatrequire different incentives, dynamics, and conditions, combining open revelation andtechnologycommercialisation.Theopensourceapproachpossessesan“essentialtension”about appropriating the returns from an innovation versus gaining adoption of thatinnovation(West,2003).FromtheWhiteRabbitcase,welearnedthattheinitiativewouldnoteven“takeoff”unlesstherightincentiveswereinplacesothatsuppliersandotherstakeholderswouldcontributetheirknowledgetoanopensourceschemewherebothR&Dprocessesandoutcomeswereopentoeverybody.WRidentifiedthefollowingmechanismsthatneededtobeputinplaceinordertostimulateprivateR&Dinanopensourcehardwarescheme,thatbenefitedavastnumberofscientificinfrastructures (higher than 30 research organisations usingWhiteRabbit), but also thetechnologyfoundlateracommercialapplicationinthemarket(e.g.inthefinancialsector,telecommunications,smartgrid,IoT,airtrafficcontroletc.):Fundamentalmechanismsidentifiedinthecaseinclude:
• New legal framework: CERN Knowledge Transfer Group along with its engineersprovidedanewlegalframework,theOpenHardwareLicence,inspiredbythe“TucsonAmateurPacketRadio”licence(TAPR5),tofosterknowledgesharingamongthediverseorganisations(itsfirstversionwasreleasedonMarch2011).
• Compromisebetweenproprietaryandopenhardware:Thelicencemadecompatible,afteraspecialrequestbymultiplefirmsandagreementoftheorganisationsbeingpartoftheWRdevelopment,thatcontributorswerenotobligedtodiscloseimprovementstothetechnology.UndertheOpenHardwarelicence,organisationswerenotobligedtonotifythechangestoupstreamlicensors. Thus,whilecontributingto theopensourcecoretechnologiesofWR,firmswereabletodevelopinparalleltheirproprietaryhardwarebasedonWRtechnology.
• Standardisation:ItwascrucialforstimulatingbothR&DrevelationbycompaniestakingpartinWRdevelopmenttoprovidesomestabilityaroundthetechnology.Goingthroughastandardisationprocesswasenvisagedtoadditionallyincreasethestability,viability,andcredibilityofthetechnologybysolicitingthefeedbackofexpertstofine-tunethetechnologyfurther.FirmswerewillingtoputprivateinvestmentandhelpdevelopingWR if they could anticipate some stability over the technology, meaning that thetechnicalspecificationswouldnotchangeovertime.ThisiswhyCERN,incollaboration
5 Description of TAPR in Open Hardware Repository: https://www.oshwa.org/research/brief-history-of-open-source-hardware-organizations-and-definitions/
46
withotherresearchinfrastructuresandcompanies(partofWRdevelopment),decidedtostartastandardisationprocessofWRtechnologies.
• Arbitrageandorchestrationofexchangesbyanacademicpartner:CERNnotonlyhadtoengagenewcompaniescapableofprovidingWRtechnologyproactivelybutalsoithadtogoverntheexchanges,beinganarbiterofdifferenceswhenneeded(Serrano,2016).ThedifferentorganisationsinvolvedinWRdevelopmentacceptedthearbitrageofCERNinthedifferentexchangesanditsleadership.
• High engagement and close interactionswithacademic partners in order to providelearning and reputational gainsby companies developing the hardware.Firmswereenvisaging an upgrade of their technical skills by closely collaborating with highlyqualifiedengineersatCERN.Additionally, organisationsengaged in thedevelopmentexpectedtobedifferentiatedfromfuturemarketcompetitorsbybeingconsideredatier-1partnerofCERN.Thebrandingandreputationofsuchcollaborationwereexpectedtosignalthequalityoftheorganisationwhileextendingthecompanies’capacitytoattractclients.
OpenscientificdataOpen Targets, Yoda and Pistoia Alliance cases showed that open research data benefitsindustryandaccelerates innovationprocessesifdifferentmechanismsareput inplace inordertoprovidetherightincentivesandconditionsforcompaniestobothsharedata,andto re-use third party data. From the three cases the following mainmechanisms wereidentified:• Agreement among industry players ofwhat data and resources are considered pre-
competitiveandwhatiscompetitive.Pre-competitivedataissusceptibletobeingshared,whilecompetitiveinformationandknowledgearegoingtobeheldsecretsimultaneously.As an example, the collaboration within Open Targets required a clear delineationbetweenwhatwasagreedtobesharedamongstorganisationsandwhatremainedclosedfortheparticipants.The‘discontinuity’inthelinegoingfromcooperationtocompetitionneeds to be demarcated to generate a comfortable environment for corporations tocollaborate and share data to accelerate the early stages of drug discovery whilepreservingtheirassetstocompeteamongsteachotherinlaterstagesofthedevelopmentprocess.FirmsinOpenTargetsagreedtocollaborateanddisclosedataandknowledgerelatedtophases1(targetidentification)andphase2(targetvalidation)tosucceedinsystematicallyfindthebesttargetstosafelyandeffectivelytreatdisease.However,firmsdonotrevealtheknowledgeandproprietarydatathattheyaregoingtousetoidentifythemultiplemoleculesactiveagainstthepotentialtargets,norotherdatausefulinlaterstagesofthedrugdevelopmentprocess(i.e.phase3-leadoptimisation-andonwards).
• Thefundamentalneedfordatastandardstoeffectivelyshareresearchdata.AsGOFAIR,PistoiaAlliance,GA4Health,RDAandothercasesshow,withoutcommondatastandardswidelyadoptedbyacademicandindustryorganisations,thebenefitsofopenresearchdatacannotbefullygraspedbyfirms.Inordertoactuallyfacilitatedataexchanges,andthustaponthebenefitsofartificialintelligence,machinelearningandanycomputationalapproach relying upon the aggregation of high volumes of quality data, the industry
47
needstoagreeandwidelyadoptstandardscollectively.PistoiaAlliancecaseshowedthatstandardsarefundamentalinbiomedicalresearch(e.g.UnifiedDataModel)butarealsoresourceintensivebecausetheyrequirehoursofdistributedteamsworkingtogethertoagree on common tech specifications and solutions. Once the standard is out, if itsuccessfully implemented by a wide range of players, it can dramatically reduceinefficiencies and accelerate information sharing within and outside organisationalboundaries.
• Arbitragebyanacademicpartner.Forinstance,inthecaseofYoda,theinitiativeledby
the Centre for Outcomes Research and Evaluation (CORE) from Yale University ofMedicine, developed amodel tomake data available to scientists, which ismediatedthroughYODAteam.YODAfacilitatesaccesstoclinicaltrialdatamadeavailablebythirdpartyDataHolderstopromoteresearchthatsupportsscientificendeavoursandpublichealth.YODAdoesnot supportnor facilitateaccess todata for commercialpurposes,litigation, or any goal that is not purely scientific. Themodelworks as follows: DataholdersgiveaccesstotheirclinicalresearchdatatoYODA.TheYODAteamfollowsastrictprotocoltograntaccesstothesedatatoresearchers.AnotherexamplewasprovidedbyOpen Targets, which showed a model to pool together distributed capabilities andresources from the different organisations and companies through matchmakingexercises to match people from distributed teams working on a joint research idea;merging companies individual investments devoted to early drugdiscovery in a jointresearch agenda; and other additional approaches that helped orchestrate thedistributed resources (data, capabilities and expertise) with a prominent role of theacademicpartnersinthegovernanceofexchanges.
• Highengagementbyacademicpartnerstoincreaselearninggainsandpossibilitiesforresearchdatare-use.BothPistoiaAllianceandOpenTargetscasesdescribedhowface-to-faceexchangesandcloseinteractionsarestillcrucialandneedtobecombinedwithaninteractiveplatformthatfacilitatesdataandresourcesexchanges(e.g.IP3inthecaseofPistoiaAllianceandOpenTargetsPlatform).Technologicalplatformapproachesneedtobecombinedwithclosecooperationamongtheorganisationsthatgeneratethedataandthosewillingtore-usethedata.AsthecaseofOpenTargetsdescribed,sharingscientificdatainthepublicdomainisanecessarybutinsufficient requirement for being able to re-use such data for drug developmentpurposes. Companies taking part of Open Targets consortium agree to privateinvestments–bothcashandin-kind–inordertobeclosetothescientiststhatgeneratethedatatobeabletounderstandthedata,howitwasgenerated,andhowtointerpretit.
• Time lagand ‘premium’ features indataplatforms forcompaniessharingtheresearchdata.ThecaseofOpenTargetsdescribesamodelwherecompaniesgeneratingthedataagree to publicly share all scientific data produced in the framework ofopen targetscollaboration.However,whilepartners in theconsortiumhaveaccess to theresearchdatafromminuteone,thedataisopentothepublicthroughtheopentarget'splatformafterpublication.ThistimedifferencegivescompetitiveadvantagetowardsfirmsinsideOpen Targets consortium. On the other side, the functionality of the Open Targets
48
platformbythepublicandpartnersdiffer.Forinstance,partnerswithintheconsortiumcanaggregateprivatedataintheplatform(e.g.,compoundlibraries)toacceleratefurtherstagesofdrugdiscovery.
GeneralpatternsacrosstrendsTheanalysisoftheresultsofthecasestudiesimplementedagreeswiththeliteraturethatclaimsthatopensciencebenefitsinnovationbyenablingaswifterpathfromsciencetonewproductsandservicesinthemarket.However,thecasestudiesalsoshow,thatthiseffectcanonly happen if different mechanisms are put in place. Table 6 below summarises thosemechanismsalsodescribedabovebasicallyforthetwoopensciencetrendsthatreflectedhigherindustryimpact:openscientificdataandopensciencehardware.Thosetwotrendswhichconcentratedmostofthecasesinthisrespect.Table6:Analyticalsummaryofmechanismsenablingthepositiveimpactofopensciencetoindustry
Openscientifichardware OpenScientificData Governance Newlegalframework:See
comparativeanalysisfordifferentgovernancemodelsinOSH.Thereisaneedforlicencesthatenablethenon-disclosureofimprovementsorthecompromisebetweenproprietaryandopenhardware.
Thereisaneedforaprioragreementbycompaniesandacademicpartnersaboutwhatisgoingtobedisclosed(pre-competitive)andwhatfallsunderthecompetitivearena:whatdata/resourcesarenotsubjecttoIP;whataresubjecttoIP.
Standards Standardisationprocesstoincreasethestability,viability,andcredibilityofthetechnologydevelopedopenlytoallowperipheraldevelopmentstoemergeandthusnewproprietarytobecommercializedaroundtheopentechnology.
Datastandardsforindustryandacademicpartnerstoenabledatasharingandre-usebutalsointegrationwithininformationsystemsinsideorganizations.
Arbitragebynon-industryfirm
Arbitrageandorchestrationofexchangesbytheacademicpartner,asatrustedpartytogoverntheexchangesamongststakeholdersandguarantee,insomecases,re-useforthatdonotdamagecompetitiveadvantage.
Arbitrage by the academicpartnerbetweendataholdersanddata(re)usersputting inplace protocols that providetrust to commercialpartnersandthird-parties.
Learninggainsthroughinteraction
Highengagementandcloseinteractionsbytheacademicpartnerforcompaniestoextractlearninggainsfromtheinteraction.
Highengagementandcloseinteractionswiththeacademicpartnersforlearninggainsandtoincreasepossibilitiesofresearchdatare-use.
49
Openscientifichardware OpenScientificData Competitiveadvantagetowardscontributors
Favouringindustrialcontributorstowardsfree-ridersthroughbrandingandreputationalgains.
Timelagandpremiumservicesforcontributors.
Finally,thestudyteaminvitesreaderstogothroughthedifferentcasestudiesinordertogetmore insights about the different mechanisms, boundary conditions and contextualcharacteristicsthatmadethecasestudiesbeselectedinthefirstplaceasinsightfulexamplesofsuccessfulopenscienceimplementations.
4.3 SocietalimpactOpensciencecertainlyhasavisibleimpactonsocietyasanindirecteffectofthescientificandindustrialimpactsanalysedabove.Enhanced,fasterscientificdiscovery–suchasthoseidentifiedthroughOpenTargets–benefitallpartsofsociety.Earlydetectionofsideeffects–providedbyYODA–lowersthecostsofdevelopmentandacceleratesthedetectionofsideeffects.Secondly, industrial innovation is a determinant of growth and employment. Thedevelopmentofnewproducts,suchasWhiteRabbit,hascreatedanewmarketopportunity,andincreasedtheproductivityofthecompaniesadoptingtheseinnovations–fromfinancetotelecom.Besidethesewell-knownbenefitstowardssocietythroughscientificandindustrialimpact,thereareother relevantaspects tobe considered.Firstly, openscience cancontribute togreaterpublicunderstandingofscience.Opensciencehasthepotentialtoenhancepublicengagementandtheunderstandingofscienceprinciplesandpracticebyraisingawareness,pro-activeparticipationandcitizens’directcontributiontoresearch(Boultonetal.,2012;Kowalczyk & Shankar, 2010). The openness of the scientific process fosters new actors’access to the research process (Nowotny, 2001). The increasing importance of citizenscienceisaforcefordisruptivechange,engagingamateursandthegeneralpublicinscience.Thoseinclusiveandparticipatoryapproachestoboosthumancapacityandcapabilityfromprofessionals, amateurs, volunteers and citizens to assist in collecting, curating andpreservingthegrowingscientificrecord(Lyon,2009).Citizen science is also a method for formal and informal science education and publicunderstanding of science, which should not be underestimated (Christian et al., n/a.).Greatercitizenengagementcanleadtogreaterparticipationinscientificexperimentsanddatacollection(OECD,2015;Boultonetal.,2012;Whyte&Pryor,2010).Makingresearchdatapubliclyavailablecanadvancepeople’sunderstandingofscienceandcitizenscienceinitiatives (Kowalczyk & Shankar, 2010). TheGlobalMosquito Alert Project has broughttogetherkeycitizensciencemonitoringinitiativestoworktogethertohelpunderstandthegeographicspreadofmosquitoswhoarecapableofcarryingdisease,inturnhelpinghealthprofessionals and citizens to manage any potential risks. 6 Zooniverse, a world-leading 6JonSwitters,DavidOsimo,CitizenScienceintheSurveillanceandMonitoringofMosquito-BorneDiseases,OpenScienceMonitorCaseStudy(Brussels:EuropeanCommission,2019)
50
platform foronline crowdsourced research,hasmore than100activeprojects (in2019)wherevolunteersparticipateincrowdsourcedscientificresearch,classifyinglargedatasetsacross multiple knowledge domains (such astronomy, ecology, cell biology, humanities,climatescience,etc.,withsignificantoverlapbetweenareas).TheZooniverseresearchbringsforwardnewdiscoveries,datasetsusefultothewiderresearchcommunity,andpublications.Hypothes.isdevelopedatooltoenableanyonetoannotateanywhere,andopenedupandstandardisedtheannotation’stechnologiesandpractices.Itaimedtomakeiteasyforusersto leave annotations across the entireweb. And, the open-source nature of the platformallowsanyone(developers,publishers,academicinstitutions,researchers,andindividuals)who use it to create their own annotation reader orwriter. The nature of the tool alsofacilitates the collectionof annotations ina structuredway, allowing tobuild theoverallpictureoftheinformationknowledge.On theotherhand, increasedresearchdata sharing canconstitutea challenge to societalvalues, mainly by increasing the risk for infringement of data protection. There is afundamental tension between the scientific value of data built through correlation ofdisparatedatasetsvariablesandtheneedforpreservinganonymity.Severalstudieshaveshownthatpersonaldatasecuritycannotbesafeguardedbyanonymisation.Companiesthatrelease data openly for research purposes have encountered problems related to the re-identifiability of subjects. Datasets that contain information on individuals supportinferencesregardingtheprobabilityofotherinformationaboutpeople(Denning&Schlorer,1980; Lyon, 2009). Conversely, the recent emphasis around the implementation of theGeneral Data Protection Regulation is forcing scientists and research institutes to adoptmore stringent data management practices that can be consistent with increased datasharing:forinstance,bylimitingstorageonowncomputersandrequiringscientiststousededicatedlaw-compliantrepositories(Stark,2019).Thiscouldturnouttoforcetheadoptionofappropriatedatamanagementstrategies(rathertheinformalonesillustratedinsection2.2)thatcouldultimatelyleadtowiderreusabilityofdata.
4.3.1 CrossanalysisofcasesWhenperformingacross-analysisofthecasestudies,fewofthemshowhighimpactonsociety(seeFigure1).Figure34:Impactonsocietyoftheanalysedcasestudies
51
Accordingtotheassessmentofthecases,thereareonlyfourcasestudies(ofthetotalof24analysed)withhighimpactonsociety:OpenTargets,Yoda,GlobalBiodiversityInformationFacility(GBIF)andZooniverse.Forother10cases,theyshowamediumlevelofimpactonsociety:WhiteRabbit,PistoiaAlliance,REANA,OSHlicences,Aspredicted,Twousecases,Global Alliance, GOFAIR, Hypothes.is and Mosquito alert. The rest of 10 remaining casestudiesshowaratherlowimpactonsociety.From research lifecycle side, the caseswith high andmedium impact are found in datageneration(OpenTargets,Mosquitoalert),dataintegration(OpenTargets,YODA,Twousecases),datapublication(YODA),replication(YODA,REANA),discovery(GlobalBiodiversityInformation Facility, Hypothes.is), prototyping and developing tools (White rabbit, OpenHardwarelicencesandHypothes.is),datadescriptionstorageandmanagement(GOFAIR),pre-registration(Aspredicted)anddatastandards(PistoiaAllianceandGlobalAlliance).Figure35:Casestudieswithhighandmediumimpactonsociety,byresearchphases
Itisimportanttomentionthatthecasestudieswithhighandmediumimpactonsocietyarenotrestrictedtoonlyoneopensciencecategory(opendata,openaccess,opencollaborationorcitizenscience).Fouroutofthe14casestudieswithhighandmediumimpactonsocietyareclassifiedinmorethanoneopensciencecategory(seeTable8).Table7:Casestudieswithhighandmediumimpactonsociety,byopensciencecategory
OpensciencecategoryClassifiedinone
areaClassifiedinmorethanonearea*
Opencollaboration 2(14.3%) 5(27.8%)Opendata 7(50.0%) 9(50.0%)Openaccess - 1(5.6%)Citizenscience 1(7.1%) 3(16.7%)Totalcasestudies(considered) 14 18Note:*Thecasestudieswithhighandmediumimpactonsocietyclassifiedinmorethanoneopensciencecategoryarecountedmultipletimes.
52
Opendataismostfrequentcategorythecasestudiesareassociatedto,regardlessofsingleormultiplecoverage–halfofthecasestudiesincludetheopendataprinciple.Itisfollowedbyopencollaborationandcitizenscience.Whenitcomestoopenaccess,onlyonecasestudyincludesit,togetherwithopencollaboration(Aspredicted).
53
5 PolicyanalysisThissectionprovidestheintegratedpolicyanalysisofthestudy.ThefirstpartpresentstheoverviewprovidedbythedatacontainedintheSherpadatabase,aboutfundersandjournalspolicies on open access and open data (Sherpa-Juliet and Sherpa-Romeo database). Thesecondpartpresentstheindepthqualitativecrossanalysisofcasestudies.
5.1 StatisticaloverviewThestatisticaloverviewisdividedinopenaccesstopublicationsandopendata.Itincludesdataonfundersandjournalpolicies.
5.1.1 OpenaccesspoliciesOne of themain factors affecting open access to publication is themandates from theresearchfunders.Datashowsthatopenaccesspublicationisenforcedbyresearchfunderstoamuchlesserextentthanopenaccessarchiving-outof145institutionsidentified,33%hasnoopenaccesspolicyinplace,whereas35%encouragesitandonly31%requiresit.Figure36:Funder’spoliciesonopenacesspublishingbycountry
Concerningnumberofresearch fundersregardingmandateonopenaccessarchivingthevastmajority (71%) requiresopenaccessarchivingand further12%encourages it, only17%ofinstitutions(allfromUK)donothaveanypolicyregardingopenaccessarchiving.
54
Figure37:Funders’policiesonopenaccessarchiving
Policiesbyresearchjournalsarealsoimportantinsettingtheboundariesforresearchers’choices.Thevastmajoritysupportssomesortofarchiving(82%):42%supportsarchivingofpre-printandpost-print,33%canarchivepost-printandfurther7%canarchivepre-print.Onlyin18%ofjournalsarchivingisnotformallysupported.US research journals stand for the majority of those journals where archiving is notsupported,followedbyUnitedKingdomandPortugal.Thesearealsothecountrieswiththelargest amount of the research journals.However, from all top five countries (India andGermanyastop4andtop5)onlyinPortugalmoreresearchjournalssupportpost-printonlyratherthanpost-printandpre-printtogether,thelatterbeingandoverallpredominanttrendinmostofthecountries.
55
Figure38:Archivingpolicesbytypeofmandate
5.1.2 OpendatapoliciesYet,concerningpoliciesofresearchfunders,morethanahalf (56.6%)donothaveanyopendatapolicyinrelationstodataarchivingandonly27.6%requiresit.
56
Figure39:Opendatapoliciesbytypeofmandate
Asinthecaseofopenaccess,thesestatisticsarelargelyinfluencedbyUnitedKingdom,acountrywiththehighestnumberofresearchfundersmappedintheSherpa-Julietdatabase.Almost76%outof64researchfundinginstitutionsintheUKdoesnothaveopendatapolicy.Furthermore,inmanyothercountries,researchfundinginstitutionseitherdonothaveopendatapolicy(e.g.Spain,China,Luxembourg)ortherearemoreinstitutionsthatdonothavean open data policy than those that encourages it or requires it (e.g. France, Belgium,Denmark,Italy,Sweden).Figure40:Opendatapoliciesbycountry
57
Regardingjournalsanditsdatasharingmodality,accordingtothelatestavailabledata,only21%ofjournalsrequiresdatasharingand32%donotaddressesdatasharingatall.Thisindicatorisaone-offanalysiscarriedoutin2017.Figure41:Journalsbydatasharingmodality
5.1.3 OtherpoliciesThereislittledataonjournals’policiesconcerningopencode.Datafrom2013showthatjournals´opencodepolicyisalmostnon-existent.Outof170journalsanalysed79%ofthemdidnotaddresscodesharing.Infact,codessharingwasexplicitlyrequiredonlyincaseof12journals(seeFigurebelow).Figure42:Journalsbycodesharingmodality
58
5.2 Indepthcross-analysisofpolicycasesThiscross-analysispresentsthesystematicinvestigationofthenationalimplementationofopensciencepoliciesintheNetherlands(NL)andFinland(FI),andopenresearchpoliciesintheUnitedKingdom(UK).Firstly,thepurposeofthisreportistofacilitatediscussionsontheimplicationsand thepolicy recommendationsmade in theEuropeanopensciencepolicycontext.Secondly,thisstudysynthesisesthenationalpoliciesconductedbytheOpenScienceMonitor(OSM)thusfar,byanalysingthemainbottlenecksandthepolicymeasuresgatheredacrossthethreecountries.Finally,theresultofthisevaluationisthefirststepinabroaderanalytical task of monitoring open science development in Europe –what open sciencestrategy(atEuropeanlevel)isneeded,why,andtowhichextent?Fromthere,thisstudywillconcludewithpotentialinsightsthatcanguidefutureanalysisonopenscience.
5.2.1 ObjectivesAcrossthenationalpolicycasestudiesintheNL,FIandUK,twokeydriversareguidingthenarrativeofthepolicies:(1)themoralimperativethatpubliclyfundedresearchgeneratesknowledge that belongs to everyone, thusmust be shared to and accessed by the publicimmediately, and (2) the economic gains of easily andwidely accessible research,whichgenerates productivity and competitiveness in their respective national research andinnovationsystems.Sinceallthreecountriesarekeentobeknownastheleadersinopenscienceandresearch,Table8presentsacomparableoverviewoftheirscientific,societalandindustrialdriverswithineachofthenationalpolicycontext.
Table8:Overviewofscientific,societalandindustrialdriversacrossNLandFIOpenScienceandUKOpenResearchPolicies
Driver NL FI UK Science Openscienceimproves
thetransparency,verifiability,efficiency,reproducibility,andsustainabilityofresearchprocesses
Opennesswillpromotetransparency,maketheresearchprocessmoreeffectiveandallowresearcherstodealwithcomplex,multidisciplinaryquestions
Openresearchimprovesefficiencyintheresearchprocess,transparency,accountability,andpublicengagementwithresearch
Industry Openscienceincreasesinnovativecapacityasindividualsbenefitmorereadilyfrompubliclyavailableinformationanduseitincombinationwiththeirownknowledgeandexperiencesto
Openscienceleadstosurprisingdiscoveriesandcreativeinsights-fosteringtheresearchsysteminFinlandtowardsbettercompetitivenessandhighertransparencyandinnovation
Openresearchcreatescloserlinkagesbetweenresearchandinnovation,benefitsproductsandservices,encouragesthecommercialisationofresearchand
59
Driver NL FI UK developnovelproductsandprocesses
stimulateseconomicgrowth
Society Everyoneoutsidethescientificcommunitymaybenefitfromopensciencebecausetheycanreadilyaccessandusescientificinformation
Thecirculationandpromotionofresearchdataoutsideacademiacontributestothegreatersocietalimpactbyincreasingscientificliteracyamongcitizens.Transparencyincreasesthecredibilityofscienceforthecitizens
Openresearchpromotespublicunderstandingofscience,evidence-basedpractices,andcitizen-scienceinitiatives
Source:OSMcasestudies
Similardriverswere identified inall threecases,althoughtheopensciencepolicieswereintroducedindifferentyearsforeachofthecountries.FortheNetherlands,itwas2016asledbytheDutchMinistryofEducation,CultureandScience,andforFinlanditwas2014,asinitiated by the FinnishMinistry of Education and Culture.Meanwhile in the UK, it wastermed open research and declared by the Minister of State for Universities, Science,ResearchandInnovationin2013.Themoralandeconomicdriversforopenscienceandopenresearchinthesethreecountrieswereexpressedclearly,whichhardlyleaveanyroomforobjection.
5.2.2 ProvisionsPublicpolicyplaysanimportantroleininfluencingresearcherbehaviour.Inrecentyears,severalinstitutionaladjustmentstothegeneralgovernanceframeworkofsciencehavetakenplace:
• Theintroductionofcompetitivefundingmechanisms,asopposedtobulkfundingintheEuropean higher education sector. Publications output and impact factors becamecommon performance indicators for quality-related research in higher educationinstitutions. This has increased competition and reinforced the importance ofassessmentmetricsinthecontextoffundingdistribution.
• Theincreasedpolicyemphasisoninnovation,toboostthecompetitivenessofEuropeanindustry. Industrial impact, characterised by collaboration with private companies,generationofpatentandintellectualpropertyrights(IPR),wereimplementedintothebroader European Commission policies, to better convert scientific research into
60
improvedgoods,servicesandprocessesforthemarketforthewelfareandwellbeingofsociety.
These twosciencepolicy trends certainly createa tensionwith the specificopensciencepolicygoalsofopennessandcollaboration, increasinglyadoptedby fundingagenciesandnationalinstitutions.Table9presentstheobservedtensionthroughtheevidencegatheredfromthenationalopensciencepoliciesofNLandFI,aswellasopenresearchpoliciesintheUK in three key features: open access, open data, and research evaluation and rewardsystems.Table9:ProvisionsaddressedintheNLandFIOpenScienceandUKOpenResearchPolicies
Feature NL FI UK Openaccess
Hybridjournalswillnolongerbepaidfor.ChallengetoturnGreenOAroutetoanacceptedmeansofpublishing.
Alackofcommonproceduresonopenpublishing.LowpublicationinOAJournalsandthecostsforOApublishingremainsexpensiveformostoftheevaluatedpublishers.
70%ofOApublishingisgoingtohybridjournals.TheaverageAPCforhybridpaymentbyUKuniversitieswereincreasedby19%,between2013and2016.
Opendata Alluniversitieshadtheirownonlinerepository,whichislinkedtoanationalportal–NARCIS.Althoughthereisaslightincreaseofopendatasetsovertheyears(exceptfor2011and2013),mostdatasetsremainedclosedorrestricted.TheNationalCoordinationPointResearchDataManagement(LCRDM)willconductastudytoestablishthequalitativeandquantitativeneedfordatastewardsandresearchsoftwareengineers,thecontentoftheirrole,thetrainingrequired,andtherewardingoftheirwork.
Stronglydata-drivenwithafocusonthedevelopmentofresearchdataarchitecture:managementofownership,metadata,andIPR.Theinitiativewasplanneduntil2017,withnofurtherstrategyorplandevelopedforthefollowingyears.
Patchyopendataprovisions:researchcouncilsareeither(1)silentonthematter,(2)providedirectsupportfortheworkofdatacentresandservices,or(3)relyonuniversityorthird-partyrepositoriestopreserveandprovideaccesstoresearchdata.
Researchevaluationandrewardsystems
Negativeevidenceforopensciencerewardinperformanceagreementsbetweentheministryanduniversities,andtenuretrackpolicybetweenuniversitiesandindividualresearchers.
Theevaluationframeworkforopennesswasdevelopedtoincentiviseresearchorganisationswithrespecttotheirpoliciesonandimplementationofopensciencepractices.
OApublications,resourcesandinfrastructurethatsupportopenresearchareconsideredinthequalityofresearchoutputsandresearchenvironmentofthe
61
Feature NL FI UK ResearchExcellenceFramework.
Source:OSMcasestudies
Finally,theGeneralDataProtectionRegulation(GDPR)wasenforcedbytheEUonthe25thofMay2018.WhiletheGDPRdoesstrengthendataprotectionandprivacyofEUpersons,italsocomplicatestheproprietaryaspectofdataasinstitutionsandindividualsarestilltryingtounderstandhowchangesinprivacylawswillimpacttheirwork.GDPRthuscreatesfurthertensionbetweenopendataandprivacyandsecurity,asillustratedinsection4.3.
5.2.3 LessonslearntTheUKisleadinginthenumberoftotalOApublications,followedbyNLandFI.GiventhattheUKhasthehighestnumberofresearchfundermandatesconcerningOApublication,nottomentionthattheResearchExcellenceFramework(REF)andresearchfundingdependsonit, it is unsurprising that the UK is leading in Green OA publications. Contrary to thepreferenceforGoldOAasannouncedbytheUKScienceMinister,DavidWilletsin2013,theGold route is still far frombeing a desirable and acceptablemeans of publishing forUKresearchers.IntheNetherlands,astrongfocusinGreenOAwasobservedingeneral,sincealluniversitieshadtheirownonlinerepository,whichmakesiteasierforresearcherstoself-archivetheirpublications.The current policies in NL, FI and UK demonstrate converging policies, with minordifferencesinthedriversonopenscience.Thefocusofallthreecountriesaremainlyonopenaccess publishing from the top-down, where funders and governing bodies are leadingprogressinopenscienceinsteadofbottom-up,researchcommunitiesledinitiatives.It isworthnotingthat the fundingbodiesofall threecountries in thepresentreport,aremembers of ‘cOAlition S’, an international consortium of research funders to accelerateimmediateOpenAccesspublishing.Clearly,theirinvolvementinPlanSdemonstratedthecountry’scommitmentinrealisingthemoralimperativeandeconomicbenefitsasoutlinedinthe‘Drivers’section.TheUnitedKingdomisseenasthestrongestinopenscienceandisinfact,dominatedbyimpact-orientedpractices thataresanctionedand imposedbyresearch funders.Thishasresultedinadivergenceinopenaccessambitionsandactualpublicationpractices,withariseinpublicationsinhybridjournals,andanimmediatedecreaseingoldOApublications.Thecostofopenscienceopeningcontinuestotakecentrestageintheopensciencepolicyarena,whilethekeyissuesofopendata,researchevaluationandrewardsystemarehardlyaddressedbypolicymakersinallthreecountries.
62
6 PolicyconclusionsThe largeresearcheffort leadstomanypossiblepolicyreflections, indifferentareas.Forsimplicity,wedividetheminthoseconcerningopensciencepolicy,andthoseconcerningfuturemonitoringefforts.
1.1. Conclusions on the future of open science 1. Datashowsclearlythatopenscienceistheretostay.
All trends clearly indicate progress: from open access to publications (considering theimpactofembargoperiodonstatistics)toopendata(withlimitedprogress)toopenAPI,openhardwareandcitizenscience.Everyindicatorandeverytrendshowprogress,evenifslow.
2. Openscienceisawide-rangingsetoftrends:itismorethanopenaccesstopublicationandopenresearchdata.
Thisperhapsobviousstatementneedstoberepeated.Inthepolicydebate,itappearsthatopenaccessattractsalltheattentionandthe“expenditure”intermsofpolicycapital.Afterthat,thetopicofopenresearchdatainpolicytermsisemergingstrongly,ledmostlybythepolicydebateaboutthedataeconomy.Yetalltheothertrendsaregrowingstronglydespitethelackofpolicyattention.AtatimewhenMicrosoftacquiresGithub,thelargestrepositoryofopensourcesoftware,forabout7billioneuros,fewjournals,fundersandregulatorshaveyet introduced policies concerning open scientific software, and the developments aremostlybottomup.
3. Theprogressdetectedbytheindicatorsdoesnotjustifyanycomplacency.Data shows a slowprogress. For instance, data sharing among researchers show limitedprogressinthetwoyearsmeasured.Progressonopenaccesstopublicationsinunequal,withadeclineingreenopenaccessthatwecurrentlyattributetoembargoes–butthatneedstobemonitoredinthefuture.Overall,itisclearthatopenscienceisnotthedefaultoption–butratherchosenbyaminorityofscientistsbothin termsofopenaccessandopenresearchdata.Inotherwords,opensciencerequirescontinuouspolicyattention.
4. The policy attention needs to be focussed in particular on getting the incentivesystemsright.
Open science requires changes by all stakeholders involved and can’t be achieved onlythroughregulation.ItisclearthatweseeaproliferationofpolicyinitiativesatnationalandEuropean level,butwhenwelookatbehaviouralchangesbyscientists the importanceoffunders’policies is limited– it ismuchmore important toact through the communityofpeers. Ultimately, open science requires a substantial effort of governance, a continuoustiltingoftheplayingfieldtoensureoptimalresults.Moreconcretely,initiativessuchastheEuropeanOpenScienceCloudrequireacontinuouseffort ingovernanceevenAFTERthelaunch.Thelaunchisthebeginning,nottheendofthepolicyprocess.Allthesuccessful,largescaleopenscienceinitiativesdescribedinthecasestudiesshowthattheonlywaytoscaleup is through continuous adaption and experimentation of new modalities to involveddifferentstakeholders.Thereasonforthisisclear:
63
5. Open science does not appear as an ideological radical choice, but as a rationaldecisionbasedontheinherentadvantages.
Mostlarge-scaleprojectshavebalancedclosedandopenmethods,dependingonthespecificchallengesandopportunities.Somemethodsworkunderspecificconditionsanddependingonthestakeholders’interests.Thereisnoonesizefitsallapproachtoopenscience.Itisclearthatbothscientistsandcompaniesareusingdifferentdegreesofopennessdependingonthecircumstances. Researchers share data mostly within their research group or projectspartners, rather than openly. Datasets in repositories have different levels of openness.Research centres develop new licences to accommodate for the needs of differentstakeholders.
6. Scientificprogressistheultimategoal–andtheimpactisvisible.Whatisclearfromthecasesanalysedisthattheachievementofscientificprogressisthemain priority, and the openness varies in order to maximize that goal. Equally in thecorporatesector,gettingnewproductsfastertomarketisaprioritywithrespecttoopennessinitself.Inotherwords,whenopensciencescales,itisameansnotagoal.Someofthehighestprofilescientificandcommercialendeavourhaveincorporatedelementsofopenscience.Andit’snotonlybasicsciencewhereopenpracticesareconsolidated.Opensciencepracticesarenowusedincommercialendeavours,suchasthedevelopmentofhighprecisiontimemeasurementorthediscoveryofnewpharmaceuticalproducts.Despitetheattentiononopenaccesstopublications,itisinopenresearchdataandopencollaborationthatweseegreaterindustrialimpact.Openpracticesandcommercialgoalscangohandinhand,butonlyunderspecificconditions.Thereisatrade-off,notanincompatibility.
7. Opensciencepolicyshouldlookbeyondopenscience.Whatisclearisthatopenscienceispartofawidertrendtowardsdatadrivenandlarge-scalecollaboration, for instance through the adoption of APIs, the diffusion of data portabilitypractices,servitisationandmore.Policiessuchasthedataeconomy,thedigitalagendaandindustrialpolicyshouldbealignedwiththeopenscienceagenda.Ultimately,opensciencepolicyshouldnotbedrivenbyideologybutbyconcreteneeds.Onlyinthiswaycantheparticipationofallrelevantstakeholdersbeensured.Toachievethis,thecommunitybuildingworkthattheEChasimplementedsofarisanecessaryfoundation,butitisonlythebeginning.Opensciencepracticesrequiresystemicchangethattakesseveralyearstotakeplace.Andgoodmetricsareafundamentalpartofthiseffort.
1.2. Conclusions on the future of the Open Science Monitor 1. The existence of an Open Science Monitor is necessary to successful policy
implementationThetransitiontoopenscienceisacomplex,multi-stakeholderorganictransition,notatopdown decision-making process. In such a context, monitoring is even more important,becauseithelpsallstakeholdersbuildacommonassessmentofthesituationandtoaligneveryone’sefforts.Obviously,amonitordoesnotpersegenerateconsensusandcommonaction,butitsabsencemakesachievingthosegoalsimpossible.Nationalcasestudiessuchas
64
theNetherlandsshowedthattheabsenceofmonitoringmechanismsforspecificpolicyareassuchasopendatawasamajorbarriertoprogress.Monitoringopenscienceisnotasimplestraightforwardtask:itrequiresclearmethodologicalchoicesandrobustmethods.Assuch,itcannotrelypurelyonthirdpartymetrics,whichoftendifferalotintermsofcoverageanddefinitions, including between member states. Moreover, the sensitivity of the issue, asshown by the conflict about the existing Open Science Monitor, requires strong centralknowledgepresidiumtojustifyanddefendmethodologicaldecisions.Putsimply,EuropeanopensciencepolicyrequiresaEuropeanopensciencemonitor.
2. The next Open Science Monitor should reduce the robustness gap between openaccessandothertrends
While openscience is recognized as amultifaceted trends, some trends aremuch bettercoveredthanotherswhenitcomestoindicators.Therearemajordifferencesbetweentheavailability and robustness of data on open access to publications, open data and opencollaboration.Dataaboutopenaccesstopublicationareeasilyavailable,althoughthereareconstant discussions about the modalities of measurement. Indicators about open datasharinghavebeenprovidedthroughadhocsurvey,inviewofitsrelevance.Butfortheothertrendsweareatlossofreliableindicatorandwecanonlyuseproxies,mainlyrelatedtotheadoptionofexistingservicesorexistingcollectionsthatarenecessarilypartial.Theothertrendsarethe“Cinderella”ofopenscienceintermsofdataavailability.Forthefuture,weneedthereforetoincreasetheattentiontomeasuringreliablythedifferenttrends.Thiscouldbe done in the first instance by extending the survey fromopen research data to otherimportanttrends,startingfromopenscientificsoftware;secondly,throughadhocsamplebasedbibliometricanalysistoassesstheavailabilityofdataandcodeassociatedtoarticles.
3. MultiplesourcesandmethodsshouldcontinuetobeusedIndicatorsproducedbythemonitorshouldrelyonthewidestsetofsourcesandmethods.The only criterion for choice should remain reliability and validity of indicators. A goodexampleishowthepresentMonitorextendedthesourcesforopenaccessmetricstoincludeUnpaywall alongside Scopus. This integration of different sources allowed a more fine-grained,timelyandextendedcoverage.Inadditiontodataintegration,datatriangulationisalsonecessarytoreinforcethelegitimacyofthefindings.ThepresentMonitorcomparedtheresultswithothersources,suchasWebofScienceforopenaccessandthe“Stateofopendata”surveybyFigshare,recognizingexplicitlythesimilaritiesandcontradictions.Inotherwords,thefutureOpenScienceMonitorshouldbeasbroadaspossibleandprovideacriticalmeta-analysisofothersurveysandstudies.
4. ThekeyfocusshouldbeadoptionOpenscienceismaturing,andsoshoulddotheindicators.Supplysideindicatorssuchasthemereavailabilityoftools,thenumberofdatarepositories,ortheexistenceofpoliciesshouldbesecondarytotheotherpriority:measuringtheadoptionofopensciencepracticesbyallstakeholders.Adoptionshouldincludenotonlythenumberorpercentageofusers,buttheintensityorfrequencyofadoption.Inotherwords,theindicatorsshouldnotonlymeasurethepercentageofscientiststhatpublishinopenaccess,sharedataandcode,butwhether
65
they do it systematically on all papers or occasionally. When measuring adoption,bibliometricindicatorsappearmoreeffectivethansurveys,insofartheyallowforobjectivesmeasurementofbehaviourratherthanself-reported.
5. Bibliometricsfirst,surveysecondThereislittledoubtthatmeasuringadoptionisbetterthroughobservationratherthanself-reporting.Askingresearcherswhethertheyadoptopensciencepracticesismorepronetobiasedresults thangatheringbibliometricdataaboutwhethertheyeffectivepublishdataandcodealongsideanarticle.Indeed,itismoredifficulttomeasuredataandcodesharing,thanitistomeasurepublicationinopenaccess,butitisstilldoable.Forinstance,samplebased bibliometric have been used by many individual studies – while open access topublications is measured on the universe of publications. Finally, survey still have animportantroletoplay,inparticularwhenitcomestoassessingdriversandbarriers.Instead,adoptionmetricsfromspecificservicesareoflimitedvalue:indicatorssuchasthenumberofcitizenscienceoropenhardwareprojectscanprovideasuggestionofgrowthinusage,butdonotallowforassessingadoptiongrowthintheoverallscientificcommunityasawhole.Finally,casestudiesremainuniqueinprovidinginsightaboutabetterunderstandingofhowopenscienceworksandtheimpactitachieves.Theyareveryimportantinthisearlyandfaststageofevolution.Insummary,wesuggestmovingtowardsgreaterusageofbibliometricindicatorsforassessingadoption,andreducingmetricsprovidedbyspecificservices,whilededicatingsurveysandcasestudiesspecificallytodriversandimpacts,asshowninthetablebelow.Table10:Highlevelmethodologicalapproach,past(x)andfuture(greyshade)
Drivers Open access Open data Open collaboration
Impact
Bibliometric (universe)
x
Bibliometric (sample)
Survey x x x Service metrics
x x x
Cases x x
6. The next open sciencemonitor should be as open as possible – and as closed asnecessary
TherehavebeensomestrongcriticismsbypartoftheopensciencecommunitytowardsthepresentMonitorbecauseoftheusageofproprietarydata.Thishasledtoaseriousinternalreview of the methodology and significant improvement, including an expansion of theoriginalsources.Themethodologywasthendiscussed inahigh-profileexpertworkshop.Theresultsconfirmedthechoicetousesomeproprietarydata,becausetheyprovidehighquality additional information (namely about affiliation). The final choice should bepragmaticanddrivenbytheobjective,whichistoprovidereliablehigh-qualitydatapointsinsupporttopolicy.Asoftoday,thereisstillroomforproprietarydatainthiscontext.Using
66
onlyopendatawouldleadtolowerqualityoutput.Still,thereareimportantoptionstobeused when using proprietary data, such as allowing researchers to access the data ondemandforreplicationandresearchpurposes.Andthisisexactlywhathasbeendone.Onasimilarnote,theprocessshouldcontinuetoberadicallyopen,butthisdoesnotmeanthat decisions should be taken by the community. There is a need for strong centralleadershipinmakingtheappropriatemethodologicalchoices.Asinthecaseofthepresentmonitor, the open process included allowing everyone to post comments on themethodology (visible to everyone), and provided systematic feedback to each individualcomment about the choices made. The choices are made centrally, but after openconsultation,expertreviewandprovidingclearfeedbackontherationalebehindrefusingoracceptingsuggestions.Inotherwords,boththedataandtheprocessshouldbeasopenaspossibleandas closedasnecessary– butalways transparent.Theopensciencemonitorshouldembracethespiritofopenscience–butaswehaveseenintheanalysis,thisdoesnotmeanexcludingatallcostanyformofclosedprocessandoutput.Justasopenscience,theopensciencemonitorrequirescarefulgovernanceoftrade-offs.
7. Thescopeofthework,andthecommunityaddressed,shouldbescienceasawhole,notopenscience
The open science monitor has to cover the developments across science – not merelytrackingthosewhopracticeopenscience. Itshouldnotprovide indicatorsonlyabout theadoptersbutobservethetransitionintheoverallscientificprocess.Thisiswhyitshouldnolongerprioritizedataprovidedbyopenscienceservices,suchasthenumberofusersofaservices, or the number of projects in an open repository. It should instead focusmoreheavily on measuring the adoption of open science practices by the whole scientificcommunity–suchasthepercentageofarticlesorresearchersthatincludetheadoptionofopensciencetrends.Equally,theusersandthestakeholdersoftheMonitorarenottheopensciencecommunity,butthescientificcommunityasawhole.Thisisasdifficultasitisimportant,becausetheopensciencecommunityisveryactiveanddedicated–andvocalonsocialmedia.Ultimately,thelegitimacyoftheMonitorcomesfromtheattentionandimpactintheoverallscientificcommunity,not from theapprovalofopenscienceadvocates.TheOpenScienceMonitorshoulddesignitsservicesaroundtheneedsofthescientificcommunity–andproactivelyreach out through articles, presentations and communication as a priority, rather thanengagemainlywiththevocalopensciencecommunity.
67
7 ReferencesAndreoli-Versbach,P.andMueller-Langer,F.(2014)Openaccesstodata:Anidealprofessed
butnotpractised.ResearchPolicy,43(9),pp.1621-1633.Archambault,É.,Amyot,D.,Deschamps,P.,Nicol,A.,Provencher,F.,Rebout,L.,andRoberge,
G. Proportion of Open Access Papers Published in Peer-Reviewed Journals at theEuropeanandWorldLevels—1996–2013,ReporttotheEuropeanCommission,April2014
Arnold, E. (2005). What the Evaluation Record tells us about Framework ProgrammePerformance.Management,(January).
Auer, S., Bizer, C., Kobilarov,G., Lehmann, J., Cyganiak, R. and Ives, Z. (2007)Dbpedia: Anucleus for a web of open data. The semantic web (pp. 722-735). Springer BerlinHeidelberg.
Baker,M(2016).1,500scientistsliftthelidonreproducibility.Nature,533(7604):452–454.DOI:https://doi.org/10.1038/533452a
Bannister,F.(2007).Thecurseofthebenchmark:anassessmentofthevalidityandvalueofe-government comparisons. International Review of Administrative Sciences, 73(2),171–188.http://doi.org/10.1177/0020852307077959
Berghmans,S.,Cousijn,H.,Deakin,G.,Meijer,I.,Mulligan,A.,Plume,A.,…Waltman,L.(2017).Opendata :Theresearcherperspective,48p.http://doi.org/10.17632/bwrnfb4bvh.1
Blumenthal, D., E.G. Campbell,M. Gokhale, R.Yucel,B. Clarridge, S.Hilgartner, andN. A.Holtzman(2006).DataWithholdinginGeneticsandtheOtherLifeSciences:PrevalencesandPredictors.AcademicMedicine,81(2),137-145.
Boulton,G.,Campbell,P.,Collins,B.,Elias,P.,Hall,W.,Laurie,G., ...andWalport,M.(2012)Scienceasanopenenterprise.TheRoyalSociety.
Bücheler, T. and Sieg, J. H. (2011). Understanding science 2.0: crowdsourcing and openinnovationinthescientificmethod.ProcediaComputerScience,7,327-329.
Burgelman,J.C.,Osimo,D.andBogdanowicz,M.(2010)Science2.0(changewillhappen….).FirstMonday,15(7).
Burgelman,J.-C.,Osimo,D.,&Bogdanowicz,M.(2010).Science2.0(changewillhappen…).FirstMonday,15(7).
Campbell,E.G.,Clarridge,B.R.,Gokhale,M.,Birenbaum,L.,Hilgartner,S.,Holtzman,N.A.andBlumenthal,D.(2002)Datawithholdinginacademicgenetics:evidencefromanationalsurvey.Jama,287(4),473-480.
Carlson, S and Anderson, B 2007 What Are Data? The Many Kinds of Data and TheirImplications for Data Re-Use. Journal of Computer-Mediated Communication, 12(2):635–651.DOI:https://doi.org/10.1111/j.1083-6101.2007.00342.x
Carrara,W., Nieuwenhuis,M., & Vollers, H. (2016). Open DataMaturity in Europe 2016.Brussels.
Chokshi, D. A., Parker, M. and Kwiatkowski, D. P. (2006). Data sharing and intellectualproperty in a genomic epidemiology network: policies for large-scale researchcollaboration.BulletinoftheWorldHealthOrganization,84(5),382-387.
CODATA-ICSTITaskGrouponDataCitationStandardsPractices2013OutofCite,OutofMind:TheCurrentStateofPractice,Policy,andTechnologyfortheCitationofData.DataScienceJournal,12:CIDCR1–CIDCR75.DOI:https://doi.org/10.2481/dsj.OSOM13-043
Collison,P.,&Nielsen,M.(2018,November16).ScienceIsGettingLessBangforItsBuck.
68
Retrieved fromhttps://www.theatlantic.com/science/archive/2018/11/diminishing-returns-science/575665/
Costas,R.,Zahedi,Z.,Wouters,P.(2015a).Do“altmetrics”correlatewithcitations?Extensivecomparisonofaltmetricindicatorswithcitationsfromamultidisciplinaryperspective.JournaloftheAssociationforInformationScienceandTechnology,66(10),2003–2019.https://doi.org/10.1002/asi
Costas, R., Zahedi, Z., Wouters, P. (2015b). The thematic orientation of publicationsmentionedonsocialmedia:large-scaledisciplinarycomparisonofsocialmediametricswith citatios. Aslib Journal of Information Management, 67(3), 260–288.https://doi.org/10.1108/AJIM-12-2014-0173
Crouzier,T.(2015).ScienceEcosystem2.0:howwillchangeoccur.Actabiomaterialia,20,51-59.
Dasgupta,P.andDavid,P.A.(1994)Towardaneweconomicsofscience.ResearchPolicy,23,487–521.
David, P. A. (2004) Can "open science" be protected from the evolving regime of IPRprotections?.JournalofInstitutionalandTheoreticalEconomics(JITE)/ZeitschriftfürdiegesamteStaatswissenschaft,9-34.
David,P.A.(2004)Patronage,Reputation,andCommonAgencyContractingintheScientificRevolution:FromKeeping’Nature’sSecrets’totheInstitutionalizationof’OpenScience’.SIEPRPolicypaperNo.03-039.
Denning,D.E.andSchlörer,J.(1980).Afastprocedureforfindingatrackerinastatisticaldatabase.ACMTransactionsonDatabaseSystems(TODS),5(1),88-102.
Dosi,G.,Llerena,P.andLabini,M.S.(2006)Therelationshipsbetweenscience,technologiesandtheirindustrialexploitation:Anillustrationthroughthemythsandrealitiesoftheso-called‘EuropeanParadox’.Researchpolicy,35(10),1450-1464.
Drummond,C2009ReplicabilityisnotReproducibility:NorisitGoodScience.In:ICML’09Proceedingsof the26thAnnual InternationalConferenceonMachineLearning (Thefourth workshop on evaluation methods for machine learning). Montreal, Quebec,Canada.Retrievedfrom:http://cogprints.org/7691/.
Eisenhardt, K. M. (1989). Building Theories from Case Study Research. Academy ofManagementReview,14(4),532–550.http://doi.org/10.5465/AMR.1989.4308385
Eisenhardt, K. M., & Graebner, M. E. (2014). THEORY BUILDING FROM CASES :OPPORTUNITIESANDCHALLENGESdiverse.AcademyofManagementJournal,50(1),25–32.http://doi.org/Article
EuropeanCommission(2009)PreparingEuropeforanewRenaissance:AstrategicviewoftheEuropeanresearcharea.FirstreportoftheEuropeanResearchAreaBoard—2009.Brussels:EuropeanCommission.
EuropeanCommission(2010)Europe2020:AStrategyforSmart,SustainableandInclusiveGrowth:CommunicationfromtheCommission.Brussels:EuropeanCommission.
EuropeanCommission(2012)Recommendationonaccesstoandpreservationofscientificinformation.Brussels:EuropeanCommission
European Commission (2014) Innovation Union Scoreboards. Brussels: EuropeanCommissionDGEnterprise
European Commission Expert Group on FAIR Data (November, 2018). Turning Fair intoReality-FinalReportandActionPlanfromtheEuropeanCommissionExpertGrouponFAIRData.EuropeanCommissionDirectorateGeneralforResearchandInnovation.
69
EuropeanCommissionHighLevelExpertGrouponScientificData2010Ridingthewave:HowEuropecangainfromtherisingtideofscientificdata(FinalreportoftheHighLevelExpertGrouponScientificData.AsubmissiontotheEuropeanCommission).EuropeanUnion.Retrievedfrom:https://www.fosteropenscience.eu/content/riding-wave-how-europe-can-gain-rising-tide-scientific-data.
Fang,F.C.,&Casadevall,A. (2015).CompetitiveScience: IsCompetitionRuiningScience?InfectionandImmunity,83(4),1229–1233.http://doi.org/10.1128/IAI.02939-14
Fecher,B.,Friesike,S.andHebing,M.(2015)Whatdrivesacademicdatasharing?PLoSOne,10(2),e0118053.
Feller,J.andFitzgerald,B.(2002).Understandingopensourcesoftwaredevelopment(pp.143-159).London:Addison-Wesley.
Franceschet,M.andCostantini,A.(2010)Theeffectofscholarcollaborationonimpactandqualityofacademicpapers.Journalofinformetrics,4(4),540-553.
Friesike, S., Widenmayer, B., Gassmann, O. and Schildhauer, T. (2015) Opening science:towardsanagendaofopenscienceinacademiaandindustry.TheJournalofTechnologyTransfer,40(4),pp.581-601.
Furman, J. L., Porter,M. E. and Stern, S. (2002)The determinants of national innovativecapacity.Researchpolicy,31(6),899-933.
Gittelman, M. and Kogut, B. (2003) Does good science lead to valuable knowledge?Biotechnology firms and the evolutionary logic of citation patterns. ManagementScience49,366–382.
Goodman,AA,Pepe,A,Blocker,AW,Borgman,CL,Cranmer,K,Crosas,M,etal.2014TenSimpleRulesfortheCareandFeedingofScientificData.PLoSComputationalBiology,10(4):e1003542.DOI:https://doi.org/10.1371/journal.pcbi.1003542
Guston, D. H. (2000) Retiring the social contract for science. Issues in Science andTechnology,16(4),32-36
Haeussler, C. (2011) Information-sharing in academia and the industry: A comparativestudy.ResearchPolicy,40(1),pp.105-122.
Haeussler,C.,Jiang,L.,Thursby,J.andThursby,M.(2014)Specificandgeneralinformationsharingamongcompetingacademicresearchers.Researchpolicy,43(3),pp.465-475.
Haustein,S.,Costas,R.,Larivière,V.(2015).Characterizingsocialmediametricsofscholarlypapers: the effect of document properties and collaboration patterns. PloS ONE.https://doi.org/10.1371/journal.pone.0120495
Herman,E.,Jamali,H.R.,Nicholas,D.,Osimo,D.,Porcu,F.andPujol,L.(2015).AnalysisofEmergingReputationandFundingMechanismsintheContextofOpenScience2.0(No.JRC94952).InstituteforProspectiveTechnologicalStudies,JointResearchCentre.
Hey,A.J.andTrefethen,A.E.(2003).Thedatadeluge:Ane-scienceperspectiveHoughton,J.,Swan,A.andBrown,S.(2011)AccesstoResearchandTechnicalInformationin
Denmark:ReporttoTheDanishAgencyforScience,TechnologyandInnovation(FI)andDenmark'sElectronicResearchLibrary(DEFF.DanishAgencyforScience,TechnologyandInnovation.
Jong, S. and Slavova, K. (2014) When publications lead to products: The open scienceconundruminnewproductdevelopment.ResearchPolicy,43(4),pp.645-654.
Jonkers, K., & Zacharewicz, T. (2016). Research Performance Based Funding Systems: aComparativeAssessment.EuropeanCommission.http://doi.org/10.2791/659483
Jubb, M., Plume, A., Oeben, S., Brammer, L., Johnson, R., Bütün, C., & Pinfield, S. (2017).
70
Monitoringthetransitiontoopenaccess.Jubb, M., Plume, A., Oeben, S., Brammer, L., Johnson, R., Bütün, C., & Pinfield, S. (2017).
Monitoringthetransitiontoopenaccess.Kowalczyk, S. and Shankar, K. (2011) Data sharing in the sciences. Annual review of
informationscienceandtechnology,45(1),247-294.Lakhani,K.R.andPanetta,J.A.(2007)Theprinciplesofdistributedinnovation.Innovations,
2(3),97-112.Lawson, S (2017).Reportonoffset agreements: evaluating current JiscCollectionsdeals.
Year2–evaluating2016deals.Liu,C.C.andStuart,T.(2014).Positionsandrewards:Theallocationofresourceswithina
science-basedentrepreneurialfirm.ResearchPolicy,43(7),pp.1134-1143.Lundvall,B.A. (1992)National systemsof innovation:Ananalytical framework.London:
Pinter.Lyon, L. (2009). Open Science at Web-Scale : Optimising Participation and Predictive
PotentialConsultativeReport.InformationSystemsJournal,29(November),1–45.Margolis,R,Derr,L,Dunn,M,Huerta,M,Larkin, J, Sheehan, J, et al. (2014).TheNational
Institutes of Health’s Big Data to Knowledge (BD2K) initiative: capitalizing onbiomedicalbigdata. Journalof theAmericanMedical InformaticsAssociation,21(6):957–958.DOI:https://doi.org/10.1136/amiajnl-2014-002974
Maxwell,J.A.(2012).QualitativeResearchDesign:AnInteractiveApproach:AnInteractiveApproach.
Mayer-Schonberger,VandCukier,K(2013)BigData:ARevolutionThatWillTransformHowWeLive,Work,andThink.Boston:HoughtonMifflinHarcourt.
McKiernan,E.C.,Bourne,P.E.,Brown,C.T.,Buck,S.,Kenall,A.,Lin,J.,McDougall,D.,Nosek,B.A.,Ram,K.,Soderberg,C.K.andSpies,J.R.(2016)Howopensciencehelpsresearcherssucceed.Elife,5,p.e16800.
Murray,F.,andS.Stern(2007)."DoFormalIntellectualPropertyRightsHindertheFreeFlowofScientificKnowledge?AnEmpiricalTestoftheAnti-CommonsHypothesis,"JournalofEconomicBehavior&Organization,63(4),648–687.
Nagel,E.,&Kaplan,A.(1965).TheConductofInquiry:MethodologyforBehavioralScience.AmericanSociologicalReview(Vol.30).http://doi.org/10.2307/2091353
NationalInstitutesofHealth(2015)CommonsHomePage|DataScienceatNIH.Retrievedfrom:https://datascience.nih.gov/commons
National Institutes of Health (2016) NIH Sharing Policies and RelatedGuidance on NIH-Funded Research Resources. Retrieved from:https://grants.nih.gov/policy/sharing.htm
NationalScienceFoundation(2011)AwardandAdministrationGuide:DataSharingPolicy.Retrieved from:https://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/aag_6.jsp#VID4
Nielsen,M. (2012). Reinventing discovery: the new era of networked science. PrincetonUniversityPress.
OECD(2015)MakingOpenScienceaReality.OECDScience,TechnologyandIndustryPolicyPapers,No.25,OECDPublishing,Paris.
Olensky, M., Schmidt, M., & Van Eck, N.J. (2016). Evaluation of the Citation MatchingAlgorithms of CWTS and iFQ in Comparison to the Web of Science. Journal of theAssociation for Information Science and Technology, 67(10), 2550-2564.
71
doi:10.1002/asi.23590.Olson,G.M.,Zimmerman,A.andBos,N.(2008)ScientificcollaborationontheInternet.The
MITPress.OpenKnowledge Foundation (2015)OpenDefinition:DefiningOpen inOpenData, Open
ContentandOpenKnowledge.Retrievedfrom:http://opendefinition.org/od/.Orelli, B (2016) The Business of Genomic Data. O’Reilly Media. Retrieved from:
http://www.oreilly.com/data/free/the-business-of-genomic-data.csp.Organisation for Economic Co-operation and Development (2007) OECD Principles and
Guidelines forAccess toResearchData fromPublic Funding. Paris: Organisation forEconomic Co-Operation and Development, p. 24. Retrieved from:http://www.oecd.org/dataoecd/9/61/38500813.pdf.
Palmer,CL,Weber,NMandCragin,MH(2011)TheAnalyticPotentialofScientificData:Understand¬ing Re-use Value. Proceedings of the American Society for InformationScienceandTechnology,48(1):
Pasquetto, I V, Sands, A E, Darch, P T and Borgman, C L (2016) Open Data in ScientificSettings:FromPolicytoPractice.In:Proceedingsofthe2016CHIConferenceonHumanFactorsinComputingSystems.Presentedatthe2016CHIConferenceonHumanFactorsin Computing Systems, New York, NY, USA: ACM, pp. 1585–1596. DOI:https://doi.org/10.1145/2858036.2858543
Pasquetto,I.,Randles,B.,&Borgman,C.(2017).OntheReuseofScientificData.DataScienceJournal,16.
Perkmann, M. and Schildt, H., (2015). Open data partnerships between firms anduniversities:Theroleofboundaryorganizations.ResearchPolicy,44(5),pp.1133-1143.
Pilat,D.andFukasaku,Y.(2007).OECDprinciplesandguidelinesforaccesstoresearchdatafrompublicfunding.DataScienceJournal,6,OD4-OD11.
Piwowar,H.A.,Vision,T.J.andWhitlock,M.C.(2011).Dataarchivingisagoodinvestment.Nature,473(7347),285-285.
Porter,M.E.(1990)Thecompetitiveadvantageofnations.HarvardBusinessReview,68(2),73-93.
Powell,A. (2012).Democratizingproductionthroughopensourceknowledge: fromopensoftwaretoopenhardware.Media,Culture&Society,34(6),691-708.
Powell,A.B.(2015).Opencultureandinnovation:integratingknowledgeacrossboundaries.Media,Culture&Society,
Procter, R.,Williams, R., Stewart, J., Poschen,M., Snee, H., Voss, A. and Asgari-Targhi,M.(2010)AdoptionanduseofWeb2.0 inscholarlycommunications.PhilosTransactAMathPhysEngSci,368(1926),4039–4056.
Raymond,E.(1999)Thecathedralandthebazaar.Knowledge,Technology&Policy,12(3),23-49.
Robinson-García, N., Torres-Salinas, D., Zahedi, Z., Costas, R. (2014). New data, newpossibilities:exploringtheinsidesofAltmetric.com.ElProfesionaldeLaInformacion,23(4),359–366.https://doi.org/10.3145/epi.2014.jul.03
Savage,C.J.andVickers,A.J.(2009)EmpiricalstudyofdatasharingbyauthorspublishinginPLoSjournals.PLoSONE,4(9),9–11.
Science,Digital;Hahnel,Mark;Fane,Briony;Treadway,Jon;Baynes,Grace;Wilkinson,Ross;et al. (2018): The State of Open Data Report 2018. figshare. Paper.https://figshare.com/articles/The_State_of_Open_Data_Report_2018/7195058
72
ShreeveJ(2005)TheGenomeWar:HowCraigVenterTriedtoCapturetheCodeofLifeandSavetheWorld.BallantineBooks:NewYork.
Sorenson,O.andFleming,L.(2004)Scienceandthediffusionofknowledge.Researchpolicy,33(10),pp.1615-1634.
Stodden,V.(2010).TheScientificMethodinPractice:ReproducibilityintheComputationalSciences.
Szkuta,K.andOsimo,D.(2016)Rebootingscience?Implicationsofscience2.0maintrendsforscientificmethodandresearchinstitutions.Foresight,18(3).
Szkuta,K.,&Osimo,D.(2016).Rebootingscience?Implicationsofscience2.0maintrendsfor scientific method and research institutions. Foresight, 18(3).http://doi.org/10.1108/FS-06-2014-0040
Taubenberger, J.K.,Reid,A.H.,Lourens,R.M.,Wang,R., Jin,G. andFanning,T.G. (2005)Characterizationofthe1918influenzaviruspolymerasegenes.Nature,437.
Tenopir,C,Allard,S,Douglass,K,Aydinoglu,AU,Wu,L,Read,E,etal.2011DataSharingbyScientists: Practices and Perceptions. PLoS ONE, 6(6): e21101. DOI:https://doi.org/10.1371/journal.pone.0021101
TheMoRRI indicators report,MonitoringReportD4.3,Pre-publicationversion (February2018): http://www.technopolis-group.com/report/evolution-responsible-research-inovation-europe-morri-indicators-report-d4-3/
UE. (2015). Better Regulation Guidelines. Guidelines European Commission.http://doi.org/10.1016/S0022-3913(12)00047-9
Van Leeuwen, T., Meijer, I., Yegros, A., Costas, R. (2017). Developing indicators on OpenAccessby combiningevidence fromdivesedata sources. STI2017Conference.Paris(France)
Von Hippel, E. (2009) Democratizing innovation: the evolving phenomenon of userinnovation.InternationalJournalofInnovationScience,1(1),29-40.
Walsh,J.P.andHuang,H.(2014)Localcontext,academicentrepreneurshipandopenscience:Publication secrecy and commercial activity among Japanese and US scientists.ResearchPolicy,43(2),pp.245-260.
Waltman,L.,Costas,R.(2014).F1000RecommendationsasaPotentialNewDataSourceforResearch Evaluation: A Comparison With Citations. Journal of the Association forInformationScienceandTechnology,65(3),433–445.https://doi.org/10.1002/asi
Whyte,A.,andPryor,G.(2010)Opentoall?CasestudiesofopennessinResearch.UnitedKingdom:Nesta
Wilkinson,MD,Dumontier,M,Aalbersberg,IjJ,Appleton,G,Axton,M,Baak,A,etal.(2016)TheFAIRGuidingPrinciplesforscientificdatamanagementandstewardship.ScientificData,3:160018.DOI:https://doi.org/10.1038/sdata.2016.18
Wilsdon, J. R., Bar-Ilan, J., Frodeman, R., Lex, E., Peters, I., & Wouters, P. (2017). Next-generationmetrics:responsiblemetricsandevaluationforopenscience.
Wood,J.,Andersson,T.,Bachem,A.,Best,C.,Genova,F.,Lopez,D.R.,Los,W.,Marinucci,M.,Romary,L.,VandeSompel,H.andVigen,J.,(2010)Ridingthewave:HowEuropecangainfromtherisingtideofscientificdata.EuropeanUnion.
Wouters, P., Costas, R. (2012). Users , narcissism and control – tracking the impact ofscholarlypublications in the21stcentury. (M.VanBerchumK.Russell,Eds.), ImageRochester NY. SURFfoundation. Retrieved fromhttp://www.surffoundation.nl/en/publicaties/Pages/Users_narcissism_control.aspx
73
Yin,R.K.(2009).Casestudyresearch :designandmethods.Appliedsocialresearchmethodsseries ;(Vol.5.).http://doi.org/10.1097/FCH.0b013e31822dda9e
Zahedi, Z., Costas, R., Wouters, P. (2014). How well developed are altmetrics? cross-disciplinaryanalysisofthepresenceof“alternativemetrics”inscientificpublications.Scientometrics.https://doi.org/10.1007/s11192-014-1264-0
Zahedi,Z.,Fenner,M.,Costas,R.(2014).Howconsistentarealtmetricsproviders?Studyof1000PLOSONEpublicationsusingthePLOSALM,MendeleyandAltmetric.comAPIS.Inaltmetrics14:expandingimpactsandmetrics(pp.5–6).Bloomintong,Indiana,USA:AnACM Web Science Conference 2014 Altmetrics Workshop.https://doi.org/10.6084/m9.figshare.1041821