publication quality: metrics and beyond

PublicationQuality:MetricsandBeyondNYCCTChairsColloquium,April232015Prof.MonicaBerger,Library

ThistalkisaheavilyrevisedversionofatalkthatMauraandIgaveaboutthreeandhalfyearsago.Sincethen,somethingshavechanged.

1. Thecollegehasshifteditsexpectationsforscholarship2. Thebroaderdefinitionofscholarshipisrapidlyevolvedandevenhardertoclearlydefine.

Thiscomplexityaffectstraditionalmeasuresofquality.3. So‐called“predatorypublishing”hasarisen.Whetherthesepublishersaretotallyshamor

justlowquality,manyacademicshavebeeneitherdupedor,toalesserdegree,havedeliberatelypublishedinjournalsthatmayclaimtobepeer‐reviewedbutaretypicallyextremelylowcaliber.

Whatisthedriverbehindsomuchofthischange?Iwouldsayitistechnology.Twentyyearsago,wecouldn’timaginescholarlydiscourseinvolvinginstitutionalandsubjectrepositories,opendataorsocialmedia.Aproductoftechnology,OpenAccesspublishingismuchmorewidelyacceptedand,incertainareasandplaces,mandated.Usingasimplisticdefinition,OpenAccesscontentisfreelyavailable.Scholarlyworkisnolongerexclusivelytext‐basedanditisoftenonlinemakingitwidelyavailableforaccess,disseminationand,increasingly,repurposingfornewscholarship.Tosummarizethesetechnology‐drivenchanges,

PRINTONLINEslide2TEXTMULTIMEDIAslide3STATICINTERACTIVEslide4

FORMALINFORMALANDFORMALslide5CLOSEDOPEN,slide6

Slide7:examplesfromLavoieLavoie,Brian,EricChildress,RickyErway,IxchelFaniel,ConstanceMalpas,JenniferSchaffner,andTitiavanderWerf.2014.TheEvolvingScholarlyRecord.Dublin,Ohio:OCLCResearch.http://www.oclc.org/research/publications/library/2014/oclcresearch‐evolvingscholarly‐record‐2014.pdfInthisnewenvironment,thereisincreasingemphasisonboth“replicabilityofscholarlyoutcomes”aswellas“leveragability,”meaningthatpreviouslypublishedscholarshipcanbeseamlesslyintegratedintonewscholarship.[Lavoie,p.9].Thescholarlyrecordhasbecomebroaderanddeeper.Hereisaslideshowingexamplesofscholarshipvis‐a‐vis

METHODEVIDENCEDISCUSSION

METHOD:Anewjournal,MethodsX,focusesonresearchmethodology.EVIDENCE:Dryadisarepositoryofdatasetsassociatedwithpublishedarticlesinthelifesciences.GeoscienceDataJournalmakesdataavailableviaamoretraditionalchannel,apeer‐reviewedjournal[notavailabletoNYCCT].DISCUSSION:Hereisaresourcethatwillbefamiliartosomeofyou!ArXivisaphysicspre‐printrepositorythatallowsscholarstogiveandreceivemeaningfulfeedbackinadvanceofsubmittingforformalpublication.Andthiscontentisopenlyavailable.Whataboutdisseminationofresearchtoboththescholarlycommunityandthepublic?DISCUSSION:WhyNationsFailisabloglaunchedinparallelwiththepublicationofabookallowingfordiscussioninabroaderintellectualandpubliccontext.REVISION:Figshareisaservicethatallowsresearcherstoexpandandenhancetheirworkbyuploadingadditionalmaterialsrelatedtotheirscholarship.REUSE:F1000Postersisanopenaccessrepositoryforpostersandpresentationsinthelifesciencesthatprovidesforreuseandremixofcontent.

Slide8:ScholarlycommunicationslifecycleTraditionally,it’slogicaltothinkofthescholarlycommunicationsprocessascircular,movingfromtheinformaltotheformalandbackagain.Iliketheideaofcallingitalifecycle.ThisschemaisstilltruebutnowmorecomplexaspersomeoftheexamplesIshowedbefore!Thesliderepresentsthecycleinoceanography.

http://libguides.humboldt.edu/content.php?pid=457571&sid=3746574

Slide9:ACADEMICTRIBES

Everyacademicdisciplinehasitsowndistinctive,andoftenexoticculture.Sometimes,withinthesetribes,therearewarringandpeacefulfactions.Theseculturesheavilyaffectconsiderationsofqualityofscholarshipandpublication.ConsiderthefollowingfactorswhichIspelloutonyourhandout:

FORMAT:Whatisthegoldstandardforformat?Bookorjournalarticle?‐‐whichcanalsointurnaffect

SPEEDofpeer‐reviewandpublication/dissemination OPENNESS:doscholarstalkaboutandsharetheirworkpreorpost‐publication,generating

greaterfeedbackinadvanceofsubmissionandproducingmorereadersingeneral?Doinformalscholarlycommunicationinformthediscourse?DoesthedisciplineembraceOpenAccess?Iftheresearchhasacommercialvalue,opennessmaybelessfeasible.

THEORETICALversusAPPLIED;whatarethestandardsforappliedresearch?Considerthetime,financial,andlogisticalburdensofappliedovertheoreticalresearch.

AUTHORSHIP:Howmanyauthorsaretypicalforthediscipline?Doscholarsworksoloorinlargeteams?

QUANTITY:doscholarspublishmanyarticlesinthediscipline?

AUDIENCE:doscholarsseektoreachabroadercross‐sectionofreadersincludingnon‐academics?

Interdisciplinaryscholarshipraisesadditionalcomplicationsintermsofqualitativeanalysis.Audience(s):Whoisthescholarseekingtoreach?Isa“realworld”applicationofthescholarshipthedesiredoutcomeoristheaudiencehighlyspecificwithinacademe?Format(again):Isthescholarshiptraditionalorisitengagingwithnewformatsforscholarlycommunication?Digitalhumanitiesrepresentsaperfectexampleofanewformatthatisincreasinglyacceptedandvalued!Slide10:Lookingatjournalpublishingindifferentdisciplines

Source:Greco,A.N.,etal."TheStateofScholarlyJournalPublishing:1981‐2000."Journalofscholarlypublishing37.3(2006):155‐214.

ThisslideisfromanarticlebyAlbertGrecowhoisanexpertontheeconomicsofpublishingandwhoteachesatFordhamUniversity.Itisfascinatingtoseehowdifferentdisciplineshaveaverydifferentnumberofjournalsaswellascitationrates.Science‐math‐statisticsdominatescholarlyjournalarticlepublishingsincethesepublishersarecommercialacademicpublishersORscholarlyandprofessionalsocieties.Thesepublisherschargesteepprices,havemorefull‐timestaffandhencehavefasterturn‐aroundtime.Humanitiesandsomesocialsciencestendtobepublishedbyuniversitypressesand/orhavelessinfrastructure.

Herearesometipstohelpwithevaluation:EverydisciplinehasascholarlyorprofessionalsocietyandIwouldsuggestthatwouldbethefirstplacetolookifevaluatingtheworkofacolleagueinadifferentdiscipline.Thesocietywillprovideanoverviewofthescholarlypracticesofthatdiscipline.Some,ifnotmany,provideguidelinesforevaluatingscholarship.Itisalsohelpfultonotethatwithinsomedisciplinestherearedistinctiveandimportantsubdisciplinesthatmightuseverydifferentmethodologiesandtheirownacademicsocietiesanddifferentcriteriaforqualityinscholarship.Anthropologyisagreatexampleofsuchadisciplinewherethesubdisciplinespublishverydifferently.

HardresearchonthedifferencesindisciplinaryscholarlypracticecanbefoundinavarietyofvenuesbutI’drecommendtheJournalofScholarlyPublishingwhichwehaveaccesstoinProjectMuseaswellasTheCenterforStudiesinHigherEducation(CSHE)’sFutureofScholarlyCommunicationinitiativeatU.Cal.Berkeley.

Crisisinacademicmonographicpublishing.Asfarbackas2006,Grecopredictedjournalswillbecomemoreimportantasbookpublishingshrinks.Thishas,inpart,cometrue.Universitypresseshavecutbacksignificantlyonprintmonographicpublishing.Thereissomesignificantmovementtowardspublishingelectronic‐onlymonographsandinEuropeandtheU.K.wheretherearesignificantOpenAccessmandates,OpenAccessmonographicpublishingseemsmorecommonthanintheU.S.Themonographscrisisclearlyaffectsthehumanitiesand,toalesserdegree,thesocialsciences.

Slide11:Openaccess

OpenAccessRaisesissuesofhowwidelyreadwillanarticlebeifonlymajorresearchlibrariescanaffordthejournal.Howdifficultisittoaccessthejournalorthearticle?Fantasticvariationinpracticeandatopicofresearchers.Broadly,researchconfirmsthatOpenAccessscholarshipgetsmorereadership.

Disciplinarydifferencesrelevanttoopenaccess(partlyborrowedfromPeterSuber)

1. Somefieldshaveanestablishedcultureofpreprintexchange,andsomedon't.2. Inthesciences,journalliteratureistheprimaryliterature,whileinthehumanities,

monographsdominateandareunlikelytobeOpenAccess3. Insomefields,moneyisatstake,eitherforthedominantsocietyorcommercialpublisher

orfortheauthor!Didtheauthorhavetogetfundingtodoresearchinfirstplace?Orarenon‐profituniversitypressesdominantinthatdiscipline?

4. ToreiterateapointIbroughtoutearlier,insomefields,mostjournalpublishersarefor‐profitcorporations,whileinotherfieldsmostarenon‐profituniversities,libraries,orprofessionalsocieties.http://www.earlham.edu/~peters/fos/lists.htm#disciplines

Newdisciplines;interdisciplinaryareasWithoutgettingintoanydetails,thenewerthediscipline,themorechallengingitmightbeforthescholartoplacehisorherwork.Interdisciplinaryworkisalsomoreofachallengetobothpublishandevaluate.MoreonevaluatingarticlesLet’snowleavebehindthequestionofdisciplinarydifferences.

Slide12:Peerreview

Peer‐reviewEverypeer‐reviewedjournalwillstateapeer‐reviewpolicyonitseditorialsitewhichshouldalwaysbeavailabledirectlyontheweb.Therearedifferentlevelsofpeer‐reviewrangingfrombasicpeer‐reviewtodouble‐blindpeerreview.Ahallmarkofapredatoryjournalisthatitpromisesveryfastpeerreview.Somebookchaptersandentiremonographsarepeer‐reviewedaswell.AcceptancerateThejournalmayormaynotprovidethisdata.Thechoosierthejournal,themorelikelyitwillsharethisonitswebsite.Externaltoolsformetrics‐‐subscriptiononly—availableatCUNYGraduateCenterandotherresearch‐intensivecolleges

Slide13:WebofScience

WebofScience:PublishedbyThomsonReuters(formerlyISI),providesthe“journalimpactfactor”JIForJournalImpactFactorissimplyajournal’s: Numberofcitations/numberofarticles

JIFtellsusonlysomuchabouttheactualqualityofthejournalandtellsusnothingaboutanindividualarticle’simpact.TheSanFranciscoDeclarationonResearchAssessmentorDORApushesagainstJIFandaimstoaddresstheresearchcommunity’sproblemswithevaluatingindividualoutputs.DORAwascreatedatanAmericanSocietyforCellBiologyconferencein2012.

JIFworksbestinthesciencesandsocialsciencesandforjournalsthatarewell‐establishedandwell‐known.

TheLondonSchoolofEconomicsblogpublishedagreatpostonthe“TheImpactFactorandItsDiscontents:ReadinglistoncontroversiesandshortcomingsoftheJournalImpactFactor.”

JIF,likeotherbibliometrics,canbegamed.OneparticularlyinfluentialarticlecaninflatetheJIFoftheoverallarticle.

Yes,disciplinemattersforJIF.RemembertheslidefromGreco’sresearch.

Slide14:Impactfactorsanddisciplines/Grecoredux

Slide15:SCOPUSScopus:usesH‐factormetrics

H‐factorisauthor‐focusedandtriestoreflectboththeproductivityandinfluenceofascholar.Itwascreatedtoassessarticlequalityratherthanjournalquality.Itquantifies:

‘xnumberofarticleshavebeencitedxnumberoftimes’

H‐Indexfavorspaperswithlargenumbersofauthors Itcanonlyincreaseovertime:itdoesn’treflectthe“half‐life”ofanarticle

Slide16:EIGENFACTORANDGOOGLESCHOLAR

EigenfactorThismetriccomputestheoverallimportanceofajournalovertimeusingcitations.Itusesafancyalgorithmtocomputethescore.

GoogleScholar

Reportshowmanytimesanarticlehasbeencitedbyotherauthors

IndexesbothtraditionallibrarydatabasesaswellasOpenAccessscholarlycontent

Authorscancreateprofilesforthemselves

Slide17:SCIMAGOandALTMETRICS

SCImago

PullsdatafromScopusandallowsforfairlygranularrankingofajournalbybothbroaddiscipline,sub‐disciplineandcountry.

UsesH‐factormetrics Cancomparejournals

Altmetrics Various,newish TypicallyforopenaccesscontentbutalsoincludedintraditionalvenueslikeScopus Forarticle‐levelmetrics,countsofdownloadsaswellasmentionsinsocialmediaandother

feedback:OpenAccessandnon‐OpenAccesscontent

Slide18:Caveatsandnewconsiderations

Publishersfeedbackloop:thepublishersthatownthedatabasesthatcalculatethe

publicationqualitymeasurementsalsopublishthejournalsthatarebeingranked Impactfactorsandothertraditionalmetricsarenotconsideredrelevantinalldisciplines Researchcanpointtoflawsoftraditionalmetrics Thetrueimpactofscholarshipcannotalwaysbequantified:scholarsarepushingagainst

bibliometrics,especiallytheideaofscholarshipforthepublicgood,the“justpublics”isverypowerful…JustPublics@365atCUNYGraduateCenter

Socialmediamatters:SeeBonnieStewart’sresearchonTwitterforscholarlycommunication

Mendeleyandotherplatformsasnewspacesforscholarstoshare Howdoyouevaluatenewscholarshipsuchasadigitalhumanitiesproject?Recenteventat

GradCenter,“Evaluating,Valuing,andPromotingDigitalScholarship,”includeddiscussionofhowtoevaluate,dopeerreview,andassessdigitalhumanitiescontent.

Newmedia:TheModernLanguageAssociationinparticularhasbeenforwardaboutevaluatingworkinthisarea.Otherinstitutions,suchastheOrganizationofAmericanHistoriansandtheNationalCouncilonPublicHistory,arealsoworkingonbroadeningthedefinitionofscholarlycommunicationandworkingoutmeansforevaluation

StefanieHausteinattheUniversitédeMontréalhasdonesignificantresearchonbibliometricsandaltmetricsconsideringdisciplinesaswellassocialmedia

Ihopetolearnmoreaboutnewmodesofscholarshipoutsideofthedigitalhumanities

Slide19BEAUBIENandECKARD’srubric,aresponsetopredatorypublishing.Thisisflawedbutagoodbeginningpoint.

Slide20:PredatoryPublishers

DoyougetspammyemailsaskingyoutopublishinajournalorpresentataconferencetoyourCityTechemailaddress?Thosearealmostalwaysfromapredatorypublisher.

Predatorypublishersareeasyandsleazy…Theyrepresentasmallsubsetofopenaccessjournals—analogoustovanitypublishers

Predatorypublisherspreyontheignorant,thedesperate,andtheduplicitous Theymayrequirearticleprocessingchargesbutoftennottransparent Themainsignals:

o Rapidpeerreview=nopeerreviewor“rubberstamping”o Unfocusedscopeforjournal:theEurasianJournalofScienceandSocialScienceso Toomanyarticlesbeingpublishedsimultaneouslyo Informationaboutjournalhascontradictionsorislacking

“White‐listing”isabettersolutionthanJeffreyBeall’slistwhichis“black‐listing” Scholarsshouldneverpublishinajournalwithoutcarefulscrutiny—importanttoread

somearticles,examinetheeditorialboard,learnmoreaboutthepublisher

Slide21:Summaryandconclusions

Assessingpublicationqualityisimportant,buttrueimpactmaynotbeeasilymeasurable Anarticleofpracticalrelevancemaynotbecitedfrequentlybutitmaybeusedoftenby

practitioners

Ultimately,weasscholarshavetoaskourselves, Whodowewantourresearchandscholarshiptoimpact? Andweaspeersandadministratorsneedtoaskthesamequestionsaswegothroughatimeofsubstantialchangeandfluxwheretraditionalmetricsmaynotsufficefornewmodesofscholarshipandachangingscholarlylandscape.

Imagesources:

Lavoie,Brian,EricChildress,RickyErway,IxchelFaniel,ConstanceMalpas,JenniferSchaffner,andTitiavanderWerf.2014.TheEvolvingScholarlyRecord.Dublin,Ohio:OCLCResearch.http://www.oclc.org/research/publications/library/2014/oclcresearch‐evolvingscholarly‐record‐2014.pdf

http://libguides.humboldt.edu/content.php?pid=457571&sid=3746574 https://c2.staticflickr.com/4/3091/2924868770_3caa81a1fd.jpg[tribes] http://www.jisc.ac.uk/sites/default/files/oa‐badges.jpg[openaccess] https://farm8.staticflickr.com/7034/6735929719_fa64ccda8f_o.jpg[peerreview] http://wokinfo.com/img/sem/wossem_home_lrghdr.png[webofscience] http://s3.amazonaws.com/libapps/accounts/26412/images/scopus_04_analyzeauthor_.p

ng[scopus] http://www.sparc.arl.org/sites/default/files/sparc‐alm‐primer.pdf[altmetrics] http://justpublics365.commons.gc.cuny.edu/[justpublicswebsite] https://www.flickr.com/photos/yparis/[predatory] https://farm5.staticflickr.com/4144/5068482201_7d0c4cbe15_o.jpg[conclusions]

Referencesandresources 

Eckard,Max,andSarahBeaubien."AddressingFacultyPublishingConcernswithOpenAccessJournalQualityIndicators."JournalofLibrarianshipandScholarlyCommunication2.2(2014).CenterforStudiesinHigherEducation.TheFutureofScholarlyCommunication.AccessedApril22,2015http://www.cshe.berkeley.edu/research/future‐scholarly‐communicationEigenfactor.org<http://www.eigenfactor.org/>Greco,A.N.,etal."TheStateofScholarlyJournalPublishing:1981‐2000."Journalofscholarlypublishing37.3(2006):155‐214.Haustein,Stefanie.GoogleScholarprofilepagewithlinkstoarticles<https://scholar.google.com/citations?user=kWhLvE8AAAAJ&hl=en>JournalofScholarlyPublishingavailableviaProjectMusehttp://citytech.ezproxy.cuny.edu:2048/login?url=http://muse.jhu.edu/cgi‐bin/resolve_openurl.cgi?issn=1198‐9742(requiresloginwithlibrarybarcodeoff‐campus)JustPublics@365<http://justpublics365.commons.gc.cuny.edu/>Lavoie,Brian,EricChildress,RickyErway,IxchelFaniel,ConstanceMalpas,JenniferSchaffner,andTitiavanderWerf.2014.TheEvolvingScholarlyRecord.Dublin,Ohio:OCLCResearch.<http://www.oclc.org/research/publications/library/2014/oclcresearch‐evolvingscholarly‐record‐2014.pdf.>

LondonSchoolofEconomicsandPoliticalScience.“TheImpactFactorandItsDiscontents:ReadinglistoncontroversiesandshortcomingsoftheJournalImpactFactor,”inTheImpactBlog,July30th,2014.<http://blogs.lse.ac.uk/impactofsocialsciences/2014/07/30/reading‐list‐on‐the‐journal‐impact‐factor/>SanFranciscoDeclarationonResearchAssessment.<http://www.ascb.org/dora‐old/files/SFDeclarationFINAL.pdf>SCImagoJournalandCountryRankingshttp://www.scimagojr.comStewart,Bonnie."InPublic:theShiftingConsequencesofTwitterScholarship."HybridPedagogy(2015).< http://www.hybridpedagogy.com/journal/in‐public‐the‐shifting‐consequences‐of‐twitter‐scholarship/>Suber,Peter.“Disciplinarydifferences relevant to open access.” <http://www.earlham.edu/~peters/fos/lists.htm#disciplines>.Tananbaum,Greg.Article‐levelmetrics:aSPARCprimer.2013.<http://www.sparc.arl.org/sites/default/files/sparc‐alm‐primer.pdf>