Google Scholar as a research and

evaluation toolevaluation tool

Álvaro Cabezas-Clavijo

INDEX

• Goals

• GS as a research tool– Search engine

– Alerts

– Updates

– Library + Cite

• GS as a tool for evaluating research• GS as a tool for evaluating research– GS Citations (or Profiles)

– GS Metrics

– Bibliometric analysis• Data manipulation

• Comparison to WOS/ Scopus

• Conclusions

�To show GS possibilities to do research

�To show GS possibilities to evaluate research

�To point out strengths and weaknesses of GS as a source to perform bibliometric analysis

GOALS

�To introduce Google Scholar as a valid source for carrying out research assessments (with caution!) in Social Sciences and Humanities (SSH)

GS AS A TOOL FOR RESEARCHERSGS AS A TOOL FOR RESEARCHERS

• GS is a widespread tool for doing research

• Comprehensive source of academic literature

• Widely known by scholars and students

• Ease of use (just like google)

GS AS A TOOL FOR RESEARCHERS

•• Free and fast

• Links to primary source (when it is freely available on the

web or can be accessed from our institution)

• Non-selective source (covers all material which meet the

technical requirements)

• No restrictions on language

• No restrictions on document type

• No discipline/areas bias

GS AS A TOOL FOR RESEARCHERS (II)

• No discipline/areas bias

• Estimated size is 100 million documents100 million documents (Web of Science

� 50 million; Scopus � 53 million)

GS AS A TOOL FOR RESEARCHERS (III)

• GS is not just an academic search engine but a “family” of tools for doing and evaluating research

GS AS A TOOL FOR RESEARCHERS (IV)

ALERTS UPDATES

LIBRARY

+ CITE

• Set up by user just clicking on the “Create alert” link when

performing a search

• Delivered by 2-4 days if there is fresh info to show

• Authors or thematic searches can be configured (but not for

journals)

ALERTS

journals)

1 2

3

• Configure an alert for citations to particular papers or authors

(in this case, if they have already set up a profile)

ALERTS

Following

Christian’s new

papers or

citations

• Alerts about papers of our interest automatically generated by Google

• You only need to set up a GS profile. Algorithm takes into account:– Who we cite in our papers

– Who cite us in their papers

UPDATES

– Who cite us in their papers

– Titles of our papers

– Co-authors

– Content

• A red bell indicates the number of new papers that might be of your interest

UPDATESSome paper recommendations

Ok, I have published about

these topics, so I may be

interested in these papers

Ok, this paper cites me so it should

be related to my research

• Kind of reference manager

• Save references in one click

• Arrange references by using labels

• Edit references & export them (but one by one!)

• Can’t import references from other bibliographic software

LIBRARY + CITE

• Can’t import references from other bibliographic software

•Cite easily from GS

results: formats �

MLA, APA, Chicago

•Get the formatted

reference and

copy/paste it into a

Bibliography.

• By default; 2 folders: My citations (actually My papers) and Cited by me

• You can set up other

LIBRARY + CITE

• You can set up other labels/folders and save new references from GS searches

GS as a tool for evaluating researchGS as a tool for evaluating research

• GS provides different tools which can help in assessments of individuals, journals and more

GS AS A TOOL FOR EVALUATORS (II)

PROFILES METRICS

SOURCE FOR

BIBLIOMETRIC ANALYSES

• Individual Web page with publication list and basic

bibliometric indicators

• Papers are ranked (by default) according to their number

of citations

• Also links to co-authors (if they have set up a profile) and

PROFILES

• Also links to co-authors (if they have set up a profile) and

keywordskeywordsRanking of researchers

by keywords/areas are

generated through this

classification

TIP: there is always a field where you

can be the most cited author! ☺

• Easy to set up and edit

• Strongly recommended for young researchers as it can

increase their visibility

• You will get an alert (maximum 3 times a week) every time

you are cited

PROFILES

• 72,579 researchers with profile (March 2012)

(Estimates by Ortega & Aguillo, 2013)

• GS profiles-derived products

– Some Spanish institutions are ranking their researchers

according to GSC (Univ. Granada, CSIC,…) � stimulus to set up

a profile?

COMMUNICATION: 886 RESEARCHERS

SCIENTOMETRICS: 265 RESEARCHERS

CSIC RANKING OF RESEARCHERS

ACCORDING TO GSC

• Easy to set up (just search for

your papers)

• Terrific tool for comparing

researchers within a field or

department

• Automatically updated basic

bibliometric data

PROFILES

• Data can be easily manipulated

(a researcher can self-claim

non-authored papers)

• Can stimulate vanity and ego

• Can generate no fair

comparisons (for example, bibliometric data

• No restrictions on source,

language or area

• Everyone can measure their

performance (and their

colleagues')

comparisons (for example,

researchers from different

areas in a single univ)

• Can generate unfair analysis by

non-bibliometric experts

STRENGTHS WEAKNESSES

• GS service devoted to measure journal’s impact according to their h-index (est. April 2012)

• Rankings are provided by language (nine languages) areas (8) and subcategories (313) (just for English-language journals) and journal title’s words (i.e, oncology)

• Subject/area rankings are limited to top 20 journals and language

METRICS

• Subject/area rankings are limited to top 20 journals and language rankings are limited to top 100 journals.

• Calculations take into account output for 5 years (currently 2008-2012) and citations received until july 2013. A new update is expected to be released in July 2014.

• Journals with fewer than 100 papers published in 5 years or with no citation (h-index=0) are excluded

• arXiv and other repositories’ collections are also included

• Free product to compare and rank journals

• We can get impact information about non-JCR journals and about national and SSH publications

• Transparency: citations for every paper that contributes to the h-index can be checked

• Methodological inconsistencies such as comparing journals from different areas

• Lack of proper bibliographic control (duplicates, “dirty” data,…)

• No selection criteria

• No action against data manipulation

METRICS

can be checked

• High correlation with JCR Impact factor (0.82)

• Simple to use and understand by any scholar

• Can be easily replicated

• No action against data manipulation

• No info about new updates (expected to be updated yearly)

• Just top results are presented

• H index favours journals with high production

STRENGTHS WEAKNESSES

• GS is estimated to contain about 100 million documents (87% of scholarly papers on the web) Khabsa & Giles (2014)

• So, it is the largest source for bibliometric analysis that we can use

SOURCE FOR BIBLIOMETRIC

ANALISYS

can use

• GS-based products can also be used to help with research assessment– Publish or Perish (software for bibliometric analysis)

– H-Index Scholar (ranking of spanish researchers)

– Google Chrome add-ons

– …

PUBLISH OR PERISH

H-INDEX SCHOLAR

GOOGLE CHROME ADD-ONS

• Many things to sort out (unknown coverage, duplicates, “dirty” data) but GS can be useful for:

�Non-covered areas by WoS…

�Non-English research…

SOURCE FOR BIBLIOMETRIC

ANALISYS

�Non-English research…

�…Which means basically Social Sciences and Humanities

�institutions and individuals which lack access to WoS/Scopus

GS data can be easily manipulated

• Just need to upload an “academic” paper to a institutional

repository, university webpage or similar

• “Academic” means a paper with title, authors, abstract and of

course, bibliographic references

• GS will count the citations from these fake papers, modifying • GS will count the citations from these fake papers, modifying

indicators of all cited scholars and journals.

Delgado, Robinson &

Torres (2014)

• Small diferences between rankings generated by GS, WoS and Scopus

• Intensive Care Medicine � Spearman correlation of 0.99 forjournals and 0.93 for top researchers

Comparison to Wos / Scopus

JOURNALSJOURNALS

Cabezas & Delgado (2013)

• Spearman correlation for Communication Journals: 0.895 (GSM-WoS); 0.879 (GSM-Scopus)

• Spearman correlation for Economics & Business Journals: 0.718 (GS-WoS)

JOURNALS

Delgado & Repiso (2013)

Harzing & der Wal (2009)

• Easy to use, fast and free

• Largest source for assessment

purposes

• Provides information about areas

and materials not covered by

• Lack of bibliographic control

• Data can be manipulated

• No possibilities to massive info

download

• Time-consuming, data needs to

SOURCE FOR BIBLIOMETRIC

ANALISYS

and materials not covered by

traditional bibliometric products

• Rankings are very similar to

those offered by WoS / Scopus

• Possibility to build new products

based on GS data

• Time-consuming, data needs to

be “cleaned”

• GS and their services could be

closed down in any moment

STRENGTHS WEAKNESSES

ConclusionsConclusions

• Bibliometricians need to find other products to complement WoS / Scopus-based evaluations (GS, ERIH, national products,…)

• What if subscriptions to traditional databases are cancelled?– Spain paying 4-6 million Euros to Thomson Reuters yearly for

national access to WoS from public research organizations

CONCLUSIONS

– Spain paying 4-6 million Euros to Thomson Reuters yearly for national access to WoS from public research organizations

• SSH deserve to be measured (prominent rankings such as Shanghai or Leiden ignore Humanities and a huge part of Social Sciences!)

• GS tools are far from perfect but provide us with some valuable information

• Bibliometricians take usually the “easy way”; only measure what can be easily measured with databases such as WoS or Scopus

• A challenge for bibliometricians is to get the most out of the existing tools to evaluate SSH (and to develop new ones!)

• GS can be used as a complementary or alternative source to

CONCLUSIONS

• GS can be used as a complementary or alternative source to shed some light on SSH

• Use with extreme caution, especially when assessing individuals

• Keep an eye on the future: Institutions might be demanding GS-based analysis soon.

Further readingsFurther readings

• Delgado-López-Cózar, E., & Cabezas-Clavijo, Á. (2013). Ranking journals: could Google Scholar Metrics be an alternative to Journal Citation Reports and Scimago JournalRank?. Learned publishing, 26(2), 101-113.

• Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google Scholar Experiment: how to index false papers and manipulate bibliometric indicators. Journal of the Association for Information Science and Technology, 65(3), 446-454.

• Harzing, A. W., & Van Der Wal, R. (2009). A Google Scholar h-index for journals: An

FURTHER READINGS

• Harzing, A. W., & Van Der Wal, R. (2009). A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41-46.

• Khabsa, M., & Giles, C. L. (2014). The Number of Scholarly Documents on the PublicWeb. PloS one, 9(5), e93949.

• Ortega, J. L., & Aguillo, I. F. (2013). Institutional and country collaboration in an online service of scientific profiles: Google Scholar Citations. Journal of Informetrics, 7(2), 394-403.

• BLOG: Google Scholar Digest: analysis of papers dealing with GS http://googlescholardigest.blogspot.com.es/

• TWITTER: Follow the #googlescholar hashtag

• SOFTWARE: Publish or Perish http://www.harzing.com/pop.htm

• PRODUCTS:

FURTHER MATERIALS

• PRODUCTS:

• GOOGLE SCHOLAR CITATIONS http://scholar.google.com/citations

• GOOGLE SCHOLAR METRICS http://scholar.google.com/citations?view_op=top_venues&hl=en

Google Scholar as a research and

evaluation toolevaluation tool

Álvaro Cabezas-Clavijo

Thank you!