Bibliometric Analysis in Social Sciences and Humanities:

paradigm shift and mutual challenges

João CostaInvestigação e Ensino na NOVA

June 29, 2012

OUTLINE

1. Bibliometrics in SSH: a paradigm shift and the response of the area.

2. The challenge taken by institutions and individuals (the case of FCSH).

3. The shortcomings of bibliometrics.4. The challenge for SSH.5. The challenge for bibliometrics.

A Paradigm ShiftCommon (and old) assumptions for evaluating science in SSH:-(Double-blinded) peer review as a standard;-Books and book chaptersBooks and book chapters are as important as journal articles;-Major journals edit monograph series monograph series for longer articles;-Major conferences edit booksbooks with selected papers;-Citation in any type of format signals impact;-Impact compared within areas and domains.

BOOKS are a major signal of academic achievement.CITATION signals impact.

A Paradigm ShiftNew assumptions/concepts:-Impact factors;-Journals with higher relevance than books or book chapters;-No clear place for “cultural” products;-Comparability across different areas.BOOKS lost relevance.MARKERS OF IMPACT suddenly changed.SSH suddenly perceived as areas with little impact.

The SSH response

- Perception that adherence to paradigm shift is condition for access to funding;

- Investment on diversification of publication types.

LEIDEN STUDY witnesses the growth of SSH in the relevant publication types.

The SSH response – the case of FCSH

Rapid increase and investment in publications indexed in CI (source: Converis)

The SSH response – the case of FCSH

- Institutional stimulus for investment in indexed publications:

Individual productivity prizes;Productivity prizes to I&D centers;Membership assessment based on CI

publications in certain I&D centers.

Clear assumption of challenge.

The shortcoming of bibliometrics

Leiden Study:

The shortcoming of bibliometrics

Coverage:

FCSH productivity in 2011(including: articles in peer-reviewed journals, books as author/editor, book chapters, peer reviewed conference proceedings, journal issue)

Total: 1170Indexed: 55 (4,7%)

The shortcoming of bibliometrics

Coverage and weakness in assessing impact (R. Santos):

Citation ind. No. References

No. Articles Total No. WoK % WoKSCI 200 8 107 6 663 82,2

AHCI+SSCI 300 12 683 4 353 34,3AHCI 150 7 111 3 546 49,9SSCI 150 5 572 807 14,5

Quadro 1. Numbers of cited references and proportion indexed in WoK.Source: Random sample of articles in WoS published in 2011.

The shortcoming of bibliometrics

Hiding relevant work:Three examples:

INET-ID: production of materials supporting the promotion of FADO to World Intangible Heritage by UNESCO.

The shortcoming of bibliometrics

Hiding relevant work:Three examples:

IELT: Collection of oral patrimony.

The shortcoming of bibliometrics

Hiding relevant work:Three examples:

CHAM: Underwater Archaeology

The shortcoming of bibliometrics Coverage: highly cited authorsBook citations in 2007 (source: Times Higher Education with Thomson)

Anthony Giddens (sociology) 1303Noam Chomsky (linguistics) 812Judith Butler (philosophy) 960Ulrich Beck (sociology) 733Jurgen Habermas (philosophy) 1049

The shortcoming of bibliometrics Coverage: highly cited authorsOccurrences and citations in WoSAnthony Giddens (sociology) 3 / 1Noam Chomsky (linguistics) 19/79Judith Butler (philosophy) 21/ 57Ulrich Beck (sociology) 0 / 0Jurgen Habermas (philosophy) 3/0

The challenge for SSH

- Precise identification of markers of research performance.

- Clear adherence to evaluation of research (peer-reviewing, etc.)

- Ranking of scientific products.

The challenge for bibliometry

- Development of specific tools for different research areas;

- Identification of correspondences across areas (e.g. what’s the SSH correspondent of a short article in Chemistry in terms of impact/work volume?)

The challenge for bibliometry

- Inclusion of domain-specific products (e.g. patrimony collections, materials for language policy, education materials, research-based cultural products, etc.)

The challenge for evaluation

- Make sure that evaluation takes into account the actual productivity instead of focusing on imposing a new typology of research products.

Conclusion: mutual challenges

- More impact, wider coverage and stronger international profile for the SSH will come about if bibliometrics is taken seriously, and if the specificity of SSH is taken seriously by bibliometrics.