kempf, sondergeld: indicator-based monitoring of an interdisciplinary field of science

Indicator-Based Monitoring of an

Interdisciplinary Field of Science The Example of Educational Research

Dr. Andreas Oskar Kempf, M.A.(LIS)Leibniz-Institute for the Social Sciences (GESIS)

Ute Sondergeld, M.A., Wiss. Dok.German Institute for International Educational Research (DIPF)

Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research

Contents

1. Project Overview

2. Project Context and Objective

„Empirical Turn“ in Educational Research

New Governance Model in Research Funding

Data Sources

3. Research Indicators

4. Findings

4.1 Analysis of Project Data

4.2 Analysis of Publication Data

5. Summary

6. Outlook


1. Project Overview Title:

Development and Changing Dynamics of a Heterogeneous

Social Science Field. Using the Example of Educational Research

Duration:

05/2011 - 04/2014

Funding:

Leibniz Association (SAW funding measure)

Project Lead:

German Institute for International Educational Research (DIPF)

Project cooperation between:


2. Project Context and Objective

„Empirical Turn“ in Educational Research

DFG Research Units / Priority Programmes for Empirical

Education Research (2000/2002)

Introduction of National Educational Standards (2003)

Agreement on a National Report on Education (2004)

Framework Programme for the Promotion of Empirical

Educational Research (BMBF) (2007)


Project Context

New Governance Model in Research Funding

Basic and third party funding in higher education institutions in Germany over time


Inco

me

(in

bn€

)

Rat

e o

fth

ird

par

tyfu

nd

ing

(%)

Project Objective

Analysis of the structure and development of and within educational

research

Analysis of communication processes within educational research


EducationalResearch

Research projects

Indicators

PublicationoutputCitations

Data SourcesProject Data Sources:

- SOFIS (GESIS) - BMBF Framework Programme

Coverage: 9139 projects Period: 1995-2009 (end of project)

Publications Random sample of 270 projects of

the core disciplines: educationalsciences, psychology, sociology; periods: 1995-97, 2006-08

Data collection of research output Coverage: about 2.000 publications

Analysis of Reception Web of Science, Scopus, Google

Scholar


3. Types of Research Indicators

Hinze/Glänzel (2012)


Third party funding

Hinze/Glänzel (2012) based upon Hornbostel1999: p. 59.

4.1 Analysis of Project Data


Cologne, May 28 – 31 May │Kempf, Sondergeld │ IASSIST

Research Activity: Core Disciplines Over Time (2/2)

Decline in psychological projects


61,5%

66.4%

12.6% 9.7%

8.2% 10.0%0%

10%

20%

30%

40%

50%

60%

70%

80%

1995-1997 1998-2000 2001-2003 2004-2006 2007-2009

Shar

e

Period

Educational sciences

Psychology

Sociology

n = 9139

Cologne, May 28 – 31 May │Kempf, Sondergeld │ IASSIST

Funding Bodies Over Time (2/2)

ResultsThere, indeed, is an increase in the share of funding byfoundations and the EU. Differences in funding sourcesdiminish.Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research

0%

5%

10%

15%

20%

25%

1995-1997 1998-2000 2001-2003 2004-2006 2007-2009

Shar

e

Period

Fed. states

Fed. gov.

DFG

Foundations

EU

Third Party Funding (n = 5915)

Hypothesis

Increase in the share of funding by foundations and the EU.

Research Cooperation Over Time (1/2)

Results

Networking among research institutions increases


7.20%

9.40%

13.80%

17.80%

18.60%

0%

5%

10%

15%

20%

1995-1997 1998-2000 2001-2003 2004-2006 2007-2009

Shar

e

Period

Percentage of Cooperation Projects over Time (n = 7260)

Hypothesis

A growth in national and international cooperations

Research Cooperation Over Time (2/2)

Results

The share of cooperations is higher in third party

funded than in self-financed projects.


9.90%

17.40%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

20.00%

Self-Financed Projects Third Party Funded Projects

Shar

e

Funding Type of Projects

Funding Types in Cooperation Projects (n = 7260)

Hypothesis

A higher proportion of cooperation projects in third party

funded than in in-house projects

Empirical Research Methods

Results

There is an increase in the use of empiricalmethods between the last two time blocks.


74.5%

71.5% 69.4%64.3%

67.1%

25.5% 28.5% 30.6% 35.7%

32.9%

0%

10%

20%

30%

40%

50%

60%

70%

80%

1995-1997 1998-2000 2001-2003 2004-2006 2007-2009

Shar

e

Period

empirical

non-empirical

Use of Research Methods over Time (n = 9139)

Hypothesis

Increase in the use of empirical research methods

4.2 Analysis of Publication Data


4.2 Analysis of Publication Data – First Results

Hinze/Glänzel (2012)


Third party funding

Hinze/Glänzel (2012) based upon Hornbostel1999: p. 59.

Publication Activity


Results

The number of publications in total declines.

Likewise, discipline-specific differences seem to decline.

Hypothesis An increase in the number of publications.

412

278

325

327

268

315

1005

920

0

200

400

600

800

1000

1200

1995-1997 2006-2008

Co

un

t

Period

Sociology

Educational Sciences

Psychology

In Total

Number of Publications (n = 1925)

Discipline-Specific Publication Types (1/3)


Hypothesis

A greater number of journal articles in psychology than in sociology or

in educational sciences.

Results

In psychology there is a preference for publishing in journal

articles. Over time, a significant increase in the number of journal

articles is measurable in educational sciences. In sociology the

number of journal articles decreases significantly.

0.33

0.22

0.23

0.35

0.34 0.39

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

1995-1997 2006-2008

EMM

Period

Sociology


Psychology

Estimated Marginal Means (EMM) of Journal Articlesfor each Discipline (n = 535)

Publication Types (2/3)Peer-Review Status of Journal Articles


Hypothesis

A greater number of publications with peer-review status in psychology

Results

In general psychological projects publish more often in peer-

reviewed journals than the other disciplines do.

Over time we measured a significant decrease in sociological and

a significant increase in educational projects.

45.20%

20.50%

12.30%

31.10%42.60%

48.30%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

1995-1997 2006-2008

Shar

e

Period

Sociology


Psychology

Percentage of Journal Articles with Peer-Review Status (n = 306)

Discipline-Specific Publication Types (3/3)


0.28

0.33

0.48

0.26

0.3

0.2

0

0.1

0.2

0.3

0.4

0.5

0.6

1995-1997 2006-2008

EMM

Period

Sociology


Psychology

Estimated Marginal Means (EMM) of Monographs for each Discipline (n=558)

Hypothesis

A greater number of monographs in sociology and educational

sciences than in psychology

Results

Monographs are more often published in educational sciences and

in sociology than in psychology. In psychology and in educational

sciences publishing in monographs decreases significantly. In

educational sciences this decrease is most pronounced.

Multi-Authorship


Results

Over time, the number of authors in total per publication

grows significantly due to a significant increase in

psychology and in educational sciences.

Hypothesis

An increase in the number of authors per publication

1.8

2.1

1.71.7

1.5

2

2.2

2.7

0

0.5

1

1.5

2

2.5

3

1995-1997 2006-2008

Co

un

ts

Period

Number of Authors in Total

Sociology


Psychology

Estimated Marginal Means (EMM) of Authorsper Publication (n = 1925)

International Character of Publications

Cologne, May 28 – 31 May │ Kempf, Sondergeld │ IASSIST 2013 │ Monitoring Educational Research Page 25

9.90%

14.20%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

1995-1997 2006-2008

Shar

e

Period

Percentage of Publications Written in English in Total (n = 1925)

Hypothesis

Number of publications written in English increases.

ResultsThe number of publications written in English increases significantly.

Comparing the three core-disciplines, psychological publications havethe highest share of publications written in English.

10.00%

6.40%

20.40%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

Sociology Educational Sciences Psychology

Shar

e

Discipline

Sociology


Psychology

Discipline-Specific Differences in English Publication(1995-07 + 2006-08) (n = 1925)

5. SummaryResearch projects in educational research are characterized by:

- a growing importance in the Social Sciences,

- an increase in third-party funding,

- a growing diversity in funding sources,

- an increase in networking activities among research institutes,

- a slow increase in the use of empirical research methods since the year2005.

Publishing in educational research is characterized by:

- an increase in networking of authors,

- an internationalization effect over time,

- disciplinary differences; over time publishing in educational researchseems to align to psychological publication behaviour.


6. Outlook

- Further analysis on the interrelation

between research projects and publications,

- Thematic analysis of projects,

- Further analysis of publication data,

- Reception analysis of publications,

- Indicator specification.


AcknowledgementsAlexander Botte

Ute Sondergeld

Jürgen Wiesenhütter

Dr. Gabriel Schui

Katja Singleton

Veronika Kuhberg-Lasson

Dr. Sybille Hinze

Valerie Aman

Marion Schmidt

Peter Mutschke

Dr. Philipp Mayr

Karima Haddou ou Moussa

Dr. Andreas Oskar Kempf


Thank you

for your attention!

Contact:

Dr. Andreas Oskar Kempf, M.A.(LIS)Leibniz-Institute for the Social Sciences (GESIS)

[email protected]

Ute Sondergeld, M.A., Wiss. Dok.German Institute for International Educational Research (DIPF)

[email protected]


Bibliography

Deutsche Forschungsgemeinschaft (2012) Förderatlas 2012. Kennzahlen zur öffentlich finanzierten Forschung in Deutschland. Weinheim.URL: http://www.dfg.de/download/pdf/dfg_im_profil/evaluation_statistik/foerderatlas/dfg-foerderatlas_2012.pdf (20.05.2013)

Hinze, Sybille/Glänzel, Wolfgang (2012) Scientometric Indicators in Use: An Overview. Präsentation im Rahmen der European Summer School forScientometrics (ESSS), Leven, Juli 2012.URL: http://www.scientometrics-school.eu/images/2012Hinze.pdf (25.05.2013)

Hornbostel, Stefan (1999): Welche Indikatoren zu welchem Zweck: Input, Throughput, Output. In: Röbbecke, Martina/Simon, Dagmar (Hrsg.): Qualitätsförderung durch Evaluation? Ziele, Aufgaben und Verfahren von Forschungsbewertungen im Wandel. Dokumentation des Workshops vom 20. und 21. Mai 1999. Wissenschaftszentrum Berlin für Sozialforschung. URL: http://skylla.wzb.eu/pdf/1999/p99-003.pdf (23.05.2013)


kempf, sondergeld: indicator-based monitoring of an interdisciplinary field of science

Technology

educational research

educational sciences

research fundingbasic

research activity

research acitivity

sondergeld iassist

empirical education

indicatorbased monitoring