dag w. aksnes

www.nifustep.noNIFU STEP Norwegian Institute for Studies in Innovation, Research and Education

7th euroCRIS strategic seminar, Brussels Recording Research

Use in research management and evaluations

Dag W. Aksnes


Background

NIFU STEP is a Norwegian research institute doing contract research, analyses and consultancy for various organisations such as ministries and research councils.

The main focus is on studies of education, research and innovation. One mission to provide knowledge for policy decisions within these areas and analyze the results and effects of policy through e.g., evaluations and studies on productivity.

This presentation will focus on how CRIS data have been used in Norway in relation to research management and evaluations.


CRIS-data Norway

Data on all types of scientific/scholarly publications in all fields of research in the Higher Education Sector are collected trough two documentation systems (Frida, ForskDOK).

In the database the publication output is reported by the institutions as ordinary bibliographic references.

A dynamic authority record of so far 19,000 controlled scientific and scholarly publication channels ensures that references to non-scientific publications are not entered into the system.

Publication data from professional bibliographic data sources (e.g. Web of Science) are imported to the documentation system in order to facilitate the registration of publications by the employees.


Relevance of bibliometics in science policy/evaluations

Number of publications used as an indirect measure of knowledge production

The extent to which the publications have been cited in the subsequent scientific literature used as an indicator scientific impact and international visibility.

Co-authorship used as an indicator of collaboration, e.g. the extent of international collaboration

The journal pattern used for obtaining information on the subfield distribution of the research activities and the extent of presence in leading journals

In turn, such indicators can yield much interesting information on various issues relevant for evaluation purposes.


Bibliometric analyses of research performance

Traditionally, bibliometric analyses of research performance have usually been based on the ISI-database (Web of Science, Thomson Reuters)

– Multidisciplinary– Structured data well adapted for bibliometric analyses– Citation counts– Covers mainly publications in international journals

In Norway the CRIS-database has now been used as an additional source for bibliometric analyses

The CRIS database is well suited for this purpose:– The bibliographic references in the documentation system are

standardized and analyzable by publication channel and type of publication.

– Complete, verifiable and structured data– Covers all scientific/scholarly publications not only journal articles. – Non-scientific/non-scholarly publications can be excluded.


Using CRIS data for funding purposes and analyses

In 2004, Norway implemented a bibliometric model for performance based budgeting of research institutions.

– The funding of these institutions is now partially based on the measurement of their scientific and scholarly publishing.

– The CRIS databases are used for this purpose.

Other applications: analyses of the research output and evaluations


Bibliometric analyses of research performance

Particularly the humanities and the social sciences the ISI-base covers a small proportion of the research output

– Due to poor coverage and a publication pattern where international journals are less important.

Example: publication types and ISI-coverage in major scientific and scholarly fields:


Figure 1. Publication types* in five major areas (N=24,154)

*) Weighted: 1 book = 5 articles


Figure 2. Publication types* in social science disciplines (N=5,616)



Research evaluations

In Norway, the Research Council of Norway regularly carries out evaluations of the national research activities in the various disciplines (peer evaluation by international panels).

NIFU STEP provides bibliometric analyses as background information for the evaluations.

Till quite recently all researchers encompassed by the evaluations have submitted their publication lists.

– Bibliometric analyses have been carried out using this as a data source in combination with ISI-data

– Time consuming – No structured data

Now the CRIS-database has instead been used as a data source – The researchers do not have to submit their publication lists anymore


”Level 1 and level 2”

In addition to using CRIS as the main data source, information on the “level” of the journals and book publishers are used in the analyses

In the Norwegian system journals and publishers are divided into two levels.

The highest level (level 2) is giving extra weight and includes only the leading and most selective international journals and publishers (account for about 20 % of the world’s publications),

The national councils in each discipline participate annually in determining and revising the highest level under the guidance of the Norwegian Association of Higher Education Institutions.


Examples

The following slides show some examples of indicators and analyses provided.

Levels of analyses– Country– Institutions– Departments– Divisions/research groups– (Individuals – anonymous presentation)


Figure 3. Number and proportion of articles in leading Chemistry journals – “level 2”, Norway 1998-2007.


0

25

50

75

100

125

150

175

200

225

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Nu

mb

er

of

art

icle

s

0 %

5 %

10 %

15 %

20 %

25 %

30 %

35 %

40 %

Pro

po

rtio

n

Number of articles - level 2 Proportion of articles - level 2


Figure 4. Number of publications per year, 2003-2007, fractionalised counts. Department X.

*) Weighted: 1 book = 5 articles0123456789

101112131415161718192021

2003 2004 2005 2006 2007

Nu

mb

er o

f p

ub

lica

tio

ns


Figure 5. Department x. Number of publications per person 2003-2007, fractionalised counts.

0

2

4

6

8

10

12

14

16

18

20

22

24

26

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

Persons

Nu

mb

er

of

pu

bli

cati

on

s


Table 1. Number of publications 2003-2007, Department X

UnitNumber of persons incl.

Publications - whole counts

Publications - fractional counts

Number of publications - fractional counts per person

TOTAL 24 276 85.3 3.6

Environment 9 127 40.6 4.5

Organic Chemistry 6 43 11.5 1.9

Physical Chemistry 9 112 33.2 3.7


Table 2. Journal profile, 2003-2007 publications (whole counts). Department X

Unit

Numb. of persons incl.

Numb. of articles - level 1

Numb. of articles - level 2

Prop. of articles - level 2

Avg. numb. of articles -level 2 per person

Avg. journal citation rate (impact factor)

TOTAL

24 181 79 30 % 3.3 4.6Environment 9 95 20 17 % 2.2 3.6Organic Chemistry 6 32 10 24 % 1.7 4.1Physical Chemistry 9 60 49 45 % 5.4 5.9


Table 3 The most frequently used journals, number of publications, 2003-2007, by groups/sections, Department X

Group/section JournalNumb. of articles

Journal citation rate (impact factor)*

Level

Electrochemistry

CORROSION SCIENCE 7 3.6 2JOURNAL OF THE ELECTROCHEMICAL SOCIETY 7 4.6

2

JOURNAL OF APPLIED ELECTROCHEMISTRY 3 2.6

1

Inorganic Chemistry

JOURNAL OF THE AMERICAN CERAMIC SOCIETY 13 2.8

2

JOURNAL OF LIGHT METALS 13 – 1SOLID STATE IONICS 9 4.5 1


Other examples

Analyses of the publication output at the University of Bergen

The University Library has initiated a project providing analyses of the publication output at the university based on CRIS data

Writing of reports being presented for the management at the university and departments

Examples of indicators provided


Figure 6. Number of publication points per researcher man year per department

00,10,20,30,40,50,60,70,80,9

11,11,21,31,4

An

tall

pu

blik

asjo

nsp

oe

ng


Proportion of publication points in ”level 2” channels

Enhet 2008 2007 2006 2005AverageMatematisk institutt

30 % 31 % 28 % 21 % 27 %

Institutt for informatikk

12 % 20 % 27 % 19 % 20 %

Institutt for fysikk og teknologi

38 % 28 % 39 % 33 % 34 %

Kjemisk institutt

46 % 68 % 49 % 45 % 52 %

Molekylærbiologisk institutt

44 % 44 % 1 % 7 % 24 %

Institutt for geovitenskap

26 % 35 % 28 % 26 % 29 %

Geofysisk institutt

11 % 25 % 14 % 20 % 17 %

Institutt for biologi

19 % 18 % 22 % 20 % 20 %

Total: 24 % 29 % 30 % 24 % 27 %


Conclusions

Data quality of CRIS crucial issue for using the data for these purposes– Accurate and complete– Delimitation of scientific/scholarly publications– Use of authority registers, person and publication IDs (incl. ISI-ID)– Importing of data from professional bibliographic data sources, e.g. Web of

science– In a format adapted for bibliometric analyses

The imported records have highest quality – although these data have imperfections as well

– Mistakes more often occur for the non-imported records registered by the researchers, e.g.

• Duplications• Editorials, corrections etc. registered as full articles• Mistakes in the title and journal title• Lacking information on co-authors

dag w. aksnes

Documents