NBER WORKING PAPER SERIES
BIAS AGAINST NOVELTY IN SCIENCE:
A CAUTIONARY TALE FOR USERS OF BIBLIOMETRIC INDICATORS
Jian Wang
Reinhilde Veugelers
Paula Stephan
Working Paper 22180
http://www.nber.org/papers/w22180
NATIONAL BUREAU OF ECONOMIC RESEARCH
1050 Massachusetts Avenue
Cambridge, MA 02138
April 2016
Earlier versions of this paper were presented at the Workshop on the Organization, Economics
and Policy of Scientific Research, Turin; Institute for Research Information and Quality
Assurance, Berlin; Max Planck Institute for Innovation and Competition, Munich; and the TIES
seminar at the MIT Sloan School, Cambridge. The authors thank seminar participants and in
particular Pierre Azoulay, Christian Catalini, Paul David, Lee Fleming, Alfonso Gambardella,
Dietmar Harhoff, Sybille Hinze, Stefan Hornbostel, Jacques Mairesse, Fabio Montobbio, Henry
Sauermann, Daniel Sirtes, Scott Stern, Mark Veugelers, and John Walsh for helpful comments.
Financial support from KU Leuven (GOA/12/003) and the Research Foundation - Flanders
(FWO, G.0825.12) is gratefully acknowledged. J. Wang also gratefully acknowledges a
postdoctoral fellowship from FWO. Publication data are sourced from Thomson Reuters Web of
Science Core Collection. The views expressed herein are those of the authors and do not
necessarily reflect the views of the National Bureau of Economic Research.
NBER working papers are circulated for discussion and comment purposes. They have not been
peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies
official NBER publications.
© 2016 by Jian Wang, Reinhilde Veugelers, and Paula Stephan. All rights reserved. Short
sections of text, not to exceed two paragraphs, may be quoted without explicit permission
provided that full credit, including © notice, is given to the source.
Bias against Novelty in Science: A Cautionary Tale for Users of Bibliometric Indicators
Jian Wang, Reinhilde Veugelers, and Paula Stephan
NBER Working Paper No. 22180
April 2016
JEL No. I23,O31,O33,O38
ABSTRACT
Research which explores unchartered waters has a high potential for major impact but also carries
a higher uncertainty of having impact. Such explorative research is often described as taking a
novel approach. This study examines the complex relationship between pursuing a novel
approach and impact. Viewing scientific research as a combinatorial process, we measure
novelty in science by examining whether a published paper makes first time ever combinations of
referenced journals, taking into account the difficulty of making such combinations. We apply
this newly developed measure of novelty to all Web of Science research articles published in
2001 across all scientific disciplines. We find that highly novel papers, defined to be those that
make more (distant) new combinations, deliver high gains to science: they are more likely to be a
top 1% highly cited paper in the long run, to inspire follow on highly cited research, and to be
cited in a broader set of disciplines. At the same time, novel research is also more risky, reflected
by a higher variance in its citation performance. In addition, we find that novel research is
significantly more highly cited in “foreign” fields but not in its “home” field. We also find strong
evidence of delayed recognition of novel papers and that novel papers are less likely to be top
cited when using a short time window. Finally, novel papers typically are published in journals
with a lower than expected Impact Factor. These findings suggest that science policy, in
particular funding decisions which rely on traditional bibliometric indicators based on short-term
direct citation counts and Journal Impact Factors, may be biased against “high risk/high gain”
novel research. The findings also caution against a mono-disciplinary approach in peer review to
assess the true value of novel research.
Jian Wang
KU Leuven
MSI & ECOOM
Naamsestraat 69
3000 Leuven, BELGIUM
jian.wang@kuleuven.be
Reinhilde Veugelers
KU Leuven
MSI & ECOOM
Naamsestraat 69
3000 Leuven
BELGIUM
and Bruegel and CEPR
reinhilde.veugelers@econ.kuleuven.be
Paula Stephan
Department of Economics
Andrew Young School of Policy Studies
Georgia State University
Box 3992
Atlanta, GA 30302-3992
and NBER
pstephan@gsu.edu
1
1. Introduction
Scientific breakthroughs advance the knowledge frontier. Research underpinning breakthroughs
often is driven by novel approaches. While research that takes a novel approach has a higher
potential for major impact, it also faces a higher level of uncertainty of impact. In addition, it
may take longer for novel research to have a major impact, either because of resistance from
incumbent scientific paradigms (Kuhn, 1962; Merton, 1973; Planck, 1950) or because of the
longer time required to incorporate the findings of novel research into follow-on research
(Garfield, 1980; Wyatt, 1975). The “high risk/high gain” nature of novel research makes it
particularly appropriate for public support (Arrow, 1962). Delayed recognition may, however,
lead novel research to be undervalued in research evaluations which use classic bibliometric
indicators based on short term citations.
This bias in classic bibliometric indicators against novel research, to the extent it exists, is of
concern given the increased reliance funding agencies and hiring institutions place on
bibliometric indicators to aid in decision making and performance evaluation (Butler, 2003;
Hicks, 2012; Hicks, Wouters, Waltman, de Rijcke, & Rafols, 2015; Martin, 2016; Monastersky,
2005). Such heavy reliance may explain in part the perception that funding agencies and their
expert panels are increasingly risk-averse and the charge that competitive selection procedures
encourage relatively safe projects, which exploit existing knowledge, at the expense of novel
projects, that explore untested approaches (Alberts, 2010; Azoulay, Graff Zivin, & Manso, 2011;
Kolata, 2009; NPR, 2013; Petsko, 2012; Walsh, 2013).
The goal of this paper is to develop a measure of novel research and compare the citation profile
of novel research with that of non-novel research as well as the Impact Factor of the journals in
which novel research is published. We are particularly interested in whether characteristics of
novel research match the “high risk/high gain” profile associated with breakthrough research and
whether bibliometric measures are biased against novel research. To this end, we define research
that draws on new combinations of knowledge components as novel and develop an ex ante
measure of combinatorial novelty at the paper level, where novelty is operationalized as making
new combinations in referenced journals. Utilizing this newly-minted measure of novelty, we
explore the complex relationship between novelty and citation impact, using the life-time citation
2
trajectories of 773,311 research articles across all scientific disciplines published in 2001 and
indexed in the Web of Science (WoS), as well as the profile of papers citing them.
We find novel papers to have a larger variance in their citation distribution and be more likely to
populate both the tail of high impact as well as the tail of low impact, reflecting their “high risk”
profile. At the same time, novel papers also display a “high gain” characteristic: They have a
much higher chance of being a top cited paper in the long run, a higher likelihood of stimulating
follow-on top cited research, and a broader impact transcending disciplinary boundaries and
reaching more scientific fields. We further scrutinize the impact profile of novel research and
uncover intriguing characteristics associated with novelty. First, we distinguish between impact
in “home” and “foreign” fields and find that novel papers are significantly more likely to be
highly cited in foreign fields but not in their home field. Second, an examination of time
dynamics in the citation accumulation process reveals delayed recognition for novel research.
Specifically, although novel papers are highly cited in the long run, they are less likely to be top
cited in the short run. We also find that novel papers are less likely to be published in high
Impact Factor journals. These findings suggest that over-reliance on standard bibliometric
metrics, in particular Journal Impact Factor and short-term citation counts, may bias against
novel research.
2. Combinatorial novelty in science
We view the process of research as one of puzzle solving, whereby researchers work with pieces
of knowledge and combine them to generate new scientific knowledge. Some of these existing
knowledge pieces are embedded in the literature, some in equipment and materials, which
themselves are embedded in the literature, and others in the tacit knowledge of individuals
engaged in the research. Using knowledge pieces in well-understood ways corresponds to a
search process labeled as exploitation. Using existing knowledge pieces in new ways
corresponds to an explorative search process, which is more likely to lead to major
breakthroughs but also comes with a substantial risk of no or low impact (March, 1991). From
this perspective, novel research is more closely associated with exploration.
Drawing on this combinatorial perspective of the research process, novelty can be defined as the
recombination of pre-existing knowledge components in an unprecedented fashion. This
3
combinatorial view of novelty has been embraced by scholars in various disciplines (Arthur,
2009; Burt, 2004; Mednick, 1962; Schumpeter, 1939; Simonton, 2004; Weitzman, 1998). For
example, Nelson and Winter (1982) state that “the creation of any sort of novelty in art, science
or practical life – consists to a substantial extent of a recombination of conceptual and physical
materials that were previously in existence.” Romer (1994) and Varian (2009) also argue that
new combinations of existing components provide a potentially huge source of important new
discoveries. The ability to make new combinations of existing knowledge pieces is one reason
that “outsiders” from other disciplines arguably can provide exceptional insights when they
move from one field to another, as physicist Leo Szilard did, when he switched from physics to
biology in the 1950s (Carroll, 2013, p. 352).
It is important to note that not all breakthroughs result from combinatorial novelty, and certainly
not all novel research leads to breakthroughs. It is also important to note that there is strong
anecdotal evidence that research of a novel nature not only has the potential to become a
breakthrough but also contributes to subsequent breakthroughs. The diagrams that Feynman
produced in the late 1940s provided physicists with an entirely new way of understanding the
behavior of subatomic particles and, according to the historian of physics David Kaiser, “have
revolutionized nearly every aspect of theoretical physics” (Kaiser, 2009, p. 4). The creation of
transgenic and knockout mice in the late 1980s revolutionized research on any number of
diseases. Or, consider the research of Sebastian Seung that has received considerable attention
and aims at mapping the human brain, something that no one to date has done (Cook, 2015).
Seung’s course is heavily influenced by applying a method described in a highly-cited paper
published in PloS BIOLOGY (Denk & Horstmann, 2004) that used a novel approach in human
connectome (cf infra).
The combinatorial view of novelty has been studied in the technological invention literature and
operationalized using patent information. Fleming (2001) takes the technology subclasses in
which patents are classified as representing the components of technological know-how.
Fleming (2007) looks at the occurrence of new combinations of subclasses to which a patent is
assigned and uses this as a measure of the generative creativity of inventors of the patent.
Verhoeven, Bakker, and Veugelers (2016) combine this combinatorial novelty measure with a
measure of novelty in technological and scientific knowledge origins, based on whether the focal
