Filetype PDF | Posted on 21 Sep 2022 | 4 years ago
Authentication
digital resources
312x Filetype PDF File size 0.41 MB Source: www.ncbi.nlm.nih.gov
Education and debate
Whythe impact factor of journals should not be used for
evaluating research
Per O Seglen
Institute for Studies Evaluating scientific quality is a notoriously difficult
in Research and problem which has no standard solution. Ideally, pub-
Higher Education lished scientific results should be scrutinised by true Summary points
(NIFU),
Hegdehaugsveien experts in the field and given scores for quality and • Use of journal impact factors conceals the
31, N-0352 Oslo, quantity according to established rules. In practice, difference in article citation rates (articles in the
Norway most cited half of articles in a journal are cited
Per O Seglen, however, what is called peer review is usually
professor performed by committees with general competence 10timesasoftenastheleastcitedhalf)
ratherthanwiththespecialist’sinsightthatisneededto • Journals’ impact factors are determined by
BMJ 1997;314:498–502 assess primary research data. Committees tend, there- technicalities unrelated to the scientific quality
fore, to resort to secondary criteria like crude of their articles
publication counts, journal prestige, the reputation of • Journal impact factors depend on the
authors and institutions, and estimated importance research field: high impact factors are likely in
1 journals covering large areas of basic research
and relevance of the research field, making peer
23 with a rapidly expanding but short lived
review as much of a lottery as of a rational process.
On this background, it is hardly surprising that literature that use many references per article
alternative methods for evaluating research are being • Article citation rates determine the journal
sought, such as citation rates and journal impact impact factor,not vice versa
factors, which seem to be quantitative and objective
indicators directly related to published science. The
5
citation data are obtained from a database producedby academic positions. In the Nordic countries, journal
the Institute for Scientific Information (ISI) in impact factors have, on occasion, been used in the
Philadelphia, which continuously records scientific evaluation of individuals as well as of institutions and
citations as represented by the reference lists of articles have been proposed, or actually used, as one of the
from a large number of the world’s scientific journals. premises for allocation of university resources and
167
Thereferences are rearranged in the database to show positions. Resource allocation based on impact fac-
8
howmanytimeseachpublicationhasbeencitedwithin tors has also been reported from Canada and
9
a certain period, and by whom, and the results are Hungary and, colloquially, from several other coun-
published as the Science Citation Index (SCI). On the tries. The increasing awareness of journal impact
basis of the Science Citation Index and authors’ factors, and the possibility of their use in evaluation, is
publication lists, the annual citation rate of papers by a already changing scientists’ publication behaviour
scientific author or research group can thus be towards publishing in journals with maximum
910
calculated. Similarly, the citation rate of a scientific impact, often at the expense of specialist journals
journal—known as the journal impact factor—can be that might actually be more appropriate vehicles for
calculated as the mean citation rate of all the articles the research in question.
4 Given the increasing use of journal impact
containedinthejournal. Journalimpactfactors,which
factors
are published annually in SCI Journal Citation Reports, —as well as the (less explicit) use of journal
are widely regarded as a quality ranking for journals prestige—in research evaluation, a critical examination
andusedextensivelybyleadingjournalsintheiradver- of this indicator seems necessary (see box).
tising.
Since journal impact factors are so readily Is the journal impact factor really
available, it has been tempting to use them for evaluat- representative of the individual journal
ing individual scientists or research groups. On the articles?
assumptionthatthejournalisrepresentativeofits arti-
cles, the journal impact factors of an author’s articles Relation of journal impact factor and citation rate
cansimplybeaddeduptoobtainanapparentlyobjec- of article
tive and quantitative measure of the author’s scientific For the journal’s impact factor to be reasonably
achievement. In Italy, the use of journal impact factors representative of its articles, the citation rate of
was recently advocated to remedy the purported individual articles in the journal should show a narrow
subjectivity and bias in appointments to higher distribution,preferably a Gaussian distribution,around
498 BMJ VOLUME314 15FEBRUARY1997
Education and debate
Problems associated with the use of 100
journal impact factors 90
• Journal impact factors are not statistically 80
representative of individual journal articles 70
• Journal impact factors correlate poorly with actual
citations of individual articles % Of journal's citation rate60
• Authors use many criteria other than impact when
submitting to journals 50
• Citations to “non-citable” items are erroneously
included in the database 40
• Self citations are not corrected for 30
• Review articles are heavily cited and inflate the
impact factor of journals 20
• Long articles collect many citations and give high
journal impact factors 10
• Shortpublication lag allows many short term journal 0
self citations and gives a high journal impact factor 0102030405060708090100
• Citations in the national language of the journal are % Of journal articles (in 5% categories by annual citation rate)
preferred by the journal’s authors
• Selective journal self citation: articles tend to Fig 2 Cumulative contribution of articles with different citation
preferentially cite other articles in the same journal rates (beginning with most cited 5%) to total journal impact.
• Coverageofthedatabaseisnotcomplete Values are mean (SE) of journals in fig 1; dotted lines indicate
• Booksarenotincludedinthedatabaseasasourcefor contributions of 15% and 50% most cited articles11
citations
• Database has an English language bias
• Database is dominated by American publications
• Journal set in database may vary from year to year The uneven contribution of the various articles to
• Impact factor is a function of the number of the journal impact is further illustrated in figure 2: the
references per article in the research field
• Research fields with literature that rapidly becomes cumulative curve shows that the most cited 15% of the
obsolete are favoured articles account for 50% of the citations, and the most
• Impact factor depends on dynamics (expansion or cited 50% of the articles account for 90% of the
contraction) of the research field citations. In other words, the most cited half of the arti-
• Small research fields tend to lack journals with high
impact cles are cited, on average, 10 times as often as the least
• Relations between fields (clinical v basic research, for cited half. Assigning the same score (the journal
example) strongly determine the journal impact factor impact factor) to all articles masks this tremendous
• Citation rate of article determines journal impact,but difference
not vice versa —which is the exact opposite of what an
evaluation is meant to achieve. Even the uncited
articles are then given full credit for the impact of the
the mean value (the journal’s impact factor). Figure 1 few highly cited articles that predominantly determine
shows that this is far from being the case: three differ- the value of the journal impact factor.
ent biochemical journals all showed skewed distribu- Since any large, random sample of journal articles
tions of articles’ citation rates, with only a few articles will correlate well with the corresponding average of
11 12
anywherenearthepopulationmean. journal impact factors, the impact factors may seem
reasonably representative after all. However, the corre-
lation between journal impact and actual citation rate
600 of articles from individual scientists or research groups
Biochimica Biophysica Acta 912
Biochemical Journal is often poor (fig 3). Clearly, scientific authors do not
Journal of Biological Chemistry necessarily publish their most citable work in journals
500 of the highest impact, nor do their articles necessarily
No of articles cited match the impact of the journals they appear in.
400 Although some authors may take journals’ impact fac-
tors into consideration when submitting an article,
other factors are (or at least were) equally or more
300 important,suchasthejournal’ssubjectareaanditsrel-
evance to the author’s specialty, the fairness and rapid-
ity of the editorial process, the probability of
200 acceptance,publication lag, and publication cost (page
13
charges).
100 Journal impact factors are representative only
when the evaluated research is absolutely average
(relative to the journals used), a premise which really
0 makes any evaluation superfluous. In actual practice,
however, even samples as large as a nation’s scientific
≤1 2-3 4-5 6-7 8-9 30 output are far from being random and representative
10-1112-13 14-15 16-1718-1920-2122-2324-2526-2728-29 ≥
No of citations per article per year of the journals they have been published in: for exam-
Fig 1 Citation rates in 1986 or 1987 of articles published in three ple, during the period 1989-93, articles on general
biochemical journals in 1983 or 1984, respectively11 medicine in Turkey would have had an expected
citation rate of 1.3 (relative to the world average) on the
BMJ VOLUME314 15FEBRUARY1997 499
Education and debate
articles. If correction were made for article length,
10 r=0.05 10 r=0.27 “communications” journals like Biochemical and Bio-
physical Research Communications and FEBS Letters
5 5 wouldgetimpactfactorsashighas,orhigherthan,the
high impact journals within the field, like Journal of
20 21
2 2 Biological Chemistry.
Theuseofanextremelyshorttermindex(citations
1 1 to articles published only in the past two years) in cal-
No of citations per article per year culating the impact factor introduces a strong
0.5 0.5 temporalbias,withseveralconsequences.Forexample,
articles in journals with short publication lags will con-
0.2 0.2 tain relatively many up to date citations and thus con-
tribute heavily to the impact factors of all cited
0 0 journals. Since articles in a given journal tend to cite
22
r=0.44 r=0.63 articles from the same journal, rapid publication is
10 50 self serving with respect to journal impact, and signifi-
23
5 20 cantly correlated with it. Dynamicresearchfields with
high activity and short publication lags, such as
10 biochemistry and molecular biology, have a corre-
2 5 spondingly high proportion of citations to recent
publications
—andhencehigher journal impact factors
1 2 23 24
No of citations per article per year —than, for example, ecology and mathematics.
1 Russian journals, which are cited mainly by other
0.5 25
0.5 Russian journals, are reported to have particularly
long publication lags, resulting in generally low impact
0.2 26
0.2 factors. Pure technicalities can therefore account for
0 0 several-fold differences in journal impact.
0.5 1 2 5 10 0.5 1 2 5 10 20 Limitations of the database
Impact factor of journal Impact factor of journal
Fig 3 Correlation between article citation rate and journal impact for four authors12 The Science Citation Index database covers about
8
3200 journals ; the estimated world total is about
27
126000. The coverage varies considerably between
basis of journal impact, but the actual citation was only research fields: in one university, 90% of the chemistry
14 faculty’s publications, but only 30% of the biology fac-
0.3. The use of journal impact factors can therefore
28
beasmisleadingforcountries as for individuals. ulty’s publications, were in the database. Since the
impactfactorofanyjournalwillbeproportionaltothe
Journal impact factors are calculated in a way that database coverage of its research field, such discrepan-
causes bias cies mean that journals from an underrepresented
Apart from being non-representative, the journal field that are included will receive low impact factors.
impact factor is encumbered with several shortcom- Furthermore, the journal set in the database is not
ings of a technical and more fundamental nature. The constant but may vary in composition from year to
factor is generally defined as the recorded number of year.24 29 In many research fields a substantial fraction
citations within a certain year (for example, 1996) to of scientific output is published in the form of books,
the items published in the journal during the two pre- whicharenotincludedassourceitemsinthedatabase;
ceding years (1995 and 1994), divided by the number 30
they therefore have no impact factor. In mathematics,
of such items (this would be the equivalent of the aver- leading publications that were not included in the Sci-
age citation rate of an item during the first and second ence Citation Index database were cited more
calendar year after the year of publication). However, frequently than the leading publications that were
the Science Citation Index database includes only nor- 31
included. Clearly, such systematic omissions from the
mal articles, notes, and reviews in the denominator as databasecancauseseriousbiasinevaluationsbasedon
citable items, but records citations to all types of docu- impact factor.
ments (editorials, letters, meeting abstracts, etc) in the Thepreference of the Science Citation Index data-
numerator; citations to translated journal versions are 28
baseforEnglishlanguagejournals willcontributetoa
15-17
even listed twice. Because of this flawed computa- lowimpactfactorforthefewnon-Englishjournalsthat
32
tion, a journal that includes meeting reports, are included, since most citations to papers in
interesting editorials, and a lively correspondence sec- languages other than English are given by other
tion can have its impact factor greatly inflated relative 25 27 33
papers in the same language. The Institute for
to journals that lack such items. Editors who want to Scientific Information’s database for the social sciences
raise the impact of their journals should make frequent contained only two German social science journals,
reference to their previous editorials, since the 34
whereas a German database contained 542. Specifi-
database makes no correction for self citations. The cally, American scientists, who seem particularly prone
inclusion of review articles, which generally receive 33 35
to citing each other, dominate these databases to
17 18
many more citations than ordinary articles, is also such an extent (over half of the citations) as to raise
recommended. Furthermore, because citation rate is both the citation rate and the mean journal impact of
19 14
roughly proportional to the length of the article, American science 30% above the world average, the
journals might wish to publish long, rather than short, rest of the world then falling below average.This bias is
500 BMJ VOLUME314 15FEBRUARY1997
Education and debate
aggravated by the use of a short term index: for exam- Table 1 Journal impact factors and research field
ple, in American publications within clinical medicine,
83% of references in the same year were to other Journal 1986 1987
papersbyAmericanscientists(manyofthemundoubt- Annual Review of Biochemistry 31.6 35.1
edly self citations), a value 25% higher than the stable Annual Review of Immunology 26.5 25.2
level reached after three years (which would, inciden- Annual Review of Cell Biology 14.1 22.8
tally, also be biased by self citations and citations of Annual Review of Genetics 14.0 14.3
33 Annual Review of Neuroscience 15.4 13.7
other American work). Thus,both the apparent qual- Annual Review of Pharmacology 10.1 9.9
ity lead of American science and the values of the vari- Annual Review of Physiology 7.8 9.1
ous journal impact factors are, to an important extent, Annual Review of Biophysics 7.2 7.7
determined by the large volume, the self citations, and Annual Review of Microbiology 4.9 6.4
27
the national citation bias of American science, in
combination with the short term index used by the
Science Citation Index for calculating journal impact heavily on basic science,but not vice versa.The result is
factors. that basic medicine is cited three to five times more
than clinical medicine, and this is reflected in journal
42 44 45
Journal impact factors depend on the research field impact factors. The outcome of an evaluation
Citation habits and citation dynamics can be so differ- based on impact factors in medicine will therefore
ent in different research fields as to make evaluative depend on the position of research groups or
33
comparisons on the basis of citation rate or journal institutions along the basic-clinical axis.
impact difficult or impossible. For example,biochemis- In measures of citation rates of articles, attempts to
try and molecular biology articles were cited about five take research field into account often consist of
33 expressingcitationraterelativetosomecitationimpact
times as often as pharmacy articles. Several factors
46
have been found to contribute to such differences specific to the field. Suchfield corrections range from
amongfieldsofresearch. simply dividing the article’s citation rate by the impact
28
The citation impact of a research field is directly factor of its journal (which punishes publication in
proportional to the mean number of references per high impact journals) to the use of complex, author
47 48
article,whichvariesconsiderablyfromfieldtofield(itis specific, field indicators based on reference lists
twice as high in biochemistry as in mathematics, for (which punishes citations to high impact journals).
24 However,field corrections cannot readily be applied to
example). Within the arts and humanities, references
to articles are hardly used at all, leaving these research journal impact factors, since many research fields are
36 dominatedbyoneorafewjournals,inwhichcasecor-
fields (and others) virtually uncited, a matter of
considerable consternation among science administra- rections might merely generate relative impact factors
37 of unit value. Even within large fields, the tendency of
tors unfamiliar with citation kinetics.
Inhighlydynamicresearchfields,suchasbiochem- journals to subspecialise with certain subjects is likely
istry and molecular biology, where published reports to generate significant differences in journal impact: in
rapidly become obsolete, a large proportion of a single biochemical journal there was a 10-fold differ-
19
citations are captured by the short term index used to enceincitation rates in subfields.
calculate journal impact factors, as previously
38
discussed —but fields with a more durable literature, Is the impact of an article increased by
such as mathematics, have a smaller fraction of short publication in a high impact journal?
term citations and hence lower journal impact factors.
This field property combines with the low number of It is widely assumed that publication in a high impact
references per article to give mathematics a recorded journal will enhance the impact of an article (the “free
citation impact that is only a quarter that of ride” hypothesis). In a comparison of two groups of
24
biochemistry. scientific authors with similar journal preference who
Inyoungandrapidlyexpandingresearchfields,the differed twofold in mean citation rate for articles, how-
number of publications making citations is large ever, the relative difference was the same (twofold)
relative to the amount of citable material, leading to throughout a range of journals with impact factors of
12
high citation rates for articles and high journal impact 0.5 to 8.0. If the high impact journals had contributed
39 40
factors for the field. “free” citations, independently of the article contents,
In a largely self contained research field, the mean the relative difference would have been expected to
49
article (or journal) citation rate is independent of the diminish as a function of increasing journal impact.
41
size of the field, buttheabsoluterangewillbewiderin These data suggest that the journals do not offer any
a large field, meaning higher impact factors for the top free ride. The citation rates of the articles determine
42
journals. Such differences become obvious when the journal impact factor (a truism illustrated by the
comparing review journals, which tend to top their good correlation between aggregate citation rates of
field (table 1). Leading scientists in a small field may article and aggregate journal impact found in these
thus be at a disadvantage compared with their data), but not vice versa.
colleagues in larger fields, since they lack access to If scientific authors are not detectably rewarded
43 with a higher impact by publishing in high impact
journals of equally high citation impact.
Most research fields are, however, not completely journals, why are we so adamant on doing it? The
self contained,themostimportantfieldfactorprobably answer,ofcourse,isthataslongastherearepeopleout
being the ability of a research field to be cited by adja- there who judge our science by its wrapping rather
cent fields. The relation between basic and clinical than by its contents, we cannot afford to take any
medicine is a case in point: clinical medicine draws chances. Although journal impact factors are rarely
BMJ VOLUME314 15FEBRUARY1997 501
no reviews yet
Please Login to review.
