jagomart
digital resources
picture1_Journal Pdf 98095 | 9056804


 180x       Filetype PDF       File size 0.41 MB       Source: www.ncbi.nlm.nih.gov


File: Journal Pdf 98095 | 9056804
education and debate whythe impact factor of journals should not be used for evaluating research per o seglen institute for studies evaluating scientific quality is a notoriously difficult in research ...

icon picture PDF Filetype PDF | Posted on 21 Sep 2022 | 3 years ago
Partial capture of text on file.
                                           Education and debate
                                           Whythe impact factor of journals should not be used for
                                           evaluating research
                                           Per O Seglen
                Institute for Studies      Evaluating scientific quality is a notoriously difficult
                in Research and            problem which has no standard solution. Ideally, pub-
                Higher Education           lished scientific results should be scrutinised by true                                                Summary points
                (NIFU),
                Hegdehaugsveien            experts in the field and given scores for quality and                                 • Use of journal impact factors conceals the
                31, N-0352 Oslo,           quantity according to established rules. In practice,                                 difference in article citation rates (articles in the
                Norway                                                                                                           most cited half of articles in a journal are cited
                Per O Seglen,              however, what is called peer review is usually
                professor                  performed by committees with general competence                                       10timesasoftenastheleastcitedhalf)
                                           ratherthanwiththespecialist’sinsightthatisneededto                                    • Journals’ impact factors are determined by
                BMJ 1997;314:498–502       assess primary research data. Committees tend, there-                                 technicalities unrelated to the scientific quality
                                           fore,    to resort to secondary criteria like crude                                   of their articles
                                           publication counts, journal prestige, the reputation of                               • Journal       impact factors depend on the
                                           authors and institutions, and estimated importance                                    research field: high impact factors are likely in
                                                                                                 1                               journals covering large areas of basic research
                                           and relevance of the research field, making peer
                                                                                                                 23              with a rapidly expanding but short lived
                                           review as much of a lottery as of a rational process.
                                                On this background, it is hardly surprising that                                 literature that use many references per article
                                           alternative methods for evaluating research are being                                 • Article citation rates determine the journal
                                           sought, such as citation rates and journal impact                                     impact factor,not vice versa
                                           factors, which seem to be quantitative and objective
                                           indicators directly related to published science. The
                                                                                                                                                       5
                                           citation data are obtained from a database producedby                            academic positions. In the Nordic countries, journal
                                           the Institute        for    Scientific      Information (ISI) in                 impact factors have, on occasion, been used in the
                                           Philadelphia, which continuously records scientific                              evaluation of individuals as well as of institutions and
                                           citations as represented by the reference lists of articles                      have been proposed, or actually used, as one of the
                                           from a large number of the world’s scientific journals.                          premises for allocation of university resources and
                                                                                                                                         167
                                           Thereferences are rearranged in the database to show                             positions.        Resource allocation based on impact fac-
                                                                                                                                                                                               8
                                           howmanytimeseachpublicationhasbeencitedwithin                                    tors has also been reported from Canada and
                                                                                                                                        9
                                           a certain period, and by whom, and the results are                               Hungary and, colloquially, from several other coun-
                                           published as the Science Citation Index (SCI). On the                            tries. The increasing awareness of journal impact
                                           basis    of the Science Citation Index and authors’                              factors, and the possibility of their use in evaluation, is
                                           publication lists, the annual citation rate of papers by a                       already changing scientists’ publication behaviour
                                           scientific    author or research group can thus be                               towards      publishing        in    journals       with     maximum
                                                                                                                                      910
                                           calculated. Similarly, the citation rate of a scientific                         impact,        often at the expense of specialist journals
                                           journal—known as the journal impact factor—can be                                that might actually be more appropriate vehicles for
                                           calculated as the mean citation rate of all the articles                         the research in question.
                                                                             4                                                   Given the increasing use of journal impact
                                           containedinthejournal. Journalimpactfactors,which
                                                                                                                            factors
                                           are published annually in SCI Journal Citation Reports,                                   —as well as the (less explicit) use of journal
                                           are widely regarded as a quality ranking for journals                            prestige—in research evaluation, a critical examination
                                           andusedextensivelybyleadingjournalsintheiradver-                                 of this indicator seems necessary (see box).
                                           tising.
                                                Since journal impact factors are so readily                                 Is the journal impact factor really
                                           available, it has been tempting to use them for evaluat-                         representative of the individual journal
                                           ing individual scientists or research groups. On the                             articles?
                                           assumptionthatthejournalisrepresentativeofits arti-
                                           cles, the journal impact factors of an author’s articles                         Relation of journal impact factor and citation rate
                                           cansimplybeaddeduptoobtainanapparentlyobjec-                                     of article
                                           tive and quantitative measure of the author’s scientific                         For the journal’s impact factor to be reasonably
                                           achievement. In Italy, the use of journal impact factors                         representative of its articles, the citation rate of
                                           was recently advocated to remedy the purported                                   individual articles in the journal should show a narrow
                                           subjectivity      and bias in appointments to higher                             distribution,preferably a Gaussian distribution,around
                498                                                                                                                                       BMJ VOLUME314 15FEBRUARY1997
                                                                                                                                                                                                          Education and debate
                          Problems associated with the use of                                                                  100
                          journal impact factors                                                                                90
                          • Journal          impact        factors        are      not      statistically                       80
                          representative of individual journal articles                                                         70
                          • Journal impact factors correlate poorly with actual
                          citations of individual articles                                                                  % Of journal's citation rate60
                          • Authors use many criteria other than impact when
                          submitting to journals                                                                                50
                          • Citations to “non-citable” items are erroneously
                          included in the database                                                                              40
                          • Self citations are not corrected for                                                                30
                          •    Review articles are heavily cited and inflate the
                          impact factor of journals                                                                             20
                          • Long articles collect many citations and give high
                          journal impact factors                                                                                10
                          • Shortpublication lag allows many short term journal                                                  0
                          self citations and gives a high journal impact factor                                                    0102030405060708090100
                          • Citations in the national language of the journal are                                                               % Of journal articles (in 5% categories by annual citation rate)
                          preferred by the journal’s authors
                          • Selective        journal self citation: articles tend to                                     Fig 2 Cumulative contribution of articles with different citation
                          preferentially cite other articles in the same journal                                         rates (beginning with most cited 5%) to total journal impact.
                          • Coverageofthedatabaseisnotcomplete                                                           Values are mean (SE) of journals in fig 1; dotted lines indicate
                          • Booksarenotincludedinthedatabaseasasourcefor                                                 contributions of 15% and 50% most cited articles11
                          citations
                          • Database has an English language bias
                          • Database is dominated by American publications
                          • Journal set in database may vary from year to year                                                 The uneven contribution of the various articles to
                          • Impact factor is a function of the number of                                                 the journal impact is further illustrated in figure 2: the
                          references per article in the research field
                          • Research fields with literature that rapidly becomes                                         cumulative curve shows that the most cited 15% of the
                          obsolete are favoured                                                                          articles account for 50% of the citations, and the most
                          • Impact factor depends on dynamics (expansion or                                              cited 50% of the articles account for 90% of the
                          contraction) of the research field                                                             citations. In other words, the most cited half of the arti-
                          • Small research fields tend to lack journals with high
                          impact                                                                                         cles are cited, on average, 10 times as often as the least
                          • Relations between fields (clinical v basic research, for                                     cited half. Assigning the same score (the journal
                          example) strongly determine the journal impact factor                                          impact factor) to all articles masks this tremendous
                          • Citation rate of article determines journal impact,but                                       difference
                          not vice versa                                                                                                  —which is the exact opposite of what an
                                                                                                                         evaluation is meant to achieve. Even the uncited
                                                                                                                         articles are then given full credit for the impact of the
                    the mean value (the journal’s impact factor). Figure 1                                               few highly cited articles that predominantly determine
                    shows that this is far from being the case: three differ-                                            the value of the journal impact factor.
                    ent biochemical journals all showed skewed distribu-                                                       Since any large, random sample of journal articles
                    tions of articles’ citation rates, with only a few articles                                          will correlate well with the corresponding average of
                                                                                    11                                                                           12
                    anywherenearthepopulationmean.                                                                       journal impact factors,                     the impact factors may seem
                                                                                                                         reasonably representative after all. However, the corre-
                                                                                                                         lation between journal impact and actual citation rate
                          600                                                                                            of articles from individual scientists or research groups
                                                                             Biochimica Biophysica Acta                                        912
                                                                             Biochemical Journal                         is often poor              (fig 3). Clearly, scientific authors do not
                                                                             Journal of Biological Chemistry             necessarily publish their most citable work in journals
                          500                                                                                            of the highest impact, nor do their articles necessarily
                       No of articles cited                                                                              match the impact of the journals they appear in.
                          400                                                                                            Although some authors may take journals’ impact fac-
                                                                                                                         tors into consideration when submitting an article,
                                                                                                                         other factors are (or at least were) equally or more
                          300                                                                                            important,suchasthejournal’ssubjectareaanditsrel-
                                                                                                                         evance to the author’s specialty, the fairness and rapid-
                                                                                                                         ity     of     the editorial process, the probability of
                          200                                                                                            acceptance,publication lag, and publication cost (page
                                                                                                                                        13
                                                                                                                         charges).
                          100                                                                                                  Journal impact factors are representative only
                                                                                                                         when the evaluated research is absolutely average
                                                                                                                         (relative to the journals used), a premise which really
                             0                                                                                           makes any evaluation superfluous. In actual practice,
                                                                                                                         however, even samples as large as a nation’s scientific
                               ≤1  2-3   4-5  6-7  8-9                                                        30         output are far from being random and representative
                                                      10-1112-13 14-15 16-1718-1920-2122-2324-2526-2728-29   ≥
                                                                            No of citations per article per year         of the journals they have been published in: for exam-
                    Fig 1 Citation rates in 1986 or 1987 of articles published in three                                  ple, during the period 1989-93, articles on general
                    biochemical journals in 1983 or 1984, respectively11                                                 medicine in Turkey would have had an expected
                                                                                                                         citation rate of 1.3 (relative to the world average) on the
                    BMJ VOLUME314 15FEBRUARY1997                                                                                                                                                                                                   499
               Education and debate
                                                                                                                 articles. If correction were made for article length,
                  10      r=0.05                                 10     r=0.27                                   “communications” journals like Biochemical and Bio-
                                                                                                                 physical Research Communications and FEBS Letters
                    5                                             5                                              wouldgetimpactfactorsashighas,orhigherthan,the
                                                                                                                 high impact journals within the field, like Journal of
                                                                                                                                          20 21
                    2                                             2                                              Biological Chemistry.
                                                                                                                      Theuseofanextremelyshorttermindex(citations
                    1                                             1                                              to articles published only in the past two years) in cal-
                 No of citations per article per year                                                            culating     the impact factor introduces a strong
                  0.5                                           0.5                                              temporalbias,withseveralconsequences.Forexample,
                                                                                                                 articles in journals with short publication lags will con-
                  0.2                                           0.2                                              tain relatively many up to date citations and thus con-
                                                                                                                 tribute heavily to the impact factors of all cited
                    0                                             0                                              journals. Since articles in a given journal tend to cite
                                                                                                                                                          22
                          r=0.44                                        r=0.63                                   articles from the same journal,             rapid publication is
                  10                                             50                                              self serving with respect to journal impact, and signifi-
                                                                                                                                                23
                    5                                            20                                              cantly correlated with it.       Dynamicresearchfields with
                                                                                                                 high activity and short publication lags, such as
                                                                 10                                              biochemistry and molecular biology, have a corre-
                    2                                             5                                              spondingly high proportion of citations to recent
                                                                                                                 publications
                                                                                                                                 —andhencehigher journal impact factors
                    1                                             2                                                                                                                23 24
                 No of citations per article per year                                                            —than, for example, ecology and mathematics.
                                                                  1                                              Russian journals, which are cited mainly by other
                  0.5                                                                                                                   25
                                                                0.5                                              Russian journals,         are reported to have particularly
                                                                                                                 long publication lags, resulting in generally low impact
                  0.2                                                                                                     26
                                                                0.2                                              factors.    Pure technicalities can therefore account for
                    0                                             0                                              several-fold differences in journal impact.
                         0.5      1       2          5       10             0.5   1     2       5    10    20    Limitations of the database
                                           Impact factor of journal                       Impact factor of journal
               Fig 3 Correlation between article citation rate and journal impact for four authors12             The Science Citation Index database covers about
                                                                                                                                    8
                                                                                                                 3200 journals ; the estimated world total is about
                                                                                                                            27
                                                                                                                 126000. The coverage varies considerably between
                                       basis of journal impact, but the actual citation was only                 research fields: in one university, 90% of the chemistry
                                            14                                                                   faculty’s publications, but only 30% of the biology fac-
                                       0.3.   The use of journal impact factors can therefore
                                                                                                                                                                       28
                                       beasmisleadingforcountries as for individuals.                            ulty’s publications, were in the database.               Since the
                                                                                                                 impactfactorofanyjournalwillbeproportionaltothe
                                       Journal impact factors are calculated in a way that                       database coverage of its research field, such discrepan-
                                       causes bias                                                               cies mean that journals from an underrepresented
                                       Apart from being non-representative, the journal                          field that are included will receive low impact factors.
                                       impact factor is encumbered with several shortcom-                        Furthermore, the journal set in the database is not
                                       ings of a technical and more fundamental nature. The                      constant but may vary in composition from year to
                                       factor is generally defined as the recorded number of                     year.24 29 In many research fields a substantial fraction
                                       citations within a certain year (for example, 1996) to                    of scientific output is published in the form of books,
                                       the items published in the journal during the two pre-                    whicharenotincludedassourceitemsinthedatabase;
                                       ceding years (1995 and 1994), divided by the number                                                                      30
                                                                                                                 they therefore have no impact factor. In mathematics,
                                       of such items (this would be the equivalent of the aver-                  leading publications that were not included in the Sci-
                                       age citation rate of an item during the first and second                  ence Citation Index database were cited more
                                       calendar year after the year of publication). However,                    frequently than the leading publications that were
                                       the Science Citation Index database includes only nor-                                31
                                                                                                                 included. Clearly, such systematic omissions from the
                                       mal articles, notes, and reviews in the denominator as                    databasecancauseseriousbiasinevaluationsbasedon
                                       citable items, but records citations to all types of docu-                impact factor.
                                       ments (editorials, letters, meeting abstracts, etc) in the                     Thepreference of the Science Citation Index data-
                                       numerator; citations to translated journal versions are                                                               28
                                                                                                                 baseforEnglishlanguagejournals willcontributetoa
                                                             15-17
                                       even listed twice.         Because of this flawed computa-                lowimpactfactorforthefewnon-Englishjournalsthat
                                                                                                                                   32
                                       tion,    a   journal     that    includes      meeting reports,           are included,         since most citations to papers in
                                       interesting editorials, and a lively correspondence sec-                  languages other than English are given by other
                                       tion can have its impact factor greatly inflated relative                                                         25 27 33
                                                                                                                 papers in the same language.                    The Institute for
                                       to journals that lack such items. Editors who want to                     Scientific Information’s database for the social sciences
                                       raise the impact of their journals should make frequent                   contained only two German social science journals,
                                       reference to their previous editorials, since the                                                                                  34
                                                                                                                 whereas a German database contained 542. Specifi-
                                       database makes no correction for self citations. The                      cally, American scientists, who seem particularly prone
                                       inclusion of review articles, which generally receive                                                33 35
                                                                                                                 to citing each other,           dominate these databases to
                                                                                               17 18
                                       many more citations than ordinary articles,                   is also     such an extent (over half of the citations) as to raise
                                       recommended. Furthermore, because citation rate is                        both the citation rate and the mean journal impact of
                                                                                                           19                                                                   14
                                       roughly proportional to the length of the article,                        American science 30% above the world average, the
                                       journals might wish to publish long, rather than short,                   rest of the world then falling below average.This bias is
               500                                                                                                                           BMJ VOLUME314 15FEBRUARY1997
                                                                                                                                       Education and debate
             aggravated by the use of a short term index: for exam-              Table 1 Journal impact factors and research field
             ple, in American publications within clinical medicine,
             83% of references in the same year were to other                    Journal                                   1986         1987
             papersbyAmericanscientists(manyofthemundoubt-                       Annual Review of Biochemistry             31.6         35.1
             edly self citations), a value 25% higher than the stable            Annual Review of Immunology               26.5         25.2
             level reached after three years (which would, inciden-              Annual Review of Cell Biology             14.1         22.8
             tally, also be biased by self citations and citations of            Annual Review of Genetics                 14.0         14.3
                                        33                                       Annual Review of Neuroscience             15.4         13.7
             other American work). Thus,both the apparent qual-                  Annual Review of Pharmacology             10.1          9.9
             ity lead of American science and the values of the vari-            Annual Review of Physiology                7.8          9.1
             ous journal impact factors are, to an important extent,             Annual Review of Biophysics                7.2          7.7
             determined by the large volume, the self citations, and             Annual Review of Microbiology              4.9          6.4
                                                                       27
             the national citation bias of American science,              in
             combination with the short term index used by the
             Science Citation Index for calculating journal impact               heavily on basic science,but not vice versa.The result is
             factors.                                                            that basic medicine is cited three to five times more
                                                                                 than clinical medicine, and this is reflected in journal
                                                                                                   42 44 45
             Journal impact factors depend on the research field                 impact factors.          The outcome of an evaluation
             Citation habits and citation dynamics can be so differ-             based on impact factors in medicine will therefore
             ent in different research fields as to make evaluative              depend on the position of research groups or
                                                                                                                              33
             comparisons on the basis of citation rate or journal                institutions along the basic-clinical axis.
             impact difficult or impossible. For example,biochemis-                  In measures of citation rates of articles, attempts to
             try and molecular biology articles were cited about five            take research field into account often consist of
                                                         33                      expressingcitationraterelativetosomecitationimpact
             times as often as pharmacy articles.          Several factors
                                                                                                       46
             have been found to contribute to such differences                   specific to the field.  Suchfield corrections range from
             amongfieldsofresearch.                                              simply dividing the article’s citation rate by the impact
                                                                                                         28
                  The citation impact of a research field is directly            factor of its journal     (which punishes publication in
             proportional to the mean number of references per                   high impact journals) to the use of complex, author
                                                                                                                                             47 48
             article,whichvariesconsiderablyfromfieldtofield(itis                specific, field indicators based on reference lists
             twice as high in biochemistry as in mathematics, for                (which punishes citations to high impact journals).
                         24                                                      However,field corrections cannot readily be applied to
             example). Within the arts and humanities, references
             to articles are hardly used at all, leaving these research          journal impact factors, since many research fields are
                                                           36                    dominatedbyoneorafewjournals,inwhichcasecor-
             fields (and others) virtually uncited,           a matter of
             considerable consternation among science administra-                rections might merely generate relative impact factors
                                                        37                       of unit value. Even within large fields, the tendency of
             tors unfamiliar with citation kinetics.
                  Inhighlydynamicresearchfields,suchasbiochem-                   journals to subspecialise with certain subjects is likely
             istry and molecular biology, where published reports                to generate significant differences in journal impact: in
             rapidly become obsolete, a large proportion of                      a single biochemical journal there was a 10-fold differ-
                                                                                                                       19
             citations are captured by the short term index used to              enceincitation rates in subfields.
             calculate    journal     impact    factors,    as   previously
                        38
             discussed     —but fields with a more durable literature,           Is the impact of an article increased by
             such as mathematics, have a smaller fraction of short               publication in a high impact journal?
             term citations and hence lower journal impact factors.
             This field property combines with the low number of                 It is widely assumed that publication in a high impact
             references per article to give mathematics a recorded               journal will enhance the impact of an article (the “free
             citation   impact that is only a quarter that of                    ride” hypothesis). In a comparison of two groups of
                             24
             biochemistry.                                                       scientific authors with similar journal preference who
                  Inyoungandrapidlyexpandingresearchfields,the                   differed twofold in mean citation rate for articles, how-
             number of publications making citations is large                    ever, the relative difference was the same (twofold)
             relative to the amount of citable material, leading to              throughout a range of journals with impact factors of
                                                                                            12
             high citation rates for articles and high journal impact            0.5 to 8.0.  If the high impact journals had contributed
                                    39 40
             factors for the field.                                              “free” citations, independently of the article contents,
                  In a largely self contained research field, the mean           the relative difference would have been expected to
                                                                                                                                               49
             article (or journal) citation rate is independent of the            diminish as a function of increasing journal impact.
                               41
             size of the field,  buttheabsoluterangewillbewiderin                These data suggest that the journals do not offer any
             a large field, meaning higher impact factors for the top            free ride. The citation rates of the articles determine
                       42
             journals.     Such differences become obvious when                  the journal impact factor (a truism illustrated by the
             comparing review journals, which tend to top their                  good correlation between aggregate citation rates of
             field (table 1). Leading scientists in a small field may            article and aggregate journal impact found in these
             thus be at a disadvantage compared with their                       data), but not vice versa.
             colleagues in larger fields, since they lack access to                  If scientific authors are not detectably rewarded
                                                           43                    with a higher impact by publishing in high impact
             journals of equally high citation impact.
                  Most research fields are, however, not completely              journals, why are we so adamant on doing it? The
             self contained,themostimportantfieldfactorprobably                  answer,ofcourse,isthataslongastherearepeopleout
             being the ability of a research field to be cited by adja-          there who judge our science by its wrapping rather
             cent fields. The relation between basic and clinical                than by its contents, we cannot afford to take any
             medicine is a case in point: clinical medicine draws                chances. Although journal impact factors are rarely
             BMJ VOLUME314 15FEBRUARY1997                                                                                                                          501
The words contained in this file might help you see if this file matches what you are looking for:

...Education and debate whythe impact factor of journals should not be used for evaluating research per o seglen institute studies scientific quality is a notoriously difficult in problem which has no standard solution ideally pub higher lished results scrutinised by true summary points nifu hegdehaugsveien experts the field given scores use journal factors conceals n oslo quantity according to established rules practice difference article citation rates articles norway most cited half are however what called peer review usually professor performed committees with general competence timesasoftenastheleastcitedhalf ratherthanwiththespecialist sinsightthatisneededto determined bmj assess primary data tend there technicalities unrelated fore resort secondary criteria like crude their publication counts prestige reputation depend on authors institutions estimated importance high likely covering large areas basic relevance making rapidly expanding but short lived as much lottery rational proce...

no reviews yet
Please Login to review.