Filetype PDF | Posted on 20 Aug 2022 | 4 years ago
Authentication
digital resources
239x Filetype PDF File size 0.62 MB Source: files.stlouisfed.org
79
Richard G. Anderson and William G. Dewald
Richard G. Anderson is a researchofficer atthe Federal Reserve Bank
ofSt Louis, William C. Dewald is director ofresearchatthe Federal
Reserve Bank ofSt. Louis.
II Replication and Scientific
Standards in Applied Economics
a Decade After the Journal of
Money, Credit and Banking
Project
INCE EARLY1993, the Research Department an annotated version of the computer program
oftheFederalReserve Bank ofSt. Louis has made is prepared and all statistical results recalculated.
the data and programs for articles published in Finally, bibliographic and other references are
the Bank’s Reviewavailable to the public on its checkedby the analyst against original source
1 docmnents. We believe this practice both assures
electronic bulletin board. During the first year,
files from articles in the Review were downloaded the accuracy of the empirical results and allows
fromthe bulletin board more than 200 times. the interested reader to delve into the details
Morerecently, about 30 files havebeen down- of the author’s research.
loaded each month.
TheResearchDepartment ofthe Bank develops THE ROLE OF DATA IN ECONOMIC
the program and data files on our bulletin board EXPERIMENTS
during a replication of each article prior to pub- Although empirical knowledge inboth the
lication. A research analyst first checks the physical and social sciences arises from repeated
author’s data against original sources. Because experiments, therole of datadiffers. Inthe physical
databases may havebeen updated or revised sciences, scientists controla relatively smallnumber
after the research began, thiscart require searching of variables such as temperature, atmospheric
for the original published data. In a few cases, pressure, diet or family characteristics. Since
data errors have been corrected, fortunately with some variables are neither observed nor controlled,
only minorimpact on the author’s results. Next, no two repetitions of an experiment will be
Thebulletin board is advertised as the Federal Reserve Anderson (1986)summarizedtheJournalofMoney, Credit
Economic Database, orFRED. FRED’s phonenumber is and Bankingproject mentioned in the title,
(314) 621-1824. (TheFederalReserveSystem does not
have aserveron theInternet,) Dewald, Thursby and
NOVEMBER/DECEMBER 1994
80
identical. Response surface analysis and the newer The trepidation of authors aside, scientific
field of research synthesis providetools for ana- progress depends on challenging received wisdom.
lyzing the dependence of experimental results In applied economics, these challenges fall into
2
on the settings ofthe conditioning variables. three categories: replication of published results
In economics, however, unlike the physical using the previous authors’ data and programs;
sciences, researchers can only condition on the applying new statistical methods or techniques
observed values of the environmental variables, to authors’ datasets; and application ofexisting
not control them. Consider a simple model of statistical methods (including those used by
4
an economic experiment: previous authors) to new datasets. That most
applied economic research falls within the third
1. Form hypotheses. category is not surprising, since the first two
2. Collect data. depend on access toprevious authors’ datasets.
3. Develop theoretical and econometric Only with the authors’ data may the reader
repeat, or replicate, all five steps of the scientific
framework. experiment. Selecting a new set of values for the
4. Estimate. conditioning variables from published sources
5. Test hypotheses, draw conclusions. may yield results close to those obtained by
theauthor, or results that are quite different.
The values of the conditioning variables are Unfortunately, it is difficult to predict the sensi-
collected in step 2, Published articles typically tivity of authors’ results to variations in the values
describe steps 1, 3 and 5,but are most often silent ofthe conditioning variables. For an example
on step 2. In principle, a researcher armed with ofthe wide range of results that may arise when
the values ofthe conditioning variables and the mixing different sources and vintages of data, see
computer code for step 4 should be able to exactly thecomputer simulation experiment reported in
3 Dewald, Thnrsby and Anderson (1986),
reproduce an economic experiment. Unlike the
physical sciences, the experiment is deterministic,
given the data. THE JMCB PROJE:CT AF:TER 10 YEARS
Appraising the robustness of the results of TheJournal ofMoney, Credit and Banking
an economic experiment requires knowing the project, conducted from 1982-84 at the editorial
values of the conditioning variables used by the offices ofthe JMCB at The Ohio State University
researcher. Obtaining the data may sometimes in Columbus, was the first attempt by a profes-
be difficult. Datasets and programs may be mislaid sional journal to make authors’ programs and
or lost during the interval between completion of 5
data available to its readers. During the project,
the research andpublication ofan article. Further, the JIVICBasked authors to submit data and pro-
requests to authors for data may raise suspicions grams to the journal’s office. Conceptually, we
that the reader hopes (or expects) to find errors regarded the research reported in each article
in theauthors’ research. An individual researcher as the outcome ofan experiment. A complete
has strong incentives not to share data and pro- understanding of the experiment required the
grams. Ifthe materials are shared and results researchers’ data and computer programs, as
confirmed, the confirmation provides little (if any) well as the published summary descriptions and
reward to the researcher beyond the original conclusions. For a subset of these submissions,
publication of his findings. Ifresults are found weattempted torepeat steps 2 and 4bycollecting
faulty, however, the researcher faces the likeli- data from the sources cited by the authors and
hood of some professional embarrassment. re-running the authors’ computer programs.
2 SeeCooper andHedges(1994). Theprimary research team wasWilliam Dewald, Jerry
Acomplication notdealt with hereare errors and inconsis- Thursby, Richard Anderson and Hashem Dezbaksh. The
tencies in econometric computer programs. In the Journalof project’sfindings are summarized in Dewald, Thursby and
Money, Credit andBanking project (described furtherbelow), Anderson (1986). The projectwas supported in part by the
we requestedthat authors providethe version, release date National Science Foundation,
andserial number of the computer program used fortheir
estimation. SeeLovell and Selover (1994) for examples of
the variation in econometric packages.
Various classification schemes and nomenclatures have
beendiscussed by Kane (1984), M’ittelstaedt andZorn (1984),
Hubbard andVetter (1991), Lindsay and Ehrenberg (1993),
and Fuess (1994).
FEDERALRESERVE BANKOF ST. LOUIS
81
concluded that it is important for journals to
IMe request data from authors immediately following
completion of the research, and for the journal
Dets forthe4*408Pr ject to retain the data to avoid its loss during the
interval between completion ofthe research
aalasesnaqueateay MolttttwøYear and publication of the paper.
throug M*y1982 In the second part of theJMCB project, we
198344 as ~ ~ 88 69 studied whether the materials submitted by
Js~ ~ 1 authors were in fact sufficient to allow another
eb. 211 S 64 researchertorepeattheirexperiment.
Mefeh G 9\ 4 t I Formany articles, repeating step 2—searching
AM I 28 ‘1 1
0! 4 11 2 for the authors’ data in their stated sources—was
June 1 G 62impossible.Descriptionsofsourceswereeither
S 4 s too vague to allow us to locate the data and/or
Aug 4 G 2 the datawere not included in the cited sources.
Sep 442! 1 Although 54 datasets were submitted tothe
JMCB during the project, we judged only eight
Get 1 824assatisfactoryand14asvaluelessinhelpingus
Nov St understand the authors’ published work. Others
Dec 0 1 8 4 were deficient in at least one important respect.
For a few articles, we discovered data errors
asesetsAvanmbtnylasuaakàtMay49* during comparison topublished sources. In the
Ba s~s ass a as most severe case, we found that an author’s con-
Feb 6 4 6 2 clusions were reversed (prior to publication of
Ma 2 6 the article) when an error was corrected. Where
Agg~ 3 8 2 the data were adequate, we usually obtained
14ev 4 577 ~1 S numerical results from authors’ programs very
close to those reported by the authors.
Totat S 15 22424 17 1* ‘14 17 Beyond encouraging readers to explore authors’
‘N methodology and the robustness of published
501)140 JMOIð thtdfte,steff N t~ata~ts results, we believed that requesting data and
aysabe cmtheJMCastaff programs from authors would encourage them
to exercise added care during their research.
Wefound during the JMCB project that many Wealso expected that other journals would adopt
authors could not furnish data and program similar requirements toincrease the value oftheir
following publication of their articles. We ini- articles to readers. Although the JMCB project
tially requested data from the authors of 62 articles stimulated discussion of the role of replication
published during 1980-82, prior to the beginning in economics, no other journal adopted a policy
ofthe project on July 1, 1982. About one third of requesting data from authors during the I 980s.
ofthe authors did not respond toeither a first or Somejournals adopted editorial statements that
second request for data. Among the responding authors should stand ready to provide data and
authors, one-half either could not locate their programs to other researchers. Such statements,
data or chose not to submit them. Most of these in and ofthemselves, maynot solve the two major
authors said that they could have done so if the problems identified during the project: Dataoften
materials had been requested when the manu- are mislaid prior to publication of the article, and
script was first submitted to the JMCB. Data and the author may he suspicious of the motives of a
programs were often apparently mislaid during researcher requesting his data and programs.
the relatively long delay between completion of A decade after theJMCBproject, the replication
the research and publication of its findings. of previous studies as a part ofnew research seems
Wenext requested data from the authors of an infrequent occurrence. During the last decade,
papers that either had only recently been accepted no papers or notes in the JMCBhave focused
for publication or were under editofal review primarily on replication, and only about two
More than three-fourths furnished their data. We papers per year have included a direct compari-
NOVEMBER/DECEMBER1994
82
sonofthe authors’ results tothose in previous a much larger number of journals. The editors
studies, whether published in theJMCB or else- of22 journals, however, declined invitations
where.~Thecollection of databy professional fromNSF’s economics program to request that
0
journals also remains rare. The JMCB discontin- their authors place data in the ICPSRarchive.
ued requesting data from authors in 1993. To The National Science Foundation has also
our knowledge, today only two journals—the adopted guidelines to reduce the cost of replica-
Journal ofApplied Econometrics and theJournal tions. The guidelines require that authors place
ofBusiness andEconomic Statistics—routinely any data used and/or developed in conjunction
request data from authors, and neither requests with an NSF-funded project in a public archive
their programming. not later than six months following the end of
FromJanuary 1983 through mid-1989, theJMCB the grant period. Applications for additional
received nearly 150 submitted datasets and about NSF funds must contain a statement ofhow the
300 requests, as shown in Table i.7 Except for a author has complied with this requirement.
surge in requests following the publication of TheICPSR accepts data from any author who
Dewald, Thursby and Anderson(1986), on balance has received NSF funds.°
only a few datasets were requested each month National Science Foundation initiatives have
eventhough the number ofavailable datasets also assisted users of copyrighted and confidential
increased significantly during the decade, data. Somedata obtained by researchers from
The higher request rate during the past two commercial vendors are copyrighted and may
years for data from the St. Louis bulletin board not be further distributed by the researcher
may suggest that the modest costs ofrequesting without the vendor’s permission. One such
data from theJMCB still exceeded the marginal vendor, the Centerfor Research in Security Prices
valueto an individual researcher of replicating (CRISP), has agreed to maintain researchers’
a previous study. To obtain data fora]MCB article, datasets as part of its own database and make
a readerhad to call the editorial office to ask the them available to all licensed users of its data.
priceofthe data, submit payment by mail, wait for Forconfidential data, the Bureau of the Census
thedatatobereproduced and mailed, and perhaps and the NSF are exploring opening regional
re-enter the data into a computer. By contrast, offices that would allow researchers access to
the St. Louis bulletin board is free, delivery is confidential data, including datasets used in
immediate, and data are machine-readable. previously published studies. A pilot office
is operating in Boston.
THE ROLE OF THE NATIONAL REPLICATION OF THE JMGB PROJECT
SCIENCE FOUNDATION IN ST. LOUIS
The economics program of the National Ourexperience at the JM~JBduring 1982-84
Science Foundation (NSF) has sought to build was itself only onetrial of an experiment. Would
a heightened awareness of the value of data another sample of authors also have difficulty
collection, archival and distribution among providing data following publication of their
economists during the last decade. Following articles? Or were our original findings anom-
publication of Dewald, Thursby and Anderson alous, leading us to greatly exaggerate the
(1986), the NSF established an archive for the problem, as some critics have suggested?
storage and distribution of authors’ data at the
Inter-university Consortium for Political and During 1992-93, we repeated theJMCBexperi-
Social Research (ICPSR) at the University of ment at St. Louis. in part, by requesting data and
Michigan. Initially, some anticipated that the programs from the authors ofpapers presented
NSF’s effort would extend the JMCB’s practice at the Bank’s annual economic policy conference
ofrequesting and distributing authors’ data to in October 1992. We did not tell authors prior
Replication also has been relatively rare even in journals o Materialsshould be submitted to User Support, ICPSR, P.O.
that encourage submission of such papers- See Fuess Box 1248, Ann Arbor, MI 48106. Data may be retrieved
(1994). from ICPSR via an Internet server; see Goffe (summer
Recall that in 1982 we began requesting data for articles 1994; March 1994).
that had been published asearly as 1980.
o Some authors havesince proposed models ofhow such col-
lective disinterest among professionaljournals might arise
and be sustained. See Feigenbaum and Levy (1994, 1993),
FEDERALRESERVE BANKOF ST. LOUiS
no reviews yet
Please Login to review.
