Filetype PDF | Posted on 20 Aug 2022 | 4 years ago
Authentication
digital resources
193x Filetype PDF File size 0.30 MB Source: bmcmedresmethodol.biomedcentral.com
Thabane et al. BMC Medical Research Methodology 2010, 10:1
http://www.biomedcentral.com/1471-2288/10/1
COMMENTARY Open Access
A tutorial on pilot studies: the what, why and
how
1,2* 1,2 1,2 1,2 1,3 1,2 3
Lehana Thabane , Jinhui Ma , Rong Chu , Ji Cheng , Afisi Ismaila , Lorena P Rios , Reid Robson ,
Marroon Thabane1,4, Lora Giangregorio5, Charles H Goldsmith1,2
Abstract
Pilot studies for phase III trials - which are comparative randomized trials designed to provide preliminary evidence
on the clinical efficacy of a drug or intervention - are routinely performed in many clinical areas. Also commonly
know as “feasibility” or “vanguard” studies, they are designed to assess the safety of treatment or interventions; to
assess recruitment potential; to assess the feasibility of international collaboration or coordination for multicentre
trials; to increase clinical experience with the study medication or intervention for the phase III trials. They are the
best way to assess feasibility of a large, expensive full-scale study, and in fact are an almost essential pre-requisite.
Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and poten-
tially help to avoid doomed main studies. The objective of this paper is to provide a detailed examination of the
key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot study; 2) the
relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and mis-
conceptions about pilot studies; 4) the criteria for evaluating the success of a pilot study; 5) frequently asked ques-
tions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to
report the results of pilot investigations using the CONSORT format.
1. Introduction study is synonymous with a feasibility study intended to
The Concise Oxford Thesaurus [1] defines a pilot pro- guide the planning of a large-scale investigation. Pilot
ject or study as an experimental, exploratory, test, preli- studies are sometimes referred to as “vanguard trials” (i.
minary, trial or try out investigation. Epidemiology and e. pre-studies) intended to assess the safety of treatment
statistics dictionaries provide similar definitions of a or interventions; to assess recruitment potential; to
pilot study as a small scale assess the feasibility of international collaboration or
coordination for multicentre trials; to evaluate surrogate
“...test of the methods and procedures to be used marker data in diverse patient cohorts; to increase clini-
on a larger scale if the pilot study demonstrates cal experience with the study medication or interven-
that the methods and procedures can work” [2]; tion, and identify the optimal dose of treatments for the
“...investigation designed to test the feasibility of phase III trials [4]. As suggested by an African proverb
methods and procedures for later use on a large from the Ashanti people in Ghana “You never test the
scale or to search for possible effects and associa- depthofariverwithbothfeet“, the main goal of pilot
tions that may be worth following up in a subse- studies is to assess feasibility so as to avoid potentially
quent larger study” [3]. disastrous consequences of embarking on a large study -
which could potentially “drown” the whole research
Table 1 provides a summary of definitions found on effort.
the Internet. A closer look at these definitions reveals Feasibility studies are routinely performed in many
that they are similar to the ones above in that a pilot clinical areas. It is fair to say that every major clinical
trial had to start with some piloting or a small scale
investigation to assess the feasibility of conducting a lar-
* Correspondence: thabanl@mcmaster.ca ger scale study: critical care [5], diabetes management
1
Department of Clinical Epidemiology and Biostatistics, McMaster University,
Hamilton ON, Canada
© 2010 Thabane et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative
Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and
reproduction in any medium, provided the original work is properly cited.
Thabane et al. BMC Medical Research Methodology 2010, 10:1 Page 2 of 10
http://www.biomedcentral.com/1471-2288/10/1
Table 1 Some Adapted Definitions of Pilot Studies on the Web (Date of last access: December 22, 2009)
Definition* Source
A trial study carried out before a research design is finalised to assist in defining http://www.cirem.org.uk/definitions.html
the research question or to test the feasibility, reliability and validity of the
proposed study design
A smaller version of a study is carried out before the actual investigation is done. http://www.mh.state.oh.us/what-we-do/promote/research-and-
Researchers use information gathered in pilot studies to refine or modify the evaluation/learning-lab/research-glossary.shtml
research methodology for a study and to develop large-scale studies
A small scale study conducted to test the plan and method of a research study. http://www.umm.edu/nursing/docs/glossary_research_terms.
pdf
A small study carried out before a large-scale study to try out a procedure or to http://www.psych-sci.manchester.ac.uk/actnow/glossary/
test a principle
An experimental use of a treatment in a small group of patients to learn if it will be http://www.lungcanceralliance.org/news/glossary.html
effective and safe on a broad scale
The initial study examining a new method or treatment http://www.cdc.gov/des/consumers/resources/glossary.html#P
A small study often done to assist the preparation of a larger, more http://www.informedesign.umn.edu/Glossary.aspx?id=1952#
comprehensive investigation.
Small, preliminary test or trial run of an intervention, or of an evaluation activity such http://www.nsf.gov/pubs/2005/nsf0531/nsf0531_6.pdf
as an instrument or sampling procedure. The results of the pilot are used to
improve the program or evaluation procedure being piloted before it is used on
a larger scale.
*Emphasis is ours
intervention trials [6], cardiovascular trials [7], primary investigate the pharmacokinetics of a drug (i.e. how a
healthcare [8], to mention a few. drug is distributed and metabolized in the body) includ-
Despite their noted importance, the reality is that pilot ing finding a dose that can be tolerated with minimal
studies receive little or no attention in scientific research toxicity. Phase II trials provide preliminary evidence on
training. Few epidemiology or research textbooks cover the clinical efficacy of a drug or intervention. They may
the topic with the necessary detail. In fact, we are not or may not be randomized. Phase III trials are rando-
aware of any textbook that dedicates a chapter on this mized studies comparing two or more drugs or inter-
issue - many just mention it in passing or provide a cur- vention strategies to assess efficacy and safety. Phase IV
sory coverage of the topic. The objective of this paper is trials, usually done after registration or marketing of a
to provide a detailed examination of the key aspects of drug, are non-randomized surveillance studies to docu-
pilot studies. In the next section, we narrow the focus of ment experiences (e.g. side-effects, interactions with
our definition of a pilot to phase III trials. Section 3 other drugs, etc) with using the drug in practice.
covers the general reasons for conducting a pilot study. For the purposes of this paper, our approach to utiliz-
Section 4 deals with the relationships between pilot stu- ing pilot studies relies on the model for complex inter-
dies, proof-of-concept studies, and adaptive designs, ventions advocated by the British Medical Research
while section 5 addresses the challenges of pilot studies. Council - which explicitly recommends the use of feasi-
Evaluation of a pilot study (i.e. how to determine if a bility studies prior to Phase III clinical trials, but stresses
pilot study was successful) is covered in Section 6. We the iterative nature of the processes of development, fea-
deal with several frequently asked questions about pilot sibility and piloting, evaluation and implementation [11].
studies in Section 7 using a “question-and-answer”
approach. Section 8 covers some ethical aspects related 3. Reasons for Conducting Pilot Studies
to pilot studies; and in Section 9, we follow the CON- Van Teijlingen et al. [12] and van Teijlingen and Hund-
SORT format [9] to offer some suggestions on how to ley [13] provide a summary of the reasons for perform-
report the results of pilot investigations. ing a pilot study. In general, the rationale for a pilot
study can be grouped under several broad classifications
2. Narrowing the focus: Pilot studies for - process, resources, management and scientific (see also
randomized studies http://www.childrens-mercy.org/stats/plan/pilot.asp for a
Pilot studies can be conducted in both quantitative and different classification):
qualitative studies. Adopting a similar approach to Lan- Process: This assesses the feasibility of the steps that
caster et al. [10], we focus on quantitative pilot studies - need to take place as part of the main study. Examples
particularly those done prior to full-scale phase III trials. include determining recruitment rates, retention rates,
Phase I trials are non-randomized studies designed to etc.
Thabane et al. BMC Medical Research Methodology 2010, 10:1 Page 3 of 10
http://www.biomedcentral.com/1471-2288/10/1
Resources: This deals with assessing time and budget 5. Challenges of and Common Misconceptions
problems that can occur during the main study. The about Pilot Studies
idea is to collect some pilot data on such things as the Pilot studies can be very informative, not only to the
length of time to mail or fill out all the survey forms. researchers conducting them but also to others doing
Management: This covers potential human and data similar work. However, many of them never get pub-
optimization problems such as personnel and data man- lished, often because of the way the results are pre-
agement issues at participating centres. sented [13]. Quite often the emphasis is wrongly placed
Scientific: This deals with the assessment of treat- on statistical significance, not on feasibility - which is
ment safety, determination of dose levels and response, the main focus of the pilot study. Our experience in
and estimation of treatment effect and its variance. reviewing submissions to a research ethics board also
Table 2 summarizes this classification with specific shows that most of the pilot projects are not well
examples. designed: i.e. there are no clear feasibility objectives; no
clear analytic plans; and certainly no clear criteria for
4. Relationships between Pilot Studies, Proof-of- success of feasibility.
Concept Studies, and Adaptive Designs In many cases, pilot studies are conducted to generate
Aproof-of-concept (PoC) study is defined as a clinical data for sample size calculations. This seems especially
trial carried out to determine if a treatment (drug) is sensible in situations where there are no data from pre-
biologically active or inactive [14]. PoC studies usually vious studies to inform this process. However, it can be
use surrogate markers as endpoints. In general, they are dangerous to use pilot studies to estimate treatment
phase I/II studies - which, as noted above, investigate effects, as such estimates may be unrealistic/biased
the safety profile, dose level and response to new drugs because of the limited sample sizes. Therefore if not
[15]. Thus, although designed to inform the planning of used cautiously, results of pilot studies can potentially
phase III trials for registration or licensing of new drugs, mislead sample size or power calculations [21] – parti-
PoC studies may not necessarily fit our restricted defini- cularly if the pilot study was done to see if there is likely
tion of pilot studies aimed at assessing feasibility of to be a treatment effect in the main study. In section 6,
phase III trials as outlined in Section 2. we provide guidance on how to proceed with caution in
An adaptive trial design refers to a design that allows this regard.
modifications to be made to a trial’s design or statistical There are also several misconceptions about pilot stu-
procedures during its conduct, with the purpose of effi- dies. Below are some of the common reasons that
ciently identifying clinical benefits/risks of new drugs or researchers have put forth for calling their study a pilot.
to increase the probability of success of clinical develop- The first common reason is that a pilot study is a
ment [16]. The adaptations can be prospective (e.g. small single-centre study. For example, researchers often
stopping a trial early due to safety or futility or efficacy state lack of resources for a large multi-centre study as
at interim analysis); concurrent (e.g. changes in eligibil- a reason for doing a pilot. The second common reason
ity criteria, hypotheses or study endpoints) or retrospec- is that a pilot investigation is a small study that is simi-
tive (e.g. changes to statistical analysis plan prior to lar in size to someone else’s published study. In review-
locking database or revealing treatment codes to trial ing submissions to a research ethics board, we have
investigators or patients). Piloting is normally built into come across sentiments such as
adaptive trial designs by determining a priori decision
rules to guide the adaptations based on cumulative data. So-and-so did a similar study with 6 patients and
For example, data from interim analyses could be used got statistical significance-oursuses12patients
to refine sample size calculations [17,18]. This approach (double the size)!
is routinely used in internal pilot studies - which are We did a similar pilot before (and it was
primarily designed to inform sample size calculation for published!)
the main study, with recalculation of the sample size as
the key adaptation. Unlike other phase III pilots, an The third most common reason is that a pilot is a
internal pilot investigation does not usually address any small study done by a student or an intern - which can
other feasibility aspects - because it is essentially part of be completed quickly and does not require funding.
the main study [10,19,20].. Specific arguments include
Nonetheless, we need to emphasize that whether or
not a study is a pilot, depends on its objectives. An I have funding for 10 patients only;
adaptive method is used as a strategy to reach that I have limited seed (start-up) funding;
objective. Both a pilot and a non-pilot could be adaptive. This is just a student project!
Thabane et al. BMC Medical Research Methodology 2010, 10:1 Page 4 of 10
http://www.biomedcentral.com/1471-2288/10/1
Table 2 Reasons for conducting pilot studies
Main Reason Examples
Process: This assesses the feasibility of the processes that are key to the Recruitment rates
success of the main study
Retention rates
Refusal rates
Failure/success rates
(Non)compliance or adherence rates
eligibility criteria
- Is it obvious who meets and who does not meet the eligibility
requirements?
- Are the eligibility criteria sufficient or too restrictive?
Understanding of study questionnaires or data collection tools:
- Do subjects provide no answer, multiple answers, qualified answers, or
unanticipated answers to study questions?
Resources: This deals with assessing time and resource problems that Length of time to fill out all the study forms
can occur during the main study
Determining capacity:
- Will the study participants overload your phone lines or overflow your
waiting room?
Determining process time
- How much time does it take to mail out a thousand surveys?
Is the equipment readily available when and where it is needed?
What happens when it breaks down or gets stolen?
Can the software used for capturing data read and understand the data?
Determining centre willingness and capacity
- Do the centres do what they committed to doing?
- Do investigators have the time to Perform the tasks they committed to
doing?
- Are there any capacity issues at each participating centre?
Management: This covers potential human and data management What are the challenges that participating centres have with managing
problems the study?
What challenges do study personnel have?
Is there enough room on the data collection form for all of the data you
receive?
Are there any problems entering data into the computer?
Can data coming from different sources be matched?
Were any important data values forgotten about?
Do data show too much or too little variability?
Scientific: This deals with the assessment of treatment safety, dose, Is it safe to use the study drug/intervention?
response, effect and variance of the effect
What is the safe dose level?
Do patients respond to the drug?
What is the estimate of the treatment effect?
What is the estimate of the variance of the treatment effect?
My supervisor (boss) told me to do it as a pilot. powered appropriately to assess statistical significance.
Further, there is a vast number of poorly designed and
None of the above arguments qualifies as sound rea- reported studies. Assessment of the quality of a pub-
sonsforcallingastudyapilot.Astudyshouldonlybe lishedreportmaybehelpfultoguidedecisionsof
conducted if the results will be informative; studies con- whether the report should be used to guide planning or
ducted for the reasons above may result in findings of designing of new studies. Finally, if a trainee or
limited utility, which would be a waste of the research- researcher is assigned a project as a pilot it is important
ers’ and participants’ efforts.Thefocusofapilotstudy to discuss how the results will inform the planning of
should be on assessment of feasibility, unless it was the main study. In addition, clearly defined feasibility
no reviews yet
Please Login to review.
