colony-health-hero

The topic of pilot studies or proof of concept studies is rarely taught in healthcare education. In addition, they tend to be largely overlooked by the scientific community. Yet, they can be quite important from the point of view of a healthcare startup looking to introduce a new product to the market in terms of saving wasted time and money on larger studies in the long run. In this report, we will attempt to define this process, offer 5 important reasons to invest in a pilot study as a way of revealing problems that may arise during a full-scale study, and how to use it to direct a larger pre-market study.

What is a pilot study?

In the healthcare field, a proof of concept study, or pilot study, is an exploratory test or preliminary trial intended to guide further development of a drug, device, or product. It is designed to occur before a large scale study in order to assist in the preparation of a more comprehensive examination. Think of it as a practice run where the researchers are able to test their methods and study design ahead of time and make necessarily changes before embarking on a more extensive study.

The elements most commonly examined during a preliminary study include: the procedures involved, tools/instruments, recruitment rate, and estimated sample size required for statistical power in a full-scale study. In order to optimize this process and reduce cost and time to reach market for new devices, the purpose of the pilot study must be clear and established from the outset.

Why conduct a pilot study?

A pilot study can be a valuable way to detect any weaknesses in study design and refine the study processes before continuing forward with a larger, more expensive study. Here are five important reasons to include a pilot study in your research plan: [1]

1. To refine your study process:

A proof of concept study can examine the process in question in order to determine the feasibility of the study’s methods and process before continuing with a large-scale study. It is crucial to determine the necessary sample size prior to beginning a study. Failure to recruit sufficient numbers in a trial will reduce the statistical power, and one of the main reasons for abandoning trials early.[2]
Other aspects of a study’s process that may be tested during a pilot study include:

  • Estimate of recruitment, retention, and refusal rates
  • Estimate sample size
  • Test randomization and blinding procedures
  • Determine eligibility criteria
  • Examining data collection tools such as participant surveys to assess whether they are effective in gathering participant responses.

2. To determine what resources your study will require in order to proceed in an efficient manner and to give you meaningful results.

Resource studies include the time and budget concerns that may arise during a larger study. For example:

  • Time for participants to fill out and submit any required forms
  • Investigators availability and schedule requirements
  • Whether the study site can accommodate the flow of participants
  • How long the study will take for each person to complete
  • Is the proper equipment available and in good working order?
  • How to manage technical glitches, equipment failure/maintenance
  • Time and money required for follow-up calls, mailing surveys, etc.

3. To be prepared for any management concerns.

This area deals with the potential personnel and data management issues such as:

  • What are our personnel needs?
  • How will the data be collected?
  • Where will the data be stored?
  • What are our proposed data analysis techniques?

4.To address crucial scientific questions that must be addressed:

Before proceeding with a study involving a medical device or new drug, several scientific or drug safety questions must be addressed:

  • Is the proposed intervention safe?
  • If a drug is being studied, what is the safe dosage?
  • What are potential side effects?
  • What is the expected treatment effect?

5. To convince potential investors or funding bodies that a large-scale study is worth supporting.

A properly executed pilot study can be invaluable in terms of demonstrating the potential of a medical device or drug and earning the required funding for further study.

What can I expect from the results of a pilot study?

The conclusion reached at the end of a pilot study is typically one of the following:

  • Stop- the main study will not be feasible
  • Continue with the main study, but modify the protocol
  • Continue with the main study as is, with close monitoring for necessary changes
  • Continue with the main study as is- the pilot study is determined that it is likely to succeed

What are potential pitfalls and limitations of a pilot study?

Several researchers have attempted to take a closer look at pilot studies in order to gather information on their common features and to gain insight into their potential limitations.

Statistical Power

Keep in mind the statistical power of a pilot study: A successful pilot study does not assure success on a larger scale simply because a pilot study is relatively small and cannot have the statistical power to guarantee success. However, a well-designed pilot study can make a significant difference in increasing the chance of success in a full-scale examination. Main investigators must keep in mind that the main function of the pilot study is not to make a determination of statistical significance, but instead to examine the feasibility of their study.

Data Contamination

Data contamination can occur when pilot study data is combined with the main results or when the same study participants are used in the pilot study then new data obtain from them for the main study. However, the data may be pooled if the methodology and sampling frame remain the same[1].

Final thoughts

A pilot study can be an excellent opportunity to examine the feasibility of a large-scale study and address many weaknesses in the study design before embarking on an expensive full-scale study. In order to optimize the pilot study design, researchers recommend having clear feasibility objectives from the beginning, solid analytic plans, and well-established criteria for success of feasibility. Overall, it is important to keep in mind that the main function of a pilot study should not be to test a critical hypothesis, but rather to optimize the design of subsequent pivotal trials.

References

  1. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Goldsmith CH: A tutorial on pilot studies: The what, why and How. BMC Medical Research Methodology. 2010, 10: 1-10.1186/1471-2288-10-1.
  2. Ross S., Grant A., Counsell C., Gillespie W., Russell I. & Prescott R. (1999) Barriers to participation in randomized controlled trials: a systematic review. Journal of Clinical Epidemiology 52, 1143–1156.