The COVID-19 pandemic has led to an increased interest among news outlets and the public in the advancement of clinical trials. Results from studies are sometimes presented as headline news, but unfortunately often with little reflection or critical analysis. Here we try to explain some of the basic concepts and terms, and reflect on how study design affects what conclusions can be drawn from the studies.
Types of trials
Prospective vs. retrospective
A clinical trial is by definition prospective. It looks ahead in time, intending to evaluate a new drug, treatment or diagnostic. The approach allows for a level of control by defining the treatments that the patients receive, which data are collected and how, and blinding. New drug trials are naturally prospective, albeit some trials may use a historical control group, see below.
In contrast, a retrospective study looks back in time, for example by reviewing medical records and studying disease characteristics or patient outcomes after the fact. Retrospective and historical data have the drawback that they are difficult to “control” adequately. Sample and data collection and management practices may not be defined and handled with sufficient stringency, and patient selection may introduce more variability and unknowns, as diagnostic methods and criteria shift over time as knowledge and technology evolves.
Observational vs. experimental
An observational study does just what the name states – observes without intervening. The subject of observation may of course vary. One study could for example observe the natural progression of a disease to which there is no therapy, whereas another may study the effectiveness of different standard therapies.
An experimental study, in contrast, intervenes in patient care. The study seeks to investigate if a new therapy or diagnostic can give a better outcome than an established treatment. It intervenes by adding, changing or withdrawing an element of the standard therapy. The effect of the new or modified therapy is then compared with the standard therapy to assess which one is better (superiority study) or if they are equal (non-inferiority study) in effect.
Control groups are instrumental in good clinical research – without a control group, there is nothing to compare the new therapy with and thereby any conclusions from the study are inherently uncertain. A good control group should be matched with the treatment group on a number of parameters that are relevant for the disease or treatment. These parameters can include for example age, sex, socioeconomic status or disease severity. Generally the matching is done on the population level so that both groups have a similar distribution of the relevant parameters.
A special case is the matched pair setup, where each patient in the treatment group is matched with a patient in the control group that has similar characteristics, for example both are female, aged 42 and have the same disease severity score. All patients should be randomized, or randomly assigned to each group to avoid bias in the selection. Also, the control group may or may not receive a placebo therapy. In some studies, a historical control group is used. This means that the control group is not studied at the same time as the treatment group, but have been studied at an earlier point in time. However, historical controls are difficult to match as research methods and standards of care evolve continuously.
The placebo effect
The placebo effect can be described as the effect of anticipation. Patients may experience both positive and adverse effects of an inactive formulation, such as an injection of saline or a pill without active substance. Some studies have noted that placebo drugs that are supposedly newer and more expensive have a larger effect than placebo drugs that are old and cheap.
Blinding vs. open label
Blinding is also a measure to reduce bias from expectations and can take place on multiple levels. Patients may be blinded – both the treatment and control groups receive the same tablets or injections, but they are not told if they receive the study drug or the comparator. On the next level, all the staff attending to the patients may be blinded, so that they do not know what they are giving the patients. The blinding can also be expanded to also cover those that evaluate the results and analyze data to further reduce the risk of bias during the study. An open-label study is not blinded at all, both the patients and researchers know which treatment is given.
The endpoints, both primary and secondary, should be predefined in the study protocol and not changed after the study has started. Endpoints should be relevant for the disease, patient population and the anticipated effect of the drug. For example, a study evaluating the effect of a new drug against influenza should not have mortality as a primary endpoint, as the mortality rate is relatively low and requires a large study population to see an effect. Similarly, a trial for a new treatment of COVID-19 in a patient population with mild to moderate disease severity may not be feasible to perform with a mortality endpoint. The problem, however, is that a study using length of hospital stay as the endpoint does little to establish any life-saving effect of the new drug.
A hallmark of scientific research is publication with peer review, where the manuscript is assessed by experts on the subject matter that are independent of the researchers performing the study. The peer review process can be viewed as an external quality control, checking that the methods used are relevant and that the conclusions drawn are valid. In the fast pace of modern research, and especially in times of an ongoing pandemic, some papers are published as pre-prints prior to review on pre-print servers like bioRxiv, which might impact negatively on their quality.
Research in a hurry
Clinical research is complex with a multitude of study designs and analyses available. This makes it difficult to assess the strengths and weaknesses of studies even in the best of times. In the worst of times, like in the urgency created by a pandemic, assessment may be even more difficult, but also more important. When studies are designed and published in a hurry, the risk of introducing errors and bias increases as checks and balances are prioritized lower.
Checklist for assessment of a clinical study
Is the study:
- Well designed?
– Well controlled with relevant control groups and placebo
– Blinded, at least double-blinded, for participants and clinical staff
- Clinically relevant?
– Relevant patient groups
– Relevant endpoints
- Correctly analyzed and reported?
– Prespecified protocols and analyses
– Correctly selected analyses
– Significant results, both statistically and in effect size
– Peer reviewed in a reputable journal or without review as a pre-print
Learn more in the ReAct Toolbox
More news and opinion
- 4 key reflections on the recently launched WHO GLASS-report
- Key points from ReAct’s comments to the Independent Panel on Evidence
- ReAct Interview: From zoologist to community engagement on AMR
- ReAct Africa expands
- COVID-19 resolution – a missed opportunity to address global pandemic response more broadly
- What everyone needs to know about clinical research
- New ReAct Policy Brief: Successful cancer treatment relies on effective antibiotics
- Impact of COVID-19 on vaccine-preventable diseases and antibiotic resistance
- ReAct Africa and Africa CDC: COVID-19 webinars
- Antibiotic pollution: India scores a global first with effluent limits
- COVID-19 and AMR – what do we know so far?
- Learning from bedaquiline in South Africa – comprehensive health systems for new antibiotics
- ReAct Interview: How does antibiotics in food animal production end up in the environment?
- Key take aways from CSO workshop on AMR in Kenya
- New fact sheet: Effective antibiotics – essential for childrens’ survival
- Shortages and AMR – why should we care? 4 consequences of antibiotic shortages
- Our microbiome and noncommunicable diseases
- The 2020 AMR Benchmark Report – concerning findings with questionable framing
- 4 key reflections from engaging hospitals in India for antibiotic stewardship
- Teacher Gustavo Cedillo, Ecuador, teaches children about the bacterial world