Scientists sound alarm on badly run medical studies

Most of the Covid ivermectin trials provide a case study in how not to do medical science. Photo: Diverse Stock Photos via Flickr (CC BY-NC 2.0)

• Scientists studied over 1,600 clinical trials that test medical interventions. They found that over 60% showed a high risk of bias, meaning we cannot have confidence in the findings of those trials.

• “No patient or member of the public should be in a bad trial and ethical committees, like funders, have a duty to stop this happening,” say the scientists.

• It is vital to include people with statistical expertise in trial design, they argue.

A new research paper has signalled a crisis in medical research: “over 60% of trials are so methodologically flawed we cannot believe their results”. Researchers estimate that 88% of trial spending is wasted.

Dodgy research design and bad statistical methodology mean that most randomised trials are a waste of time, money and effort, and of no or dubious scientific value, say Stefania Pirosca, Frances Shiely, Mike Clarke and Shaun Treweek, in a new paper published in the journal Trials in early June.

Their paper examined 1,659 randomised trials, involving 400,000 participants, that took place between May 2020 and April 2021 in 84 countries as well as 193 multinational trials.

The majority of trials (62%) showed a high risk of bias. More than half of trial participants were in these high risk of bias trials. Trials where the risk of bias was unclear accounted for 30% of those reviewed, while trials with a low risk of bias – those that can be trusted – accounted for just 8% of the total.

Bad trials – ones where we have little confidence in the results – are not just common, they represent the majority of trials in all countries and across most clinical areas. For instance, all trials looking at drugs and alcohol exhibited a high risk of bias. The most reliable field was anaesthesia, with 60% of trials exhibiting a low risk of bias.

The research team drew trial data from 96 reviews from 49 of the 53 clinical Cochrane Review Groups. Cochrane is an international organisation that helps to gather and propagate the results of medical research to better guide medical decision-making. This is done by experts compiling and evaluating research trials and results in “standardised, high-quality systematic reviews”.

Bad science was common everywhere. “No patient or member of the public should be in a bad trial and ethical committees, like funders, have a duty to stop this happening,” the paper’s authors write.

South Africa was bad, but Spain and Germany may be worse

In the seven trials reviewed that took place in South Africa, four had a high risk of bias, two had an unclear bias risk, and one trial was “good science”. This share of bad science is roughly similar to those found in the clinical trials done in the UK and USA. The most reliable health research science was done in multinational trials - with these, 23% of trials were a low risk of bias. (The authors didn’t identify the trials.)

The least reliable science, in countries that conducted 20 or more RCTs, was done in Spain and Germany, with 86% and 83% of the trials exhibiting a high risk of bias.

While results from just one year were interrogated, the paper’s authors found that their results map to similar studies, and that bad science can be expected to be the norm, over time.

This amounts to a massive waste of money and effort.

Statisticians and research method experts have been sounding the alarm on biased research for years, since Doug Altman’s 1994 paper in the British Medical Journal, “The scandal of poor medical research“.

Doctors want to know if they can rely on a particular treatment to produce a desired outcome, and need research that confers a degree of confidence. One way to do that – the most popular – is randomised control trials.

Randomised trials are great - but you need statisticians

Randomised control trials, also known as randomised trials, or RCTs, are for many (though not all), the gold standard for achieving scientific knowledge about a medical intervention – whether a drug or another type of therapy. The way that RCTs are conducted is crucial, as it is adherence to the method that gives people relying on the research confidence that the results produce scientific knowledge.

But, if there is a high risk that the results were biased by errors in how they were conducted and how results were achieved, they should not be relied on. Pirosca and colleagues did not examine the type (or domain) of bias in the studies, arguing that having a high risk in one type of bias is sufficient to undermine the trial’s results.

In short, for Pirosca and colleagues, health research in randomised trials is bad when there is an identifiable risk of bias in the way that the results were obtained.

The large number of high risk of bias trials appears to be due to “a lack of input from methodologists and statisticians at the trial planning stage combined with insufficient knowledge of research methods among the trial teams”. You would not, they say by analogy, think it appropriate that a statistician conduct surgery, just because they are doing work in a surgical domain.

Bad science during Covid

Recent medical scandals in the headlines have highlighted the risks of bad science in medicine. The Covid pandemic has brought a boom in medical research, and popular attention to the results of medical research. This environment has produced some remarkable science, but it has also created scientific fiascos, like the one that surrounded ivermectin.

As GroundUp has previously reported, a review of studies investigating ivermectin as a possible therapy for Covid initially suggested that the deworming drug led to better outcomes in those that used it. On the face of it, the small studies that supported this conclusion seemed to provide promise for a low-cost, life-saving Covid intervention. But once the methodology and statistics were looked at closely, many of these papers were deemed unscientific – for instance, patients were excluded from analysis for no good reason. And once these trials were excluded from the review, the drug’s promise as a Covid treatment vanished.

Medical research watchdog Retraction Watch currently lists 12 papers purporting to investigate ivermectin that were subsequently withdrawn or for which concerns have been expressed. According to their records, 235 Covid papers have been withdrawn to date.

But the crisis is not insurmountable. Pirosca and colleagues say that relatively simple fixes would dramatically reduce the amount of untrustworthy health research – by ensuring that methodological principles that underlie RCTs are not compromised.

More expenditure on statistical expertise will save money

A 2015 review examined 142 trials that exhibited a high risk of bias. The authors found that in half of the high risk trials, the methodological adjustments required to reduce the risk of bias would have been low or zero cost. Easy adjustments at the design stage would have made important improvements to 42% of trials that exhibited high risk of bias.

Pirosca and colleagues propose that no medical RCT should be funded or given ethical approval if it cannot prove that the team conducting the trial has a member that has methodological and statistical expertise. Every RCT should, in its design, use risk of bias tools to make sure that results are not compromised.

The expertise that could restore the worth to medical research is in short supply.

More methodologists and statisticians are needed, and money should be invested in training people with this expertise, and investing in applied methodology research and supporting infrastructure. The authors call for 10% of a funder’s budget.

This might seem like a lot of money, but, argue Pirosca and co, it would be a fraction of the cost of the wasted research in the year under review – estimated to be billions of rands.

The task is urgent: “Randomised trials have the potential to improve health and wellbeing, change lives for the better and support economies through healthier populations … Society will only see the potential benefits of randomised trials if these studies are good, and, at the moment, most are not.”

Donate using SnapScan.