Unloading the dice on research
A compulsory register of trials could give a more accurate view of studies and test results
I just rolled a six-sided die a few times. I rolled: six, five, five, six, five. My question is: do you think the die is biased? One way to think about that question is to ask how likely I would be to roll only fives and sixes completely by chance. Not that likely: the odds are 242 to 1 against.
Before you conclude that I have crooked dice, let me mention something that had slipped my mind. As well as those fives and sixes, I also rolled four, three, three, three, two, two and one. But those results just didn’t seem that interesting, so I didn’t tell you about them. If that omission seems relevant, you’re beginning to appreciate the importance of the nerdy-sounding “trial register”.
Every day, across the world, researchers are conducting randomised controlled trials (RCTs). Some are rigorous, painstaking quests for truth, while others may cut a few corners in the search for a career-defining publication, or a licence for a new drug compound. But even if each individual trial was unimpeachable, the results would mean very little if there was a systematic bias in favour of a particular kind of result.
I reported my die-rolling with a bias towards high numbers. You might uncharitably suspect that an industry-sponsored drug trial would be more likely to see the light of day if the results show that the drug works, and you would be correct (a state of affairs ably summarised by Ben Goldacre in his new book Bad Pharma). Trials can also go missing because they end in chaos or disaster: such stories may not be worth an academic paper but they must be recorded. And trials go missing because the results are so boring that the researchers cannot bring themselves to write them up properly for publication.
As my die-rolling shows, unless we see every trial that was begun, we have a distorted picture of what is happening. There is probably only one way to achieve that goal: a compulsory register of trials. Researchers who conduct trials and abandon them or don’t publish, need to become pariahs. Systematic reviews of a particular field will be able to consult the registers and track down any unpublished trials.
A few years ago, the International Committee of Medical Journal Editors announced that the prestigious journals under its control would no longer publish research based on clinical trials unless those trials had been formally registered before they began. This had the very welcome effect of increasing, dramatically, the number of registered trials and the rate at which new trials were registered. Unfortunately, as Sylvain Mathieu and others explained in the Journal of the American Medical Association in 2009, more than half the research they examined flouted that rule and was published anyway. The threat not to publish seems to have been empty.
And what of economics, which has in recent years discovered the joys of randomised trials? There is good news: the American Economic Association is creating a trial register for trials in economics; it is due to be up and running next year. The register will be voluntary for now, but two leading practitioners, Esther Duflo of MIT and Dean Karlan of Yale, both told me they are hopeful that a strong social norm will form in favour of registering trials. We shall see.
Trial registries are a particular challenge in the social sciences. While an RCT in medicine is designed to test whether a specific treatment does or does not work, in the social sciences they are more likely to be used to search for interesting hypotheses. Social science RCTs are often partnerships between academics and practical organisations, and the trial may evolve over time in a way that a clinical trial would not.
All this complicates the business of registering the trial and then updating the entry as things change. It makes a trial registry harder to maintain. But it also makes the registry even more essential.
Also published at ft.com.
7 Comments
Ken Taylor says:
The dice trial, properly reported would include the missing data. To do otherwise is unethical. It is ethics that causes research to be properly reported and ethics can not be guaranteed by procedure. Just like what happens so often with ethics committees, the register just becomes a another bureaucratic shield for the unethical to hide behind.
There is an infinity of things that turn out not to be interesting. Trying to record uncompleted research is too hard. If it’s written up well enough to usefully record then it’s completed research and deserves publication.
4th of November, 2012Kyle Lippincott says:
Ken: I believe that the individual dice rolls were stand-ins for separate trials, not doctored results of one trial. That is, the 5s and 6s were each synonymous with one trial that had results the publication wanted, and the 1-4s did not. In a set of drug trials, you can imagine that each 5 mentioned is a single trial (with its own set of test subjects) that came to the conclusion “drug shows 20% improvement over placebo” and each 6 is a single trial that came to the conclusion “drug shows 40% improvement over placebo”.
Your argument is that this is one trial, which I totally agree with you – if this were a single trial and data was just dropped to get the results that the author wanted, then that’s unethical and a huge problem.
If, however, they’re individual trials that were started but abandoned (because they were coming up with data that the sponsor didn’t like, and so they didn’t publish, or any other number of reasons), then anyone reading about all of the published trials has a skewed view. If there are 20 trials funded by a drug company to prove how safe and effective their drug is, and 16 of them prove that it’s just as safe as, but just as effective as placebo, then they likely won’t publish those. So we’re down to 4-6 trials (maybe they published a couple of the inconclusive ones to try to not look suspicious), and whatever they say. Now anyone looking at this will come to the conclusion that 4 out of 6 trials agree: this drug is awesome! But the proper conclusion is that 4 out of 20 trials say it’s awesome. Quite a bit different.
4th of November, 2012Tim Harford says:
Ken – Kyle has the correct interpretation of what I meant. Sorry that was unclear. Point taken about bureaucracy.
As for whether there is an important difference between doing one trial and suppressing some data, or doing several trials and suppressing whole trials – hm. The practical or ethical distinctions are not obvious to me but they seem to be taken for granted by most people. I am not sure why.
4th of November, 2012Kyle Lippincott says:
I think regarding one trial with suppressed data and multiple trials while suppressing the trial, it’s a matter of the intent of the deception. If I start a trial and its results are boring or not what I expected to show, I have very little incentive to publish it (regardless of any sponsorship of the trial).
10 separate and unassociated parties not publishing because they didn’t find the results interesting isn’t that odd or unethical to me (and while it’s unfortunate for anyone attempting to gain a globally accurate viewpoint on the subject by reading the research, it’s hard to proclaim any unethical behavior on the part of the individual trial authors).
10 parties not publishing because they were forbidden from doing so by their sponsor(s) since the results didn’t say what the sponsor(s) wanted is concerning. Here we have a problem with an entity/entities acting out of intent to deceive, which is where the problems come in, in my mind.
ANY parties publishing factually incorrect or intentionally selective studies is a huge problem, since now anyone looking to research performed on this subject is given lies.
It’s really the intent behind the omission, vs. the scale of the omission. If I’m producing a paper, and basing it on the results of other papers, but I ignore the ones that disagree with me, I’m just as compromised ethically as if I were performing a single trial and ignored the 1-4s. If I’m funding research and forbid the money from going to trials that disagree with my desired results, I’m in the same situation. If I’m performing a clinical trial and my results are inconclusive or very uninteresting (for whatever reason) and decide not to publish.. well, that’s the whole point of the registry. If I’m writing a paper, look at all other published works, and conclude that my original hypothesis is not supported by the other research performed, and thus don’t write my paper, am I in an ethically ambiguous situation? I don’t believe so.
4th of November, 2012Margaret Wilde says:
When drug trial results are suppressed because they show that the drug is ineffective or has serious side-effects this is misleading for prescribers and potentially catastrophic for the patients/victims who eventually take the drug. So many big pharmaceutical companies have been guilty of this and other sorts of fraud and although they often pay out huge sums in ‘compensating’ victims and as fines, the shattered lives and the great suffering entailed by those damaged by the drugs are vastly under-reported and receive much less consideration than they merit.
The ‘rule’ that companies should have as a primary consideration the maximising of profit is often used as a justification for keeping secret results unfavourable to sales and profits, and is regarded as normal business practice and sort-of legal, is my understanding. I think that the ‘rules’, regulations, laws, or whatever they are should be changed so that suppressed and misleading drug information resulting in avoidable harm to patients be categorised as a criminal offence (not a corporate criminal offence) of the CEOs who head these companies. They should be tried in the criminal courts for GBH or manslaughter or whatever the appropriate legal category and then be sentenced as individuals to terms of imprisonment or whatever the judge decides appropriate.
I write as a person whose life and health have been gravely damaged by prescribed drugs and like everyone in that situation I want to save others from such suffering.
8th of November, 2012Brett Keller says:
Tim — was hoping you’d connect this to the ongoing scandal re: Roche, Tamiflu, and the BMJ. The BMJ is instituting a new open data policy for RCTs, in part because it now appears that Roche has for several years withheld unfavorable trial data of Tamiflu, even after the US and European governments spent billions of dollars stockpiling it. (See http://www.bmj.com/content/345/bmj.e7305) Just one more example of why publish ALL results is important, and individuals and organizations often don’t play by ethical rules unless compelled to do so by rules.
9th of November, 2012Stephen says:
The probability of rolling a 5 OR a 6 on a single trial is one-third. By stating the probability to be 1 in 243 it is implied that the 5 dice rolls constitute a single experiment.
9th of November, 2012