I’m not a big fan of reproducibility projects. Shoddy papers shouldn’t be tolerated, but the truth is that sometimes rigorously done research isn’t reproducible — and when that happens, science gets interesting. It should go without saying that a peer-reviewed paper isn’t a guarantee of truth. If done properly, a paper is a record of a rigorous attempt to discover something about the world, no more, no less. What we believe about nature should reflect the accumulated evidence of many researchers and many papers, and that means the scientific literature should reflect our latest tentative, bleeding-edge thinking, even at the risk of being wrong. It’s counterproductive to hold up publication until some other lab reproduces your result, or to retract papers that don’t hold up, unless they had clear methodological flaws or artifacts that should have been caught in review.

Two recent articles capture what I think is the right attitude on reproducibility. First, as David Allison and his colleagues write, as a community of researchers, editors, and reviewers, we’re not doing as well as we should be when it comes to meeting high standards for best statistical and other methodological practices:

 In the course of assembling weekly lists of articles in our field, we began noticing more peer-reviewed articles containing what we call substantial or invalidating errors. These involve factual mistakes or veer substantially from clearly accepted procedures in ways that, if corrected, might alter a paper’s conclusions.

There is no excuse for this kind of sloppiness.

On the other hand, here is Columbia’s Stuart Firestein:

The failure to replicate a part or even the whole of an experiment is not sufficient for indictment of the initial inquiry or its researchers. Failure is part of science. Without failures there would be no great discoveries.

So yes, let’s clean up science by rooting out obvious “invalidating practices” that all too often plague papers in journals at all tiers. But let’s not be naive about how science works, and what the scientific literature is supposed to be. To paraphrase what  I wrote recently, if some of our studies don’t turn out to be wrong, than we’re not pushing hard enough at the boundaries of our knowledge.

I’ve always thought the Reproducibility Project represented an incredibly naive approach to the scientific method. This excellent news piece in Science sums up many of the reasons why. As Richard Young says in the piece, “I am a huge fan of reproducibility. But this mechanism is not the way to test it.” Here’s why:

1) Reproducibility in science is not achieved by having a generic contract research organization replicate a canned protocol, for good reason: cutting edge experiments are often very difficult and require specialized skills to get running. Replication is instead achieved by other labs in the field who want to build on the results. Sometimes this is done using the same protocol as the original experiment, and sometimes by obtaining similar results in a different system using a different method.

2) For this reason, I don’t have much confidence that the results obtained by the Reproducibility Project will accurately reflect the state of reproducibility in science. A negative result could mean many things — and most likely it will reflect a failure of the contract lab and not an inherent problem with the result. Contrary to the claims of the projects leaders, the data produced by the Project will probably not be useful to people who are serious about estimating the scope of irreproducibility in science. At its worst, it could be extremely misleading by painting an overly negative picture of the state of science. It’s already been damaging by promoting a too-naive view of how the process of successful science actually works.

3) As the Science piece points out, there is a much better, cheaper, and scientifically sensible way to achieve better reproducibility. If many papers out there are suspect because they lack proper controls, don’t use validated reagents, fail to describe methods adequately, or rely on flawed statistics, then we don’t need to spend millions of dollars and thousands of hours of effort trying to repeat experiments. We need to make sure editors and reviewers require proper controls, reagents, statistics, and full methods descriptions.

It’s worth reading the full article, but below the fold are some salient quotes:

[Richard Young] says that if the project does match his results, it will be unsurprising —the paper’s findings have already been reproduced. If it doesn’t, a lack of expertise in the replicating lab may be responsible. Either way, the project seems a waste of time, Young says. “I am a huge fan of reproducibility. But this mechanism is not the way to test it.”

I like the concept,” says cancer geneticist Todd Golub of the Broad Institute in Cambridge, who has a paper on the group’s list. But he is “concerned about a single group using scientists without deep expertise to reproduce decades of complicated, nuanced experiments.”

Early on, Begley, who had raised some of the initial objections about irreproducible papers, became disenchanted. He says some of the papers chosen have such serious flaws, such as a lack of appropriate controls, that attempting to replicate them is “a complete waste of time.” He stepped down from the project’s advisory board last year.

Amassing all the information needed to replicate an experiment and even figure out how many animals to use proved “more complex and time-consuming than we ever imagined,” [proejct leader] Iorns says.

For many scientists, the biggest concern is the nature of the labs that will conduct the replications. It’s unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say. Often a graduate student has spent years perfecting a technique using novel protocols, Young says. “We brought together some of the most talented young scientists in the area of gene control and oncology to do these genomics studies. If I thought it was as simple as sending a protocol to a contract laboratory, I would certainly be conducting my research that way,” he says.

Academic labs approach replication differently. Levi Garraway of the Harvard University–affiliated Dana-Farber Cancer Institute in Boston, who also has two papers on the project’s list, says that if a study doesn’t initially hold up in another lab, they might send someone to the original lab to work side by side with the authors. But the cancer reproducibility project has no plans to visit the original lab, and any troubleshooting will be limited to making sure the same protocol is followed, Errington says. Erkki Ruoslahti of the Sanford-Burnham Medical Research Institute in San Diego, California, has a related worry: The lab replicating one of his mouse experiments will run that experiment just one time; he repeated it two or three times.

The scientists behind the cancer reproducibility project dismiss these criticisms.

In my latest Pacific Standard column, I take a look at the recent hand-wringing over the reproducibility of published science. A lot of people are worried that poorly done, non-reproducible science is ending up in the peer-reviewed literature.

Many of these worries are misguided. Yes, as researchers, editors, and reviewers we should do a better job of filtering out bad statistical practices and poor experimental designs; we should also make sure that data, methods, and code are thoroughly described and freely shared. To the extent that sloppy science is causing a pervasive reproducibility problem, then we absolutely need to fix it.

But I’m worried that the recent reproducibility initiatives are going beyond merely sloppy science, and instead are imposing a standard on research that is not particularly useful and completely ahistorical. When you see a hot new result published in Nature, should you expect other experts in the field to be able reproduce it exactly?

Not always. To explain why, I’ll hand the mic over to Chris Drummond, a computer scientist and research officer at Canada’s National Research Council:

“Replicability is not Reproducibility: Nor is it Good Science” (PDF)

At various times, there have been discussions arising from the inability to replicate the experimental results published in a paper… There seems to be a widespread view that we need to do something to address this problem, as it is essential to the advancement of our field. The most compelling argument would seem to be that reproducibility of experimental results is the hallmark of science…I want to challenge this view by separating the notion of reproducibility, a generally desirable property, from replicability, its poor cousin. I claim there are important differences between the two. Reproducibility requires changes; replicability avoids them. Although reproducibility is desirable, I contend that the impoverished version, replicability, is one not worth having.

Drummond goes on to explain:

A critical point of reproducing an experimental result is that irrelevant things are intentionally not replicated. One might say, one should replicate the result not the experiment…The sharing of all the artifacts from people’s experiments is not a trivial activity.

In practice, most of us implicitly make Drummond’s distinction between replication and reproduction: we avoid exact replication when it isn’t absolutely necessary, but we are concerned about reproducing the general phenomena in our particular system.

And sometimes well-done research won’t be very reproducible, because it’s on the cutting edge, and we may not understand all of the relevant variables yet. You see this over and over in the history of science – the early days of genetics and the initial discoveries of high energy rays come to mind here. Scientists should do careful work and clearly publish their results. If another lab comes up with a different result, that’s not necessarily a sign of fraud or poor science. It’s often how science makes progress.