Solving the Reproducibility Crisis: Immutable Data Infrastructure for Science

Start with what the number actually says

The figure everyone quotes is $28 billion a year. It comes from Freedman, Cockburn and Simcoe, writing in PLOS Biology in 2015, and it is worth being precise about its scope, because it is routinely stretched past what the authors claimed.

It covers US preclinical life-science research only, and it is a multiplication rather than a measurement: about $56.4 billion of annual US preclinical spend, times an estimated irreproducibility rate of roughly 50%.

That rate is where the uncertainty lives. The authors derive it from four error categories - study design, biological reagents and reference materials, laboratory protocols, and data analysis and reporting - and give it as 53.3% with bounds of 18% to 88.5%. Carry those bounds through the same multiplication and the honest interval is something like $10 billion to $50 billion. The $28 billion everyone quotes, including the earlier version of this post, is the midpoint of a very wide range.

Two further things the paper says and its citations usually do not. The prevalence rates feeding the estimate come from a small number of studies that do not agree on what "irreproducible" means, and in places extrapolate from clinical settings because no preclinical data existed. And irreproducible is not a synonym for wasted or wrong - a result that has not been reproduced may simply not have been retested.

It is still a large number. It is not a precise one, and it does not mean what "$28 billion wasted on bad science" implies.

The survey most worth reading alongside it is Monya Baker's, published in Nature in 2016. Of 1,576 researchers, more than 70% had tried and failed to reproduce another scientist's experiment, and more than half had failed to reproduce one of their own. Fifty two percent judged the situation a "significant crisis."

Read it with its limits in view: it is a News Feature rather than a peer-reviewed study, and the questionnaire was self-selected online rather than a probability sample, so the respondents are people who cared enough to answer. Baker also reports a tension worth sitting with - fewer than 31% thought a failed reproduction means the original result was probably wrong.

That last statistic is the interesting one, and it constrains everything that follows. If researchers cannot reproduce their own work - with full access to their own data, their own code and their own lab - then the problem cannot be mainly about data being unavailable to strangers.

Which causes infrastructure can actually reach

When Baker asked what contributes to irreproducibility, the answers that topped the list were selective reporting, pressure to publish, and low statistical power. Further down came unavailable methods, unavailable raw data, and insufficient technical expertise.

That ordering is the whole argument of this post, so I want to state the uncomfortable half plainly:

Content addressing does nothing about the top of that list. A study that was underpowered is underpowered whether or not its data has a permanent identifier. Hypothesising after results are known does not become visible because the dataset is on IPFS. p-hacking survives immutability perfectly well. Anyone telling you decentralised storage solves the reproducibility crisis is selling something, and it is worth noticing that the previous version of this article did exactly that.

What infrastructure can reach is the lower band, and that band is not trivial. It is the difference between a reproduction attempt that fails informatively and one that never starts.

The failures that are addressable

"The data is gone." Vines and colleagues chased the data behind 516 papers published between 1991 and 2011 and found the odds of a dataset still existing fell about 17% per year - from essentially all of the two year old papers to a third of the twenty year old ones. Two caveats matter: that rate is measured only among authors who replied and said what had become of their data, and the study deliberately covers one narrow data type across plants and animals rather than science generally.

The broader picture comes from Tedersoo and colleagues, who went after the data behind 875 papers in Nature and Science across nine disciplines. Full data were available for 54.2% of articles before contacting anyone, and of the direct requests they sent for the rest, 41.3% got no reply at all despite reminders. "Available on reasonable request" turns out to describe an intention rather than a mechanism.

A content identifier does not by itself fix any of that - somebody still has to host the bytes. What it changes is the shape of the problem: from "find whoever has it and hope they answer" to "fetch these bytes from whoever is hosting them", and the second is a question with an engineering answer.

"Which version was this?" A methods section citing final_dataset_v2.csv is citing a filename. If a correction was applied between analysis and publication - and corrections are normal, honest events - there is no way for a later reader to establish which state produced the figures. A content identifier is the version, so the question stops existing:

# The dataset as analysed
curl -X POST https://api.ekayana.com/api/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@experiment_data.csv"
# -> { "cid": "bafkrei...", "size": 4210318, ... }

Correcting a data entry error produces a different CID rather than overwriting the old one. Both remain citable, and a DID document can record the relationship between them, so "we fixed row 47" is a statement a reader can verify instead of one they have to take on trust.

"The environment differed." This one is solved outside our stack and worth saying so. Container registries already support content addressing; pinning a base image by digest rather than by tag is a one line change that most projects still do not make:

# Tags move. Digests do not.
FROM python:3.11-slim@sha256:1e8a1a3d6f2bd1cfb8dc4b1eba8b1f8f0f2ba9a6f1cd1b2a3c4d5e6f7a8b9c0d

# Refuse to install anything whose hash was not pinned in the lockfile
COPY requirements.lock .
RUN pip install --require-hashes -r requirements.lock

FROM python:3.11-slim without the digest is not reproducible; the same Dockerfile built six months apart yields different images. The fix costs nothing and is independent of any of the rest of this.

Provenance is the part worth building

If you record the identifiers of every input, the code, the environment and the outputs, and you sign that record, you get something a reader can act on: a claim about what produced what, that fails loudly when it is wrong.

{
  "@context": "https://w3id.org/ro/crate/1.1/context",
  "@type": "Dataset",
  "identifier": "did:bio:devnet:FWePP4F42jYMNguSBjVsTszizNufYrwKkmN7BAZ1sMS6",
  "inputs": {
    "raw": "bafkreid2appi7dgxybpite5mlbtz65lb7t6bfksdjqal4q7edbk3qv4wjy",
    "calibration": "bafkreigvbdydplcyarvmskvlevkmbb2da3efrfr7xogtc4ezceqkxtvd3m"
  },
  "software": {
    "repository": "https://github.com/example/analysis",
    "commit": "8f14e45fceea167a5a36dedd4bea2543",
    "image": "sha256:1e8a1a3d6f2bd1cfb8dc4b1eba8b1f8f0f2ba9a6f1cd1b2a3c4d5e6f7a8b9c0d"
  },
  "randomSeed": 42
}

Note what this record does not claim. It does not say the analysis was appropriate, the sample size adequate, or the conclusion supported. It says: run this code, in this environment, over these exact bytes, and you should get these exact outputs. That is a narrow guarantee, and narrow guarantees are the ones that hold.

We do not score reproducibility

An earlier draft of this post showed an API returning a "reproducibility score" of 98 out of 100, computed from a weighted rubric. No such endpoint exists, and we are not going to build one.

The reason is not effort. It is that a single number would be immediately gamed and would launder a judgement call as a measurement. Whether a study is reproducible is a question about methods, and methods are assessed by people who understand the field. What a platform can honestly report is factual and checkable: whether the inputs resolve, whether the recorded digests match, and whether the provenance chain is complete. Those are three yes-or-no questions, and they are worth more than a score precisely because nobody can argue about what they mean.

What this is worth

Infrastructure sets a floor. It makes it possible to establish what was actually run, and it stops a category of silent failure - the substituted file, the ambiguous version, the dataset that quietly stopped existing. That floor did not exist before and it is worth having.

It does not raise the ceiling. The ceiling is set by study design, statistical power, and whether the incentive structure rewards being careful, and those are not problems that a hash function is going to touch. The honest framing is that this work removes a class of excuse. Once the data reliably resolves and the environment is pinned, a failed reproduction means something specific about the science - which is exactly the conversation that has been hard to have.

References: Freedman, Cockburn & Simcoe, "The Economics of Reproducibility in Preclinical Research", PLOS Biology 13(6):e1002165 (2015) · Baker, "1,500 scientists lift the lid on reproducibility", Nature 533:452-454 (2016) · Vines et al., "The Availability of Research Data Declines Rapidly with Article Age", Current Biology 24(1):94-97 (2014) · Tedersoo et al., "Data sharing practices and data availability upon request differ across scientific disciplines", Scientific Data 8:192 (2021)