AI Papers Became Far More Open Over a Decade

The most encouraging change in artificial-intelligence research may be less about what models can do and more about whether other people can inspect the work behind them.

A new peer-reviewed analysis examined 56,800 papers published from 2014 through 2024 at five major AI conferences: AAAI, ICLR, ICML, IJCAI and NeurIPS. Among the 52,328 papers the researchers classified as empirical, the share that documented both open code and open datasets rose from 11 percent in 2014 to 64 percent in 2024.

That is a meaningful shift in the infrastructure of knowledge. A result is more useful when another team can see the implementation, obtain the data and understand how the experiment was run. But openness is not the same as successful reproduction—and the study is careful about that boundary.

Research references: [1]

A decade of more inspectable work

The research team evaluated seven documentation signals: open code, use of open datasets, specified dataset splits, pseudocode, hardware details, software dependencies and experiment setup.

The changes were broad. Open-code availability rose from 13 percent of empirical papers in 2014 to 69 percent in 2024. Use of open datasets increased from 68 percent to 91 percent. The share documenting at least five of the seven signals rose from 8 percent to 43 percent.

Those improvements occurred while the field expanded rapidly. The five conferences published 1,206 papers in 2014 and 12,026 in 2024. Better documentation therefore did not come from a smaller or static research community; it happened while publication volume grew roughly tenfold.

This matters because AI results can be unusually sensitive to details that do not fit neatly into a headline: dataset splits, leakage, random seeds, dependencies, compiler settings, hardware and hyperparameter choices. Sharing code and data cannot eliminate those problems, but it removes barriers that otherwise force independent researchers to reconstruct crucial parts of an experiment.

The headline limit: this was not a mass replication project

The study did not rerun 56,800 papers.

Instead, it used large language models to classify documentation practices at scale. The authors optimized prompts against an existing manually annotated dataset and manually evaluated a random sample of 160 papers. Most classification categories exceeded 90 percent F1 in that evaluation, although dataset-split detection was lower at 79.56 percent and two other categories were near 81 to 86 percent.

The paper's estimate that reproducibility rose from 28 percent in 2014 to 64 percent in 2024 is therefore an inference, not a direct replication rate. The researchers combined observed code-and-data-sharing patterns with reproduction rates from an earlier, much smaller study: 33 percent for papers with open data only and 86 percent for papers sharing both open code and open data.

That calculation assumes those historical rates apply across five conferences and eleven years. It also excludes papers using private datasets from the estimate. The result is useful as a directional indicator, not a precise audit of how many modern AI papers would reproduce today.

The directly observed finding is simpler and stronger: far more papers now expose code, data and experimental details that make independent scrutiny possible.

Checklists were not the obvious cause

Four of the five conferences introduced reproducibility checklists or related requirements between 2019 and 2023. Yet the study found no statistical evidence that those introductions accelerated the rate of documentation improvement across the seven variables. In 35 conference-variable comparisons, 15 improved faster after checklist adoption—a pattern consistent with chance in the authors' test.

That does not mean checklists are useless. The trends toward better documentation were already underway, and formal requirements may reinforce a culture that produced them. But the study cannot support a clean causal claim that checklists created the shift.

The more plausible reading is that a broader open-science movement changed expectations across the field, while conferences gradually formalized practices researchers were already adopting.

Openness is a human coordination technology

AI research is often discussed as a race for capability. Reproducibility asks a different question: can other people understand, test and build on the claim?

An open repository lowers the cost of skepticism. It lets students learn from real implementations, lets independent teams probe failures and lets useful methods travel beyond the lab that produced them. It also makes disagreement more productive, because competing interpretations can start from shared artifacts instead of marketing summaries.

There are legitimate limits. Some datasets cannot be released because of privacy, consent, security or contractual constraints. In 2024, 9 percent of the empirical papers in this dataset used private data. Responsible openness does not mean exposing protected information; it means sharing what can be shared and explaining what cannot.

The clearest remaining opportunity is code. In 2024, the study found that 27 percent of empirical papers used open datasets but did not share code. Closing even part of that gap would make more work inspectable without requiring sensitive data to become public.

The study itself follows the practice it measures: its prompts, code, results and analysis are available in a public Zenodo repository. The authors reported support from the U.S. National Science Foundation, the Research Council of Norway and the Alexander von Humboldt Foundation, and declared no competing interests.

The warranted conclusion is hopeful but precise. Leading AI conferences became markedly more open over the decade examined. That does not prove every result is trustworthy. It does mean the community has built more of the conditions that let trust be earned.

Research references: [2]