pith. machine review for the scientific record. sign in

arxiv: 2604.24658 · v2 · submitted 2026-04-27 · 💻 cs.LG

Recognition: unknown

The Last Human-Written Paper: Agent-Native Research Artifacts

Jiachen Liu , Jiaxin Pei , Jintao Huang , Chenglei Si , Ao Qu , Xiangru Tang , Runyu Lu , Lichang Chen
show 29 more authors
Xiaoyan Bai Haizhong Zheng Carl Chen Zhiyang Chen Haojie Ye Yujuan Fu Zexue He Zijian Jin Zhenyu Zhang Shangquan Sun Maestro Harmon John Dianzhuo Wang Jianqiao Zeng Jiachen Sun Mingyuan Wu Baoyu Zhou Chenyu You Shijian Lu Yiming Qiu Fan Lai Yuan Yuan Yao Li Junyuan Hong Ruihao Zhu Beidi Chen Alex Pentland Ang Chen Mosharaf Chowdhury Zechen Zhang
Authors on Pith no claims yet

Pith reviewed 2026-05-08 03:53 UTC · model grok-4.3

classification 💻 cs.LG
keywords agent-native research artifactsresearch reproducibilityAI agents for scienceexploration graphsmachine-executable packagesscientific publishingfailure traces
0
0 comments X

The pith

Machine-executable research packages replace narrative papers so AI agents can reproduce and extend work more reliably.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Scientific papers turn a branching research process into a linear story, which drops failed experiments, rejected ideas, and the exact implementation details needed for replication. This creates two hidden costs that become barriers once AI agents must read, run, and build on the work. The paper introduces Agent-Native Research Artifacts that keep the full scientific logic, executable code with every specification, a graph of every exploration path including dead ends, and raw evidence for each claim. On two benchmarks the new format lifts question-answering accuracy from 72.4 percent to 93.7 percent and reproduction success from 57.4 percent to 64.4 percent. Preserved failure traces speed progress on extension tasks yet can also box in agents that might otherwise explore beyond prior runs.

Core claim

Traditional papers impose a Storytelling Tax by discarding failed experiments and branches and an Engineering Tax by omitting implementation details that agents need. An Agent-Native Research Artifact replaces the paper with a four-layer executable package: scientific logic, full code specifications, an exploration graph that retains every attempt and dead end, and raw-output evidence grounding every claim. Three supporting tools capture live decisions during development, compile legacy papers into the new format, and automate objective review checks.

What carries the argument

The Agent-Native Research Artifact, a structured package whose four layers (scientific logic, executable code with full specifications, exploration graph of all paths including failures, and raw evidence) replace the linear narrative paper.

If this is right

  • AI agents will reproduce published results at higher rates because they receive full specifications and preserved failure traces instead of having to guess missing steps.
  • Human reviewers can shift focus to significance and novelty once automated checks handle objective verification of claims and code.
  • Research teams can automatically record every decision and dead end through a live manager, eliminating the need to reconstruct history later.
  • Legacy papers and code repositories can be converted into the new format by a compiler, allowing older work to become usable by agents.
  • Open-ended extension tasks gain from access to prior failure traces, though the same traces may limit some agents from venturing outside the recorded paths.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Publishing could gradually move from human-readable stories toward dual-use artifacts that serve both people and machines without forcing one format to serve both.
  • Agents might learn to treat the exploration graph itself as a reusable map rather than restarting from a clean slate on every new question.
  • Over time the constraint effect of preserved traces could be tested by comparing agents that receive only the graph versus agents that receive only the final successful path.

Load-bearing premise

The benchmarks and the specific way the new artifacts were applied to them capture the real difficulties AI agents face when trying to understand, reproduce, and extend published research.

What would settle it

Measure whether agents using the new artifacts still show the reported accuracy and reproduction gains when tested on a fresh set of papers whose development never used the artifact format.

Figures

Figures reproduced from arXiv: 2604.24658 by Alex Pentland, Ang Chen, Ao Qu, Baoyu Zhou, Beidi Chen, Carl Chen, Chenglei Si, Chenyu You, Fan Lai, Haizhong Zheng, Haojie Ye, Jiachen Liu, Jiachen Sun, Jianqiao Zeng, Jiaxin Pei, Jintao Huang, John Dianzhuo Wang, Junyuan Hong, Lichang Chen, Maestro Harmon, Mingyuan Wu, Mosharaf Chowdhury, Ruihao Zhu, Runyu Lu, Shangquan Sun, Shijian Lu, Xiangru Tang, Xiaoyan Bai, Yao Li, Yiming Qiu, Yuan Yuan, Yujuan Fu, Zechen Zhang, Zexue He, Zhenyu Zhang, Zhiyang Chen, Zijian Jin.

Figure 1
Figure 1. Figure 1: Publishing compiles a rich research object into a lossy narrative (left); ARA preserves the original as a high-fidelity, agent￾executable knowledge package (right). this process into a polished linear story, discarding every failed experiment, rejected hypothesis, and abandoned ap￾proach. This emphasis on success leaves failures undocu￾mented; although modern platforms archive final artifacts, the branchin… view at source ↗
Figure 2
Figure 2. Figure 2: The Storytelling Tax: research proceeds as a branching tree with dead ends (left), but publication compiles it into a linear narrative (right), discarding all failure knowledge. repositories but does not address the epistemic structure of research itself. None of these efforts jointly structure scien￾tific logic, executable code, and exploration history into a single operable object (§8). We propose the Ag… view at source ↗
Figure 3
Figure 3. Figure 3: The reproduction information gap across 8,921 Paper￾Bench requirements. (a) PDFs systematically under-specify code development tasks. (b) The three largest gap types are precisely the categories ARA’s structured layers address. • We identify two structural costs of compiling research into narrative—the Storytelling Tax and the Engineering Tax—and introduce the Agent-Native Research Artifact (ARA): a protoc… view at source ↗
Figure 5
Figure 5. Figure 5 view at source ↗
Figure 6
Figure 6. Figure 6: The Live Research Manager operates at session bound￾aries: a three-stage pipeline (Context Harvester → Event Router → Maturity Tracker) distills each researcher–agent conversation into typed events that accumulate across ARA layers over time. We consider an ARA sufficient when a sufficiently capable coding agent can reproduce the core claim zero-shot from it, without human intervention or external context … view at source ↗
Figure 7
Figure 7. Figure 7 view at source ↗
Figure 8
Figure 8. Figure 8 view at source ↗
Figure 9
Figure 9. Figure 9: Three-stage ARA-native review pipeline. Stages 1–2 invoke the ARA Seal levels of view at source ↗
Figure 10
Figure 10. Figure 10: The (Human+AI)2 research network. Each researcher works through a research agent that interfaces with a shared ARA network via /submit, /retrieve, and /fork; agents may also collaborate directly. contribution significant—does it address a real problem that matters? Is the core insight genuinely novel, or an incremen￾tal recombination of known ideas? Is this the right formu￾lation of the problem, and are t… view at source ↗
Figure 11
Figure 11. Figure 11: Aggregate reproduction success rates across all 15 papers, stratified by difficulty. The ARA advantage widens mono￾tonically with difficulty (+4.9% easy, +5.6% medium, +8.5% hard), tracking the tiers where reproduction depends most heavily on configuration content the PDF underspecifies. and completed all medium and hard subtasks; the baseline agent fought the JAX environment and completed only 3 training… view at source ↗
Figure 12
Figure 12. Figure 12: Extension trajectories on five RE-Bench tasks under Claude Sonnet 4.6. One task per column: top row is score-vs-wall-clock￾time, bottom row is score-vs-cumulative-cost; the y-axis is shared down each column. Faint markers are individual scoring attempts, solid lines are the best-so-far envelope, and stars mark the best attempt; dotted grey lines mark each task’s RE-Bench reference. Arrows in the column ti… view at source ↗
Figure 13
Figure 13. Figure 13: Per-paper ARA − baseline delta (percentage points) on each difficulty stratum, sorted by mean advantage. Green in￾dicates ARA wins, red indicates baseline wins. Gains concen￾trate in the medium and hard columns across most papers; the few baseline wins are confined to a small set, most prominently self-expansion and ftrl. environment and completed only 3 training attempts before exhausting its budget. The… view at source ↗
Figure 14
Figure 14. Figure 14: triton_cumsum on Sonnet 4.5: paper vs. ARA score-vs-time (left) and score-vs-cost (right). Faint markers are raw scoring attempts, solid line is the best-so-far envelope, stars mark best-attempt positions. Dotted line is the original RE-Bench reference (0.47) reported on different H100 silicon; the harness-measured per-hardware baseline (∼0.64) is where both arms start. The 4.6 trajectories are in the bod… view at source ↗
Figure 15
Figure 15. Figure 15: restricted_mlm on Sonnet 4.5: paper vs. ARA score-vs-time (left) and score-vs-cost (right). Faint markers are raw scoring attempts, solid line is the best-so-far envelope, stars mark best-attempt positions. Dotted lines mark the untrained-MLP baseline (1.84) and the RE-Bench reference (1.13). Both arms are anchored at the 1.84 baseline at t = 0; ARA-4.5’s pre-agent harness baseline crashed (corrupted star… view at source ↗
read the original abstract

Scientific publication compresses a branching, iterative research process into a linear narrative, discarding the majority of what was discovered along the way. This compilation imposes two structural costs: a Storytelling Tax, where failed experiments, rejected hypotheses, and the branching exploration process are discarded to fit a linear narrative; and an Engineering Tax, where the gap between reviewer-sufficient prose and agent-sufficient specification leaves critical implementation details unwritten. Tolerable for human readers, these costs become critical when AI agents must understand, reproduce, and extend published work. We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package structured around four layers: scientific logic, executable code with full specifications, an exploration graph that preserves the failures compilation discards, and evidence grounding every claim in raw outputs. Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste. On PaperBench and RE-Bench, ARA raises question-answering accuracy from 72.4% to 93.7% and reproduction success from 57.4% to 64.4%. On RE-Bench's five open-ended extension tasks, preserved failure traces in ARA accelerate progress, but can also constrain a capable agent from stepping outside the prior-run box depending on the agent's capabilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes the Agent-Native Research Artifact (ARA), a machine-executable research package to supplant traditional narrative papers. ARA comprises four layers: scientific logic, executable code with full specifications, an exploration graph that preserves failed experiments and dead ends, and evidence grounding every claim in raw outputs. It is supported by a Live Research Manager for capturing decisions during development, an ARA Compiler for translating legacy PDFs and repositories, and an ARA-native review system. On PaperBench and RE-Bench, ARA improves question-answering accuracy from 72.4% to 93.7% and reproduction success from 57.4% to 64.4%; on five open-ended extension tasks, preserved failure traces accelerate progress for some agents but can constrain others depending on agent capabilities.

Significance. If the benchmark gains prove robust, the work could meaningfully lower barriers for AI agents to understand, reproduce, and extend published research by mitigating the Storytelling Tax and Engineering Tax. The evaluation on external benchmarks (PaperBench, RE-Bench) without self-referential fitting or invented parameters, together with the explicit preservation of exploration graphs, constitutes a concrete strength that supports falsifiable claims about agent performance.

major comments (2)
  1. [Evaluation section (benchmark results)] Evaluation section (benchmark results): The headline gains (QA accuracy 72.4%→93.7%, reproduction 57.4%→64.4%) are reported without details on experimental controls, such as the procedure for converting legacy papers via the ARA Compiler, whether the Compiler was tuned on the same corpus as the test set, or the precise agent baselines and prompting setups. This leaves the attribution of gains to the ARA structure versus other implementation choices only partially supported and weakens the central claim that ARA addresses general challenges for agents on arbitrary published work.
  2. [Open-ended extension tasks subsection] Open-ended extension tasks subsection: The finding that preserved failure traces both accelerate and constrain agents is presented as agent-dependent, yet no additional controls (e.g., ablation removing the exploration graph or tests with a broader range of agent architectures) are reported. This makes the mixed outcome on the five RE-Bench tasks difficult to generalize beyond the specific agents and tasks tested.
minor comments (2)
  1. [Abstract] The abstract lists the four layers and three mechanisms but does not name the mechanisms (Live Research Manager, ARA Compiler, ARA-native review) until the body; adding the names to the abstract would improve immediate clarity.
  2. [Architecture description] A summary table or diagram illustrating the interactions among the four ARA layers and the three supporting mechanisms would aid readers in grasping the overall architecture.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We address each major comment below, clarifying the evaluation methodology and acknowledging limitations where they exist. Revisions have been made to improve transparency and support for our claims.

read point-by-point responses

  1. Referee: Evaluation section (benchmark results): The headline gains (QA accuracy 72.4%→93.7%, reproduction 57.4%→64.4%) are reported without details on experimental controls, such as the procedure for converting legacy papers via the ARA Compiler, whether the Compiler was tuned on the same corpus as the test set, or the precise agent baselines and prompting setups. This leaves the attribution of gains to the ARA structure versus other implementation choices only partially supported and weakens the central claim that ARA addresses general challenges for agents on arbitrary published work.

    Authors: We agree that the original manuscript provided insufficient detail on experimental controls, which weakens the support for attributing gains specifically to the ARA structure. In the revised version, we have expanded the Evaluation section with a dedicated 'Experimental Controls' subsection. This includes: a precise description of the ARA Compiler's conversion pipeline for legacy PDFs and repositories (step-by-step preprocessing, handling of narrative text, and code extraction); explicit confirmation that the Compiler was developed and validated exclusively on a held-out corpus disjoint from PaperBench and RE-Bench to avoid any tuning or leakage on the test sets; and full specifications of all agent baselines, including model versions, exact prompting templates, temperature values, and decoding parameters. With these controls held constant, the reported gains can be more confidently attributed to the four-layer ARA format rather than ancillary implementation choices. We believe this directly addresses the concern and bolsters the central claim. revision: yes

  2. Referee: Open-ended extension tasks subsection: The finding that preserved failure traces both accelerate and constrain agents is presented as agent-dependent, yet no additional controls (e.g., ablation removing the exploration graph or tests with a broader range of agent architectures) are reported. This makes the mixed outcome on the five RE-Bench tasks difficult to generalize beyond the specific agents and tasks tested.

    Authors: We acknowledge that the results on the open-ended extension tasks have limited generalizability due to the absence of explicit ablations and a narrow set of agent architectures. The manuscript already frames the mixed outcomes as agent-dependent based on observed capabilities. In the revision, we have added a dedicated limitations paragraph in the Open-ended extension tasks subsection that explicitly discusses this dependency and includes results from one additional agent architecture to provide a modestly broader perspective. However, a full ablation removing the exploration graph was not performed, as re-executing the five tasks without failure traces would require substantial additional compute that was not available within the revision timeline. We have noted this limitation and flagged a comprehensive ablation study as important future work. The current findings are presented as preliminary evidence of nuanced effects rather than a general claim. revision: partial

standing simulated objections not resolved
  • A complete ablation study removing the exploration graph across all tasks and evaluation with a significantly broader range of agent architectures would require additional experiments beyond the scope and resources of the current revision.

Circularity Check

0 steps flagged

No circularity: proposal evaluated on external benchmarks

full rationale

The paper introduces the ARA protocol as a design proposal with four layers and supporting mechanisms (Live Research Manager, ARA Compiler, ARA-native review), then reports empirical gains on PaperBench and RE-Bench. No equations, derivations, fitted parameters, or self-referential predictions appear in the provided text. Claims rest on benchmark measurements rather than reducing to inputs by construction or load-bearing self-citations. This matches the default case of a self-contained design paper with external validation; no load-bearing step reduces to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the premise that narrative papers lose critical information for AI agents. No numerical free parameters are introduced. The main axiom is a domain assumption about AI capabilities with structured versus narrative inputs. The ARA itself is the primary invented entity.

axioms (1)
  • domain assumption Narrative scientific papers impose a Storytelling Tax and Engineering Tax that become critical barriers when AI agents must understand, reproduce, and extend the work.
    This premise is stated directly in the abstract as the motivation for ARA.
invented entities (1)
  • Agent-Native Research Artifact (ARA) no independent evidence
    purpose: A machine-executable research package with four layers to replace traditional narrative papers.
    New protocol proposed in this work; no independent evidence outside the paper's benchmarks is provided.

pith-pipeline@v0.9.0 · 5714 in / 1405 out tokens · 85428 ms · 2026-05-08T03:53:42.590982+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Toward an Engineering of Science: Rebalancing Generation and Verification in the Age of AI

    cs.CY 2026-05 unverdicted novelty 5.0

    AI lowers the cost of generating plausible scientific artifacts without lowering verification costs, so the paper proposes blueprints as typed graph components that decompose claims, evidence, and assumptions to enabl...

Reference graph

Works this paper leans on

15 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Whole Tale

    URL https://www.wandb.com/. Software available from wandb.com. Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Autonomous chemical research with large language mod- els.Nature, 624(7992):570–578, 2023. doi: 10.1038/ s41586-023-06792-0. Booeshaghi, A. S., Luebbert, L., and Pachter, L. Science should be machine-readable.bioRxiv, 2026. doi: 10. 64898/2...

    work page doi:10.1016/j.future.2017.12.029 2023

  2. [2]

    Hua, T., Hua, H., Xiang, V ., Klieger, B., Truong, S

    doi: 10.3233/ISU-2010-0613. Hua, T., Hua, H., Xiang, V ., Klieger, B., Truong, S. T., Liang, W., Sun, F.-Y ., and Haber, N. ResearchCodeBench: Benchmarking LLMs on implementing novel machine learning research code.arXiv preprint arXiv:2506.02314, 2025. Huang, M. DecMetrics: Structured claim decomposition scoring for factually consistent LLM outputs.arXiv ...

    work page doi:10.3233/isu-2010-0613 2010

  3. [3]

    S 2 ORC : The Semantic Scholar Open Research Corpus

    doi: 10.18653/v1/2020.acl-main.447. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The AI scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024. Luo, Y ., Yu, Z., Wang, X., Zhu, Y ., Zhang, N., Wei, L., Du, L., Zheng, D., and Chen, H. What makes AI research replicable? Executable knowle...

    work page doi:10.18653/v1/2020.acl-main.447 2020

  4. [4]

    Medawar, P

    doi: 10.1242/dmm.015123. Medawar, P. B. Is the scientific paper a fraud?The Listener, 70:377–378, 1963. Reprinted inThe Strange Case of the Spotted Mice, Oxford University Press, 1996. OpenAI. AGENTS.md: A standard for agent-oriented repository documentation. https://github.com/ openai/agents.md, 2025. Accessed 2026-03-01. Pineau, J., Vincent-Lamarre, P.,...

    work page doi:10.1242/dmm.015123 1963

  5. [5]

    We evaluate all methods on three task sequences with 10 seeds each

    Combinatorial experiment matrix (24.1%).The sin- gle largest category consists of requirements that enumerate which model variant must be run on which dataset, with which configuration, for how many seeds. In PDFs, this combinatorial structure is compressed into a single sentence (“We evaluate all methods on three task sequences with 10 seeds each”) or a ...

  6. [6]

    Compute cosine similarity between δi and δmlp,i for layers 0, 2, 4, 6, 8, 10, 12, 14, 16, 18 using 1,199 prompts from RealToxicityPrompts

    Evaluation protocol (18.5%).Requirements specify- ingwhichmetric to compute, onwhichtest split, usingwhich evaluation-time configuration (e.g., beam size, number of evaluation episodes, specific layers to probe). These details are often scattered: the metric definition appears in §3, the test split in §4, the evaluation episodes in the appendix, and the l...

  7. [7]

    AdamW optimizer with learning rate 5e-6, weight decay 0.01; batch size 64; 6,000 training steps

    Hyperparameters (17.2%).Classic training config- uration: learning rates, batch sizes, optimizer parameters, temperature, weight decay, LoRA rank, number of epochs. While these are themost commonly discussedreproduction barrier, they account for only 17% of leaf requirements. In PDFs, hyperparameters are typically consolidated in an ap- pendix table, but ...

  8. [8]

    instrumentation

    Metric computation and logging (10.4%).Require- ments that the agent mustrecordspecific intermediate quan- tities during runs: loss curves, attention distributions, cost tracking (dollars per 1k questions), episodic returns logged every N steps. This “instrumentation” knowledge is rarely specified in papers—authors implicitly know what to log but do not d...

  9. [9]

    After adapting with DPO, the principal component of the residual streams shift in the same direction, and the activation of the toxic vectors decrease

    Result interpretation (8.6%).Qualitative claims about what the results shouldshow—directional trends, compar- ative rankings, mechanistic explanations. These carry the highest weight in PaperBench rubrics (weight = 2) because they test whether the agentunderstandsthe results, not just whether the code ran. Examples: • mechanistic-understanding: “After ada...

  10. [10]

    CNN has three convolutional layers with 32, 64, and 64 channels and filter sizes of 8, 4, and 3

    Architecture specification (5.8%).Layer counts, chan- nel sizes, activation functions, output head structure, em- bedding dimensions. In PDFs, architecture details are split across a figure (showing the high-level diagram), the meth- ods section (describing components in prose), and the ap- pendix (listing dimensions in a table). An agent must men- tally ...

  11. [11]

    Output attention: softmax(qK T /√dmodel)·V

    Mathematical formulation (4.5%).Specific equations that must be implemented exactly: loss functions, atten- 24 The Last Human-Written Paper: Agent-Native Research Artifacts tion operations, PDE boundary conditions, update rules. In PDFs, equations are referenced by number, but the reader must trace through variable definitions scattered across mul- tiple ...

  12. [12]

    Single CNN encoder per policy; new encoder initialized with weights of the previous one

    Implementation tricks (4.2%).Non-obvious design choices that distinguish faithful reproduction from naive re-implementation: weight freezing schedules, initialization from prior checkpoints, gradient clipping thresholds, nor- malization details, optimizer switching strategies. These are the hardest items to recover from a PDF because they ap- pear as pare...

  13. [13]

    we use the standard train/test split

    Data pipeline (3.8%).Dataset acquisition, split ratios, filtering criteria, preprocessing steps, data augmentation, collocation point sampling strategies. These details are often under-specified in papers (“we use the standard train/test split”) or tucked into a single appendix paragraph. Examples: • bbox: “Split GSM8K into 7,473 training and 1,319 test s...

  14. [14]

    obvious” and omitted entirely from the paper, yet they are essential for reproduction. Examples: • bbox: “API access configured for davinci-002

    Environment and infrastructure (2.9%).Specific API endpoints, model version strings, library versions, sim- ulator names, hardware requirements. These are often as- sumed to be “obvious” and omitted entirely from the paper, yet they are essential for reproduction. Examples: • bbox: “API access configured for davinci-002”; “Code to execute fine-tuning jobs...

    2026

  15. [15]

    score": X,

    The sign pattern (8–2) is itself statistically improbable under the null hypothesis of no difference (p= 0.039 , exact binomial), confirming that the aggregate advantage is not driven by a single outlier paper. F.2. Per-Paper Reproduction Analysis This section provides the detailed per-paper analysis for the reproduction experiment (§7.3). Per-difficulty ...

    2025