Who is scientific code for? Maintaining human-readable landmarks in agent-written code

Elle O'Brien

REVIEW 4 major objections 4 minor 11 references

Scientists using coding agents are inventing personal 'landmarking strategies' that mark which parts of a codebase are for humans versus agents, quietly fragmenting collaboration.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:55 UTC pith:HHKK475E

load-bearing objection A genuinely new conceptual label ('landmarking strategies') with vivid case observations, but the empirical generalization outruns the four cases; still worth refereeing as a position piece. the 4 major comments →

arxiv 2607.25975 v1 pith:HHKK475E submitted 2026-07-28 cs.HC

Who is scientific code for? Maintaining human-readable landmarks in agent-written code

Elle O'Brien This is my paper

classification cs.HC

keywords agent-written codelandmarking strategieshuman-readable artifactsscientific programmingversion control conventionscognitive debtcollaborationcontextual inquiry

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the longstanding assumption of scientific code—that at least one person understands why it exists—is breaking down as coding agents generate more code than any one person can review. Drawing on four in-depth contextual-inquiry cases, a survey of over 800 scientific programmers, and the author's own analysis workflow, it describes how scientists are developing idiosyncratic 'landmarking strategies': personal rules for triaging artifacts into those meant for human understanding and those left as context for agents. These strategies often repurpose shared infrastructure like version control in unconventional ways, making commit messages and diffs unreliable as shared documents. The author contends this quiet de-standardization will complicate collaboration across teams with heterogeneous practices, and predicts that teams that explicitly delineate what is human-readable versus agent context will be better able to develop, document, and maintain scientific codebases.

Core claim

On the paper's terms, the discovery is that scientists are not merely accepting agent output wholesale; they are inventing personal conventions for what counts as a human-understandable landmark in the codebase. Concretely: one engineer treats commit messages as lab-notebook entries typed only by the scientist; a postdoc treats only merged diffs as the 'choice' to understand and keeps agent specifications transient; another writes spec documents to markdown and reviews specs rather than code; a non-Git user keeps all code in one growing script, leaving agent-written comments he never reads but that he intends as context for future agents. The paper generalizes from these cases to a claim abo

What carries the argument

The central object is the 'landmarking strategy'—a scientist's private, often unarticulated rule for triaging artifacts in a codebase into those meant for human review and those left as context for agents. It carries the argument because the paper claims these strategies are multiplying idiosyncratically and are not interoperable across teams, which is the mechanism that threatens collaboration.

Load-bearing premise

The central generalization rests on the author's four completed contextual-inquiry cases being representative of scientists broadly, and on the observed behaviors being caused by agentic tools rather than reflecting pre-existing personal habits.

What would settle it

A larger observational study that finds no systematic difference in the idiosyncrasy of version-control and documentation practices between agent-using and non-agent-using scientific teams would directly undercut the claim that agents are driving de-standardization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Commit messages and diffs can no longer be assumed to be reviewable human units; collaborators must ask where the scientist's 'why' is actually recorded.
Newcomers to a codebase—new students, replicators, or reviewers—will need explicit orientation to a team's landmark conventions or risk misreading what is trustworthy.
Heterogeneous tool use (agents versus simple chatbots, large versus small token budgets) creates incompatible context across collaborators, even within one team.
Teams that explicitly negotiate and document what is human-readable versus agent context will be better able to develop, document, and maintain code.
Version-control systems may need to evolve new conventions that assume both human and agent readers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A testable extension: compare commit-message style, branch structure, and comment density in repositories before and after a scientist starts using a coding agent; the fragmentation hypothesis predicts measurable increases in idiosyncratic conventions.
The mechanism resembles how personal knowledge-management practices fragment across a lab; a lightweight shared 'landmark schema' (for example, a README pointer or a standardized comment marker) could serve as a coordination artifact without forcing a single workflow.
The paper implies that code review culture, already weak in science, will face new pressure: agents can generate too much code to review, so landmarking defines what little does get reviewed and might become the new locus of oversight.
An implicit consequence: tools that surface repository context (like issue trackers) for non-Git users could be adapted to encode landmark conventions, making intent legible without requiring everyone to adopt Git.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely new conceptual label ('landmarking strategies') with vivid case observations, but the empirical generalization outruns the four cases; still worth refereeing as a position piece.

read the letter

The landmarking concept is the real contribution here. Naming the practice of triaging code artifacts into 'human-readable' versus 'agent context' gives researchers a handle on something that is happening but was previously diffuse. The four vignettes—commit messages as lab-notebook entries, minimal-diff-as-choice, spec-as-reviewable-artifact, unread agent comments left for future agents—are concrete and recognizable. The paper is also honest about its status as a position piece: the author explicitly says she has four cases so far and that she lacks direct evidence for the collaboration breakdown. That transparency earns credit.

The soft spot is exactly what the stress-test note flags: no baseline. The claim that idiosyncrasies 'may in fact be growing more particular' requires showing that these practices are new, agent-caused, and increasing relative to pre-agent habits. We don't get that. The four cases are contemporaneous, the author's own workflow is one of them, and several behaviors have clear antecedents in how scientists have always managed exploratory code. The >800-person survey is cited only for adoption rates, not for landmarking prevalence or change. So the central generalization is plausible but unproven.

That said, I don't think this is a load-bearing flaw. The paper frames itself as an ongoing contextual inquiry and repeatedly hedges the ambitious claims with 'I expect' and 'I don't have a full theory yet.' For a position piece, it is doing what position pieces should do: naming a phenomenon, illustrating it, and sketching a research agenda. The collaboration prediction is explicitly flagged as a prediction. The main remedy is resubmission-with-revision: add a coding scheme, a sampling/saturation discussion, and a plan for measuring change over time or comparing agent vs. non-agent groups. Also, the author should consider whether 'scientists are inventing' is too strong given that some observed strategies may be repurposed personal habits rather than new inventions.

The writing is clear, the citations are appropriate, and the self-citation to the survey is minor and does not drive the central claim. This paper deserves a serious referee. It is exactly the kind of intervention that can shape future work in human-AI collaboration in scientific programming, as long as the empirical limits are treated as open problems rather than solved questions.

Recommendation: send to peer review. A conditional accept is right, with the condition being a more disciplined framing of the generalization and an explicit future-work plan.

Referee Report

4 major / 4 minor

Summary. This position piece argues that as scientists adopt AI coding agents, the longstanding assumption that at least one human understands why code exists is breaking down. Based on an ongoing contextual inquiry (four completed cases), a survey of >800 scientific programmers, and the author's own analysis workflows, it introduces the concept of "landmarking strategies": personal, often unshared rules for marking which parts of a codebase are for human understanding and which are context for agents. Examples include typed commit messages as lab-notebook entries, minimal-merge diffs as decision records, spec documents committed instead of code review, and chat logs plus a single appended script for non-Git users. The paper argues that this quiet de-standardization may complicate collaboration and predicts that teams that explicitly delineate human-readable versus agent-context artifacts will be better able to maintain scientific code.

Significance. The topic is timely and the observations are vivid and thought-provoking. The paper identifies a plausible new phenomenon—the repurposing of version control and other infrastructure as personal landmarks in agent-mediated work—and connects it to existing concepts of cognitive and intent debt. The concrete prediction (teams that explicitly delineate human-readable vs agent context will fare better) is falsifiable and could be tested. However, the evidence base is thin for the central generalization: four cases, the author's own experience, and a survey used only for adoption statistics. If reframed as a hypothesis-generating position piece, the contribution is real; as a statement of what "scientists are inventing," the current support is insufficient. The paper's explicit acknowledgment of lacking direct evidence for the collaboration claim is to its credit, but it makes the abstract's stronger wording harder to accept.

major comments (4)

[Abstract and §'What I'm seeing'] The central generalization—'scientists are inventing personal conventions' and 'quiet de-standardization'—rests on four completed contextual-inquiry cases. No coding scheme, saturation analysis, or systematic cross-case comparison is reported, and the cases are not described in enough detail. For a position piece this is acceptable as hypothesis generation, but the abstract states it as an observed fact. Please either report the cases more systematically or reframe the claim as 'four observed cases suggest…' and carry that caveat through the abstract.
[§'What I'm seeing' — baseline/causality] The claim that idiosyncrasies are 'growing more particular' and are agent-caused requires a baseline. Several behaviors have recognizable pre-agent antecedents: commit messages as lab notebooks, minimal-diff merges as decision records, spec documents, and long scripts with comments. Without a comparison group of non-agent/chatbot-only programmers or retrospective participant data, the observations could reflect pre-existing personal style. The >800-person survey [7] is cited only for adoption rates (roughly three quarters via ChatGPT), not for landmarking prevalence or change, so it does not rescue the temporal or causal claim.
[§Collaboration] The collaboration prediction is load-bearing for the abstract's 'could complicate collaboration', yet the paper explicitly states 'I do not have direct evidence to corroborate this yet.' The expectation is plausible, but presenting it as an implication ('For these reasons...') conflates a hypothesis with a finding. Either gather direct evidence (e.g., paired collaborator interviews or codebase-handoff studies) or mark this as a speculative design implication in both the abstract and body.
[§'What I'm seeing' — personal narrative] The author's own Git-paralysis example is presented as a supporting data point. It is an anecdote from a single individual who is also the researcher, and the generalization to 'I would bet this is not uncommon' is unsupported. If kept, it should be labeled as motivation/hypothesis, not as a fifth observation, and the basis for the 'not uncommon' bet should be explicit.

minor comments (4)

[Methods/positioning] The paper never states how the four cases were selected, what disciplines or tools were represented, or how field notes/interviews were analyzed. A short 'method' note or a table of case characteristics would help readers judge transferability.
[References] Reference [7] lists the first author as 'O'Brien, G.', which appears inconsistent with the author's name on this paper (Elle O'Brien). If this is the author's own survey, the name should be consistent; if it is a different author, the citation may be a typo.
[Typos] There are typographical errors, e.g., 'suﬀicient' (p. 1) and 'doesn ’tmatter' (p. 2). Also the phrase 'For many scientists ( terminology note:...' uses an unneeded space after the opening parenthesis.
[Terminology] The term 'scientist' is deliberately stretched to include research software engineers, but the paper sometimes slides between 'scientists I've met' and 'all of these scientists.' Consider using 'scientific programmers' consistently or flag the scope each time.

Circularity Check

0 steps flagged

No circularity: the paper is a qualitative position piece whose central concept comes from observed cases, not from a derivation; the only self-citation is background adoption statistics and is not load-bearing.

full rationale

This paper is not a derivation chain but an interpretive position piece. The central concept, 'landmarking strategy,' is introduced as a summary of observed practices from four contextual-inquiry cases and the author's own workflows ('a landmarking strategy is a scientist's own rules for how to triage artifacts for review and record decisions'). It is not inferred from equations, fitted parameters, or a prior model. The only self-citation is reference [7], the authors' 2025 survey, which is cited solely for adoption statistics ('roughly three quarters of adopters worked through general-purpose conversational tools like ChatGPT rather than developer tools') and for background claims about limited code review and small contributor pools. That survey is an external empirical data source, and it does not define, constrain, or prove the landmarking claim. The paper explicitly acknowledges the limitation that the collaboration-fragmentation expectation lacks direct evidence ('While I do not have direct evidence to corroborate this yet'), which is an honest empirical gap rather than a circular reduction. The skeptic's concern that four cases may not support generalization is a validity or evidentiary concern, not circularity. No step in the argument reduces by construction to its inputs, and no prediction is fitted to the data it claims to predict. The minor self-citation is not load-bearing, so the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper rests on qualitative generalizations. No numeric free parameters are fitted. Its background assumptions include the validity of the cited theory-building framing, the representativeness of four cases, the accuracy of the author's cited survey, and Storey's debt framework.

axioms (4)

domain assumption Naur's 'programming as theory building' framing: a scientific codebase should be understandable by at least one person (cited [6]).
The entire problem statement depends on this being the normative baseline that agent workflows violate.
domain assumption Four completed contextual-inquiry cases are representative of the broader population of scientific programmers working with agents.
Used to support statements like 'scientists are inventing their own idiosyncratic conventions' and to predict collaboration fragmentation; no sampling or saturation evidence is provided.
domain assumption Storey's cognitive/intent debt framework is a valid lens for agent-involved software work.
The paper adopts this framing as the explanatory structure for the observed behaviors.
domain assumption The cited survey data ([7]) accurately measure adoption patterns among >800 scientific programmers.
Used to assert that roughly three quarters of adopters use general-purpose conversational chatbots, which grounds the interoperability concern.

pith-pipeline@v1.3.0-alltime-deepseek · 3358 in / 9319 out tokens · 90095 ms · 2026-08-01T00:55:50.499641+00:00 · methodology

0 comments

read the original abstract

Scientific research involving code has long rested on the assumption that at least one person understands why the code exists. As scientists adopt coding agents, this assumption is breaking down. Drawing on an ongoing contextual inquiry of scientific programmers working with agentic tools (four cases to date), a survey of over 800 scientific programmers, and my own analysis workflows, this position piece describes how scientists are inventing personal conventions, "landmarking strategies", for marking which artifacts in a codebase are meant for human understanding and which are context for agents. They repurpose shared infrastructure (version control especially) in idiosyncratic ways, and I argue that this quiet de-standardization could complicate collaboration in teams with heterogeneous software practices. Alternatively, teams that explicitly delineate what is human-readable versus agent context will be better able to develop, document, and maintain scientific code.

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 2 canonical work pages

[1]

Carver, J.C. et al. 2022. A survey of the state of the practice for research software in the United States. PeerJ Computer Science . https://doi.org/ 10.7717/peerj-cs.963

work page doi:10.7717/peerj-cs.963 2022
[2]

and Herbsleb, J.D

Howison, J. and Herbsleb, J.D. 2011. Scientific software production: Incentives and collabo- ration. Proceedings of the ACM conference on computer supported cooperative work, CSCW . https://doi.org/10.1145/1958824.1958904

arXiv 2011
[3]

Jay, C. et al. 2022. ”Not everyone can use Git”: Research Software Engineers’ recommen- dations for scientist-centred software support (and what researchers really think of them) . https://doi.org/10.48420/17313215.v1

work page doi:10.48420/17313215.v1 2022
[4]

and Myers, B.A

Kery, M.B. and Myers, B.A. 2017. Exploring exploratory programming. Proceed- ings of IEEE symposium on visual languages and human-centric computing, VL/HCC . https://doi.org/10.1109/VLHCC.2017.8103446

arXiv 2017
[5]

Milewicz, R. et al. 2019. Characterizing the roles of contributors in open-source scientific software projects. IEEE international working conference on mining software repositories . https://doi.org/10.1109/MSR.2019.00069

arXiv 2019
[6]

Naur, P. 1985. Programming as theory building. Microprocessing and Microprogramming. https://doi.org/10.1016/0165-6074(85)90032-8

doi:10.1016/0165-6074(85)90032-8 1985
[7]

O’Brien, G. et al. 2025. A survey of generative AI adoption and perceived productivity among scientists who program. https://doi.org/ 10.48550/arXiv.2512.19644

Pith/arXiv arXiv doi:10.48550/arxiv.2512.19644 2025
[8]

Sarma, A. et al. 2023. multiverse: Multiplexing Alternative Data Analyses in R Notebooks. Conference on Human Factors in Computing Systems - Proceedings . https://doi.org/10.1145/3544548.3580726

arXiv 2023
[9]

Storey, M.-A. 2026. From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI. https://doi.org/ 10.48550/arXiv.2603.22106

Pith/arXiv arXiv doi:10.48550/arxiv.2603.22106 2026
[10]

Sutherland-Keller, W. 2025. Research Software Systems: Exploration and Infrastructure in Observational Cosmology

2025
[11]

Udell, J. 2026. Vibe coding as a team sport. https://blog.jonudell.net/2026/06/17/vibe- coding-as-a-team-sport/ . 4

2026

[1] [1]

Carver, J.C. et al. 2022. A survey of the state of the practice for research software in the United States. PeerJ Computer Science . https://doi.org/ 10.7717/peerj-cs.963

work page doi:10.7717/peerj-cs.963 2022

[2] [2]

and Herbsleb, J.D

Howison, J. and Herbsleb, J.D. 2011. Scientific software production: Incentives and collabo- ration. Proceedings of the ACM conference on computer supported cooperative work, CSCW . https://doi.org/10.1145/1958824.1958904

arXiv 2011

[3] [3]

Jay, C. et al. 2022. ”Not everyone can use Git”: Research Software Engineers’ recommen- dations for scientist-centred software support (and what researchers really think of them) . https://doi.org/10.48420/17313215.v1

work page doi:10.48420/17313215.v1 2022

[4] [4]

and Myers, B.A

Kery, M.B. and Myers, B.A. 2017. Exploring exploratory programming. Proceed- ings of IEEE symposium on visual languages and human-centric computing, VL/HCC . https://doi.org/10.1109/VLHCC.2017.8103446

arXiv 2017

[5] [5]

Milewicz, R. et al. 2019. Characterizing the roles of contributors in open-source scientific software projects. IEEE international working conference on mining software repositories . https://doi.org/10.1109/MSR.2019.00069

arXiv 2019

[6] [6]

Naur, P. 1985. Programming as theory building. Microprocessing and Microprogramming. https://doi.org/10.1016/0165-6074(85)90032-8

doi:10.1016/0165-6074(85)90032-8 1985

[7] [7]

O’Brien, G. et al. 2025. A survey of generative AI adoption and perceived productivity among scientists who program. https://doi.org/ 10.48550/arXiv.2512.19644

Pith/arXiv arXiv doi:10.48550/arxiv.2512.19644 2025

[8] [8]

Sarma, A. et al. 2023. multiverse: Multiplexing Alternative Data Analyses in R Notebooks. Conference on Human Factors in Computing Systems - Proceedings . https://doi.org/10.1145/3544548.3580726

arXiv 2023

[9] [9]

Storey, M.-A. 2026. From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI. https://doi.org/ 10.48550/arXiv.2603.22106

Pith/arXiv arXiv doi:10.48550/arxiv.2603.22106 2026

[10] [10]

Sutherland-Keller, W. 2025. Research Software Systems: Exploration and Infrastructure in Observational Cosmology

2025

[11] [11]

Udell, J. 2026. Vibe coding as a team sport. https://blog.jonudell.net/2026/06/17/vibe- coding-as-a-team-sport/ . 4

2026