Pith.

Artificial Intelligence · arXiv:2610.41350 v1 · submitted 30 Oct 2026 · 96 references

Operon-S1: a fast decision controller for long-horizon biomedical agents

Genoria AI Team: Yuejie Hou, Yueyuxiao Yang, Weiting Tang, Meng Yang

The paper says

“Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own.”

more of the abstract

Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Every decision is recorded with its consequence. Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. Task-level results in this study use the public checkpoint. The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 2 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A well-built, honest decision-layer paper whose load-bearing claim—that retraining the judge improves the agent—is not yet demonstrated at task level. This line is wrong

4 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 96 references say.

The paper says, at Results, 'These gains carry over to the agent's own states'; Fig. 4c; Table 3

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The headline analysis-family transfer (0.056→1.000, +0.944 [+0.908, +0.974]) is reported with the parenthetical 'one label value'. If the held-out seqqa2 states carry a single analysis-family label, the majority-class accuracy on those exact states is already 1.000, and rsi-v6's perfect score is consistent with learning the label prior rather than with meaningful transfer to the agent's own states.

more of objection 1

The paper reports majority-class and rule-based accuracies on pooled benchmark states (Extended Data Table 6) but not on the exact held-out state set used for this claim, so the reader cannot tell whether rsi-v6 exceeded the trivial baseline. Please report majority-class, rule-based, and balanced accuracy on the exact held-out analysis-family states, or replace this example with a multi-class judgement.

The paper says, at Results, 'These gains carry over to the agent's own states'; Extended Data Table 6; Eq. (6)

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The second transfer number, execution-labelled error-class accuracy 0.174→0.408 (item-level paired +0.186 [+0.045, +0.329]), is the only unconfounded held-out item-level evidence, and it is weak in absolute terms: 0.408 is far below the rule-based controller's 0.953 on benchmark states and only about 0.025 above the pooled majority rate of 0.383.

more of objection 2

Under the deployed action rule (Eq. 6, tau_act=0.6, delta=0.2), raw accuracy of 0.408 implies that most acted-on error-triage decisions would name the wrong class; the paper does not report the acted-on share or the acted-on accuracy for rsi-v6 under that policy. Additionally, rsi-v6's gate-split accuracy for error class is not reported anywhere, while for rsi-v2 the gate-split error-class accuracy was 0.91 (Fig. 4c); the implied curriculum-to-deployment drop is large and is neither quantified nor discussed.

The paper says, at Abstract; Results, 'Gated retraining'; Extended Data Table 2

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

No trained version is ever run as the controller in the main harness: every LAYA cell in Extended Data Table 2 is version 0, and the paper states that task-level results use the public checkpoint. The central claim that the judgement layer 'can be trained' therefore rests on decision-accuracy and replayed-state evidence only; whether a promoted version improves, leaves unchanged, or degrades task outcomes under the actual agent is untested.

more of objection 3

Given the paper's own demonstration that decision accuracy does not track task outcomes (version 0 has error-class accuracy 0.164 yet reduces steps on protocolqa2, while its seqqa2 score falls by 0.013), this is a substantive empirical gap rather than a formality. I recommend either running a promoted version in the harness on at least a subset of benchmarks (e.g., the held-out halves of seqqa2 and protocolqa2) or explicitly restating the trainability claim so that it excludes task-level consequences.

The paper says, at Abstract vs. Results, 'The local decision model costs little...'; Extended Data Table 2

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The paper's headline efficiency numbers (SeqQA cost −5.8%; protocolqa2 steps −10.3%) are single-benchmark effects, one on a benchmark added after the hypotheses were written, and the pre-registered criterion that an efficiency gain hold on at least four of five hypothesis benchmarks was not met by any arm ('at the data cutoff it holds for no arm').

more of objection 4

In addition, the LAYA version-0 score change on seqqa2 (−0.013 [−0.028, −0.001]) is a measurable degradation. The Results section is transparent about all of this, but the abstract presents the two favourable benchmark results without the H1 status and without the seqqa2 drop; I recommend reporting the H1 outcome and the seqqa2 degradation wherever the efficiency claims are summarized.

4 minor objections

The paper says, at Abstract and Results, controller latency

Pith says: minor objection 1 of 4

The abstract's '0.6 s at the median for an in-loop request' matches the error-triage point (P3, 576 ms), but in-loop medians range from 531 ms (P8) to 839 ms (P7); please report the range in the abstract or qualify the single number.

The paper says, at Results, injected-message audit

Pith says: minor objection 2 of 4

The audit statement 'no injected message contained the gold answer' applies only to the 2,361 runs whose answers can be matched as text; the remaining 3,382 runs cannot be checked. The main text discloses this immediately after the audit sentence, but the claim is repeated in the opening results section without the coverage caveat, and the coverage should travel with the claim.

show 2 more

The paper says, at Results, probe design

Pith says: minor objection 3 of 4

The sentence 'One rule over visible features, fitted to the labels, reproduces all 3,859 keep-or-redesign labels in-sample' describes an in-sample fit to the labels it reproduces; as the paper itself frames this as a check of the layer against a known answer, I suggest labelling it explicitly as a sanity check so that the in-sample reproduction is not read as validation.

The paper says, at Extended Data Table 2 and Methods, benchmarks

Pith says: minor objection 4 of 4

Table 2 lists substitute judge models for several benchmarks (BiomniBench-DA, BixBench range answers, protocolqa2), and the ratio of wall-clock changes on BiomniBench-DA is reported with a concurrency caveat. Within-arm comparisons are fair because the judge is identical across arms, but the manuscript should state once, in the main text, that scores are not directly comparable to published benchmark leaderboards because of the substitute judges and doubled time limits.

What would settle it

Run a promoted version, e.g. rsi-v6, as the LAYA controller on the held-out halves of seqqa2 and protocolqa2 under the same harness, with version 0 and no-op arms, and report paired score, cost and steps. In parallel, on the exact held-out states used for Table 3, report majority-class and rule-based accuracy for analysis family and error class alongside rsi-v6. If rsi-v6 does not beat v0/no-op on task outcomes, or if its held-out transfer accuracy is within the reported interval of the majority class or below the rule-based controller, the claim that the trainable layer improves the agent at the task level is not supported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 2 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 1 min 14 s · 42,769 tokens read, 10,809 written
  3. Editor wrote the verdict above 20 s · 43,877 tokens read, 2,484 written
  4. Referee major revision: 4 major objections, 4 minor 1 min 54 s · 43,249 tokens read, 17,571 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here is my read of Operon-S1. The genuinely new thing is not any single mechanism but the full package: 13 fixed decision points, closed questions, fixed messages that never contain answers, per-decision provenance, a leakage firewall, and a promotion gate with majority-class and random-subset controls, with labels drawn only from construction, verifiers, or execution. That is a useful design for auditable, self-improving agents, and it is implemented carefully. The no-op control, the message audit, the non-inferiority framework, and the honest labeling of protocolqa2 as exploratory all show real discipline.

What holds up: a separate fast judge can be cheap and can leave scores mostly unchanged. With the public zero-shot checkpoint, the cost savings on SeqQA and the step savings on protocolqa2 are real, and no measurable score change appears on five of six benchmarks. Retraining does raise decision accuracy on gate splits across all trained versions. That part is solid.

The soft spot is exactly where the reader and stress-test put it: the abstract’s “gains carried over” is doing more work than two numbers can support. On the seqqa2 analysis-family held-out states, the paper itself reports “one label value.” If those states are majority-class one, the 0.056 to 1.000 jump is mostly learning the prior. The error-class transfer is item-level and better, but 0.408 is still far below the rule-based controller’s 0.953 on benchmark states and only slightly above majority. And no trained version is ever run as the controller in the main harness—task-level results all use version 0. So the trainable-controller claim is supported only on decision accuracy, not on task outcomes.

The authors disclose most of this gap somewhere in the text, which I credit. The failure of their pre-registered H1 for every arm is reported plainly, not buried. That is a point in their favor.

What I would want before trusting the stronger claim: one promoted version run end-to-end at task level, with the held-out states reporting majority-class accuracy alongside the model’s, and shipped code and data. Right now the artifacts are “available from Genoria AI,” which is not the same as available.

Who gets value: anyone building agent harnesses, decision layers, or release gates for self-improving systems. It deserves a serious referee; the gap between decision accuracy and task-level benefit is the thing to push on, and it is fixable in a revision. I would take it to our reading group.

The strongest objection

The central value claim is that the decision layer can be measured, replaced and trained. The main evaluation uses LAYA version 0 zero-shot, and the authors state that task-level results use the public checkpoint and that the quantity training targets is decision accuracy rather than task score. The load-bearing bridge is therefore the claim that training gains carry over to the agent's own decision states. That bridge rests on two heldout numbers in Table 3 and Fig. 4c: seqqa2 analysis-family accuracy 0.056->1.000, and execution-labelled error-class accuracy 0.174->0.408. The analysis-family result is accompanied by the text 'one label value'. If the heldout seqqa2 states carry a single analysis-family label, then the majority class on those same states already scores 1.000, so rsi-v6's perfect score demonstrates learning the label prior rather than meaningful transfer; the majority-class and rule-based accuracies on those exact heldout states are not reported. The error-class result is better because it is item-level paired, but 0.408 remains far below the rule-based controller's 0.953 on benchmark states and only about 0.025 above the majority-class rate of 0.383 (Extended Data Table 6). These numbers are consistent with real but functionally small transfer. No promoted version is ever run as the controller in the main harness, so the step from 'better decision accuracy on replayed states' to 'better or cheaper task outcomes under the same agent' is untested. The paper discloses part of this gap explicitly, but the abstract's 'gains carried over' formulation gives the transfer evidence more weight than the single-label and below-rule numbers can carry.

The reader's weakest assumption is that decision accuracy on synthetic curriculum states transfers to task-level benefit. My read agrees, and the same concern is load-bearing because the abstract and results highlight retraining gains as evidence for the trainable-layer claim. I sharpen the reader's point in two ways. First, the flagship transfer number, analysis-family 0.056->1.000 on seqqa2, is accompanied by 'one label value'; if that means all heldout states share one label, the majority class already achieves 1.000 on those states, so the result is not evidence of transfer beyond the label prior. Second, the other transfer number, error-class 0.174->0.408, is well below the rule-based controller's benchmark-state accuracy of 0.953 and only marginally above the majority rate of 0.383, so the demonstrated transfer is functionally small even if statistically real. The paper is transparent that task-level results use version 0 and that the gate certifies decision accuracy, and I do not treat that as a hidden flaw. But it means the central controller claim still depends on an untested link: a promoted version running in the loop and improving task outcomes. The proposed concrete test would settle that link. This does not change the reader's verdict: conditional acceptance remains appropriate, pending a task-level run of a promoted version and the missing majority/rule comparison on the same heldout states.

The referee's report

The conceptual contribution — separating an agent's recurring scientific judgements from its computation and treating them as a measured, replaceable, trainable layer whose labels no model produces — is valuable and timely, and the engineering is unusually careful. Strengths that deserve explicit credit include: a no-op arm with a regression test (Eq. 8) enforcing equivalence to the original agent; a leakage firewall (Eq. 11); a promotion gate (Eq. 14) with majority-class and random-subset controls and an explicit accuracy-paradox discussion; cross-fitting for any benchmark states entering training; an append-only registry with SHA-256 identities; a dated, itemized log of deviations from the evaluation plan; and transparent reporting of the H1 failure and of the seqqa2 score losses. If the transfer claims are adequately supported, the paper would provide a reference architecture for judgement-layer design in scientific agents. At present the task-level value of the retrained versions is unproven, and the two transfer numbers carry most of that load; one of them is potentially confounded by a single-label prior.

Pith's reading of the paper (a machine's summary, not the authors')

The paper claims that the limit on long-horizon biomedical agents has moved from computation to judgement: most of the original agent's wrong answers and surplus time came from choices about conventions, skills, error classes, evidence and stopping that were made inside the reasoning model and could not be measured or fixed on their own. Operon-S1 separates judging from doing by posing those recurring judgements as 13 closed questions during a run and handing them to a fast local decision model that acts only through pre-written messages that never contain an answer. With the public checkpoint of the local model as judge, this layer cut cost on SeqQA by 5.8% and agent steps on protocolqa2 by 10.3%, with no measurable score change on five of six benchmarks, at a median of about 0.6 seconds per in-loop request and no request cost. Retraining on labels drawn only from task construction, program verifiers, or code execution raised decision accuracy on every question in each of three trained versions, and the gains carried over to the agent's own decision states on held-out items. A promotion gate admits a new version only when it beats its parent, a rule-based controller, the majority class, and a random-subset control, so the layer's evidence of improvement never comes from a language model.

What carries the argument. The load-bearing object is the decision harness: 13 decision points, each a tuple of a trigger, a deterministic state renderer, and closed questions over finite option sets, mapped through one fixed policy to four actions (brief, remind, end the tool loop, request one recheck) delivered in pre-written messages that never contain an answer. States are rendered as pure functions of the run's facts — task prompt, workspace listing, environments, skill shortlist, budget fraction — so the agent never assembles its own context and any decision can be replayed; all controllers share the policy, thresholds, and messages and differ only in the probability distributions they return, making a comparison between controllers a comparison between judgements. Two properties make the layer trainable. Labels come only from construction, a verifier, or code execution, never from a model, and a promotion gate admits a candidate only when it beats its parent, the rule-based controller, the majority class, and a random-subset control of equal training budget on every question with headroom, with expected calibration error at most 0.10 and a lower cluster-bootstrap bound above zero. The gate is part of the scientific claim: an agent that improved on labels its own training loop generated could be fooling itself, and the paper cites the result that no automated gate can in general refuse every bad update and accept every good one, so the gate's guarantees are confined to a bounded, labelled split.

Core claim. The central claim is that an agent's scientific judgements form a layer of their own, distinct from the computation they steer: each judgement can be posed as a closed question, scored against a label that no model produces, and improved by retraining under a statistical gate. The paper builds this from a 13-point decision harness whose states are rendered deterministically from the facts of the run, so every decision can be replayed, and whose actions are four fixed message types — brief, remind, end the tool loop, request one recheck — that may name a convention, a file, or a quantity but never an answer value. Six controllers ran in the same harness on seven life-science benchmarks; the no-op control established that the harness itself leaves the agent unchanged, and an audit of 16,042 injected messages found none containing a gold answer in any run whose answer can be matched as text. As judge, the public LAYA checkpoint (a 322-million-parameter local encoder) accounted for 0.8% to 5.2% of wall-clock time at no request cost, lowered cost on SeqQA by 5.8% and steps on protocolqa2 by 10.3%, and changed no score outside its interval on five of six benchmarks, while the general language model driving the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Retraining on verifiable labels raised accuracy on every decision question in three trained versions, with transfer measured on the agent's own held-out states: analysis-family accuracy on seqqa2 rose from 0.056 to 1.000 and execution-labelled error-class accuracy from 0.174 to 0.408, and option-order answer changes fell from 37% to 6%. The same closed questions were extended through a domain profile to wet-laboratory protocol troubleshooting and to germline target-enrichment panel design, where per-probe keep-or-redesign decisions reduce to a single rule while route, parameter, and escalation choices remain open decisions.

The premise it rests on. The retraining claim assumes that decision accuracy measured on synthetic curriculum states carries over to the agent's real decision states, a transfer the paper demonstrates for only two of the thirteen judgements (analysis family and error class); the further claim that more accurate decisions improve task scores is not established by the reported task-level results.

If the paper is right (Pith's inferences, not the authors')

If the layer works as claimed, a wrong answer becomes attributable to a specific judgement: provenance records link each decision to whether the agent adopted it and whether the task was scored right, so losses can be located decision by decision rather than blamed on the run as a whole.

Fast local judging becomes a near-free substitute for asking the driver model: the same 13 questions cost 8.0% to 35.2% of task cost when answered by the general language model and no request cost when answered by the local model, at a median of 0.6 seconds per request.

Improvement no longer requires changing the agent, its prompts, or its skills: retraining only the decision model under the promotion gate raised accuracy on every trained question in each of three versions, so each behaviour change can be attributed to the decision layer alone.

The harness transfers across domains by adding a profile rather than a new architecture: a protocol profile turned the same closed questions into wet-laboratory diagnosis questions, and probe-panel design was divided into measured state plus eight closed design actions.

Because the local model abstains unless a top option is probable and clearly separated from its runner-up, most decisions leave the agent untouched; the measured cost and step savings therefore come from a small set of firing decisions, so the layer's value lies partly in posing the right question at the right point.

My inference: the architecture reads as a general recipe rather than a biomedical one — any long-horizon agent whose recurring judgements can be posed as closed questions with model-free labels could carry the same layer, and the binding constraint is finding labels that no model produces; that is exactly why conventions, the judgement where the original agent lost most items, were left untrained

My inference: the reported numbers suggest decision accuracy and task score are only loosely coupled — the zero-shot checkpoint scored 0.087 on analysis family and 0.164 on error class on real benchmark states while task scores barely moved, and the trained versions that fixed those accuracies are not reported to improve task outcomes, so the layer's measurable task value may concentrate in a few

My inference: a direct test of the transfer assumption would deploy a gated trained version as judge on held-out benchmark items and report task score, which the paper does not do; until then, the claim that better judgements improve task performance rests on decision accuracy alone.

My inference: the per-probe decision that reduces to a fitted rule reproducing all 3,859 keep-or-redesign labels in-sample is best read as a calibration check on the measuring apparatus itself, which suggests the layer's remaining, untested value sits in the decisions with no simple rule — route, parameter, and escalation choices.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own. Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Every decision is recorded with its consequence. Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. Task-level results in this study use the public checkpoint. The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-02

The authors' main claim, as the classifier picked it out 0.93

“Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.54 · human participants 0.08 · controlled comparison 0.75 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.83 · mentions code or data availability 0.37

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own. background 0.60 0.08 0.04 asserted 1.00
Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. contribution 1.00 0.86 0.17 asserted 0.99
A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. method 0.96 0.87 0.10 asserted 1.00
Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. result 1.00 0.96 0.31 asserted 1.00
Every decision is recorded with its consequence. method 0.95 0.83 0.05 asserted 1.00
Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. method 0.75 0.93 0.09 asserted 1.00
Task-level results in this study use the public checkpoint. method 0.55 0.60 0.02 asserted 0.98
The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. contribution 0.53 0.85 0.05 asserted 1.00
Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained. interpretation 0.75 0.63 0.67 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.76

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-02: 1 request, 7,082 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Operon-S1 separates judging from doing in a biomedical agent. a, The slow system, the ProtoPilot agent, does all computation, from data and code to the answer. The decision harness poses the agent’s recurring judgements at 13 decision points from the start of the task to its submission (
Figure 2 of the paper
Figure 2. Judgements, not the computation, decided most of the original agent’s lost items and time. Trajectories of the original agent under two earlier skill releases. a, Post-answer time: 𝜔 (equation (2)) on the 15 BixBench-Verified-50 tasks with a locatable answer, and the post-answer share of all wall-clock time on SciAgentArena and seqqa2, a lower bound on 𝜔. b, A convention mismatch in which the reference value appeared in the agent’s own output. c, Items lost when a matching skill was skipped or loaded (210 judged trajectories); the annotation gives the range of unaided recovery…
Figure 3 of the paper
Figure 3. Efficiency, accuracy and decision provenance of six controllers in one harness. a, Paired relative change in wall-clock time and agent steps against the baseline (ratio of paired sums minus one) per controller and benchmark. Tasks: SeqQA 600, seqqa2 400, BixBench-Verified-50 50, SciAgentArena 189, BiomniBench-DA 50 (common set at the data cutoff) and protocolqa2 125. Hollow markers, BiomniBench-DA changes of JEV and LAYA, shown for reference (Methods). protocolqa2 is exploratory. b, Paired change in the official score in points (accuracy × 100; BiomniBench-DA rubric points,…
Figure 4 of the paper
Figure 4. Gated retraining of the fast decision model. a, One round of retraining: the data-synthesis pipeline (verified synthesis from an executable curriculum), leakage firewall, hard-example mining with a random-subset control of equal budget, training and calibration, and the promotion gate, whose conditions are given in equations (3) and (14); promoted versions enter the registry and the next round mines their errors. b, Gate-split accuracy per question of the zero-shot public checkpoint (version 0), the majority class (the most frequent training label) and the rule-based controller.…
Figure 5 of the paper
Figure 5. Germline target enrichment panel design. a, Division of labour: tools measure, a slow agent plans and handles exceptions in a pre-synthesis design loop and a post-experiment redesign loop, and the remaining decisions are closed choices among eight design actions. b, Mean per-target design and post-screen coverage (equation (17)) for two designs (90 nt, 45-bp step; 70 nt, 35-bp step), over all 123 targets (82 of which belong to the C4A/C4B pair) and over the 41 targets outside that pair. c, Measured causes of non-coverage and the design actions they map onto: identical paralogous…

Figures are the authors', reproduced from the paper.

References (96)
  1. [1] Jiang, Y., Tang, W., Sun, H., Tang, Z., Hou, Y.et al.A self-evolving agentic system for automated generation and execution of biological protocols. Preprint athttps://arxiv.org/abs/2606.31763(2026) on PithPith's reviewarXiv
  2. [2] Huang, K.et al.Autonomous biomedical research with an artificial intelligence agent.Science393, eadz4351 (2026) on Pith
  3. [3] Gottweis, J.et al.Accelerating scientific discovery with Co-Scientist.Nature655, 487–496 (2026) on Pith
  4. [4] Ghareeb, A. E.et al.A multi-agent system for automating scientific discovery.Nature655, 497–505 (2026) on Pith
  5. [5] Li, O., Agarwal, V., Zhou, S., Gopinath, A. & Kassis, T. K-Dense Analyst: towards fully automated scientific analysis. Preprint athttps://arxiv.org/abs/2508.07043(2025) on PitharXiv
  6. [6] Swanson,K.,Wu,W.,Bulaong,N.L.,Pak,J.E.&Zou,J.TheVirtualLabofAIagentsdesignsnewSARS-CoV-2 nanobodies.Nature646, 716–723 (2025) on Pith
  7. [7] Boiko,D.A.,MacKnight,R.,Kline,B.&Gomes,G.Autonomouschemicalresearchwithlargelanguagemodels. Nature624, 570–578 (2023) on Pith
  8. [8] Bourgon, R., Gentleman, R. & Huber, W. Independent filtering increases detection power for high-throughput experiments.Proc. Natl Acad. Sci. USA107, 9546–9551 (2010) on Pith
  9. [9] Cuadron,A.etal.Thedangerofoverthinking: examiningthereasoning-actiondilemmainagentictasks.Preprint athttps://arxiv.org/abs/2502.08235(2025) on PithPith's reviewarXiv
  10. [10] Chen, X.et al.Do NOT think that much for 2+3=? On the overthinking of o1-like LLMs. Preprint at https://arxiv.org/abs/2412.21187(2024) on PithPith's reviewarXiv
  11. [11] Kahneman, D.Thinking, Fast and Slow(Farrar, Straus and Giroux, 2011) on Pith
  12. [12] Evans, J. St B. T. & Stanovich, K. E. Dual-process theories of higher cognition: advancing the debate.Perspect. Psychol. Sci.8, 223–241 (2013) on Pith
  13. [13] Nelson, T. O. & Narens, L. Metamemory: a theoretical framework and new findings.Psychol. Learn. Motiv.26, 125–173 (1990) on Pith
  14. [14] Almeida, D. Introducing System One models & Jev.TypeSafe AI Bloghttps://typesafe.ai/blog/intro ducing-system-one-models-and-jev(2026) on Pith
  15. [15] Deka, P. From retrieval to typed decisions: calibrated System One models from biomedical sentence encoders. Preprint athttps://arxiv.org/abs/2610.02486(2026) on PithPith's reviewarXiv
  16. [16] Li, J. Fast models, slow evidence: a paired and self-audited evaluation of System-1 decision models for LLM agent harnesses. Preprint athttps://arxiv.org/abs/2610.02267(2026) on PithPith's reviewarXiv
  17. [17] Chen,L.,Zaharia,M.&Zou,J.FrugalGPT:howtouselargelanguagemodelswhilereducingcostandimproving performance. Preprint athttps://arxiv.org/abs/2305.05176(2023) on PithPith's reviewarXiv
  18. [18] Ong, I.et al.RouteLLM: learning to route LLMs with preference data. Preprint athttps://arxiv.org/abs/ 2406.18665(2024) on PithPith's reviewarXiv
  19. [19] Ding, D.et al.Hybrid LLM: cost-efficient and quality-aware query routing. InProc. 12th Int. Conf. Learning Representations(2024) on Pith
  20. [20] Madras, D., Pitassi, T. & Zemel, R. Predict responsibly: improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems31(2018) on Pith
show 76 more
  1. [21] Lightman, H.et al.Let’s verify step by step. InProc. 12th Int. Conf. Learning Representations(2024) on Pith
  2. [22] Cobbe, K.et al.Training verifiers to solve math word problems. Preprint athttps://arxiv.org/abs/2110.1 4168(2021) on Pith
  3. [23] Uesato, J.et al.Solving math word problems with process- and outcome-based feedback. Preprint athttps: //arxiv.org/abs/2211.14275(2022) on PithPith's reviewarXiv
  4. [24] Wang, P.et al.Math-Shepherd: verify and reinforce LLMs step-by-step without human annotations. Preprint at 19 Operon-S1 Genoria AI https://arxiv.org/abs/2312.08935(2023) on PithPith's reviewarXiv
  5. [25] Zheng, L.et al.Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems36(Datasets and Benchmarks Track) (2023) on Pith
  6. [26] Bai, Y.et al.Constitutional AI: harmlessness from AI feedback. Preprint athttps://arxiv.org/abs/2212 .08073(2022) on Pith
  7. [27] McAleese, N.et al.LLM critics help catch LLM bugs. Preprint athttps://arxiv.org/abs/2407.00215 (2024) on PithPith's reviewarXiv
  8. [28] Geifman, Y. & El-Yaniv, R. Selective classification for deep neural networks. InAdvances in Neural Information Processing Systems30(2017) on Pith
  9. [29] Chow, C. K. On optimum recognition error and reject tradeoff.IEEE Trans. Inf. Theory16, 41–46 (1970) on Pith
  10. [30] Angelopoulos, A. N. & Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Preprint athttps://arxiv.org/abs/2107.07511(2021) on PithPith's reviewarXiv
  11. [31] Quach, V.et al.Conformal language modeling. InProc. 12th Int. Conf. Learning Representations(2024) on Pith
  12. [32] Wu, T. & Lim, W. Y. B. REFLEX with Jev for efficient selective control in LLM agents. Preprint athttps: //arxiv.org/abs/2609.26532(2026) on PithPith's reviewarXiv
  13. [33] Deng, B., Fan, S., Zhang, H. & Xie, X. Jev for scientific decisions: evaluating semantic choices and their consequences. Preprint athttps://arxiv.org/abs/2609.24965(2026) on PithPith's reviewarXiv
  14. [34] Cao, C.et al.SearchJev: a calibrated System-1 model for search agents. Preprint athttps://arxiv.org/abs/ 2610.05107(2026) on PithPith's reviewarXiv
  15. [35] Zhang, A.et al.A general-purpose System One decision model for clinical records. Preprint athttps: //arxiv.org/abs/2610.27099(2026) on PithPith's reviewarXiv
  16. [36] AIMultiple. Laya typed-decisions, fine-tuned on browser decisions. Model card athttps://huggingface.co /AIMultiple/laya-typed-decisions-browser-ft(2026) on Pith
  17. [37] Do Rosario, A. & Atef, K. Jev-as-a-Judge for RAG claim verification: confidence-gated cascading between a non-generative verifier and a frontier LLM. Preprint athttps://arxiv.org/abs/2610.20634(2026) on PithPith's reviewarXiv
  18. [38] Shinn, N.et al.Reflexion: language agents with verbal reinforcement learning. Preprint athttps://arxiv.or g/abs/2303.11366(2023) on PithPith's reviewarXiv
  19. [39] Madaan, A.et al.Self-Refine: iterative refinement with self-feedback. Preprint athttps://arxiv.org/abs/ 2303.17651(2023) on PithPith's reviewarXiv
  20. [40] Wang, G.et al.Voyager: an open-ended embodied agent with large language models. Preprint athttps: //arxiv.org/abs/2305.16291(2023) on PithPith's reviewarXiv
  21. [41] Zhang, J., Hu, S., Lu, C., Lange, R. & Clune, J. Darwin Gödel Machine: open-ended evolution of self-improving agents. Preprint athttps://arxiv.org/abs/2505.22954(2025) on PithPith's reviewarXiv
  22. [42] Zelikman, E., Lorch, E., Mackey, L. & Kalai, A. T. Self-taught optimizer (STOP): recursively self-improving code generation. InProc. 1st Conf. Language Modeling(2024) on Pith
  23. [43] Novikov, A.et al.AlphaEvolve: a coding agent for scientific and algorithmic discovery. Preprint athttps: //arxiv.org/abs/2506.13131(2025) on PithPith's reviewarXiv
  24. [44] Guo, Y. Evidence-gated research: statistically controlled model adoption in adaptive search. Preprint at https://arxiv.org/abs/2610.01751(2026) on PithPith's reviewarXiv
  25. [45] Moser, B. B., Frolov, S., Nauen, T. C., Raue, F. & Dengel, A. The self-improvement problem. Preprint at https://arxiv.org/abs/2610.21646(2026) on PithPith's reviewarXiv
  26. [46] Jacob, F. & Monod, J. Genetic regulatory mechanisms in the synthesis of proteins.J. Mol. Biol.3, 318–356 (1961) on Pith
  27. [47] Convai Innovations. Laya Multilingual (convaiinnovations/laya-multilingual) and laya software v0.4.15.https: //huggingface.co/convaiinnovations/laya-multilingual(2026) on Pith
  28. [48] Love, M. I., Huber, W. & Anders, S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2.Genome Biol.15, 550 (2014) on Pith
  29. [49] Laurent, J. M.et al.LAB-Bench: measuring capabilities of language models for biology research. Preprint at https://arxiv.org/abs/2407.10362(2024) on PithPith's reviewarXiv
  30. [50] Laurent, J. M., Bou, A., Pieler, M., Igoe, C.et al.LABBench2: an improved benchmark for AI systems 20 Operon-S1 Genoria AI performing biology research. Preprint athttps://arxiv.org/abs/2604.09554(2026) on PithPith's reviewarXiv
  31. [51] Mitchener, L.et al.BixBench: a comprehensive benchmark for LLM-based agents in computational biology. Preprint athttps://arxiv.org/abs/2503.00096(2025) on PitharXiv
  32. [52] Phylo. BixBench-Verified-50. Dataset athttps://huggingface.co/datasets/phylobio/BixBench-Ver ified-50(2026) on Pith
  33. [53] Liu, T., Wang, A. X., Panescu, A., Chen, L. X.et al.Benchmarking AI agents for addressing scientific challenges across scales. Preprint athttps://arxiv.org/abs/2606.12736(2026) on PithPith's reviewarXiv
  34. [54] Qu, Y., Lu, Y., Tu, X.et al.BiomniBench: process-level evaluation of LLM agents for real-world biomedical research. Preprint at bioRxivhttps://doi.org/10.64898/2026.05.12.724604(2026) on Pithdoi:10.64898/2026.05.12.724604(2026
  35. [55] Phylo. BiomniBench-DA (public release). Dataset athttps://huggingface.co/datasets/phylobio/Bio mniBench-DA(2026) on Pith
  36. [56] Shaked, G. & Nagisetty, S. OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study- scale bioinformatics. Preprint athttps://arxiv.org/abs/2610.21074(2026) on PithPith's reviewarXiv
  37. [57] McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika12, 153–157 (1947) on Pith
  38. [58] Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. InProc. 34th Int. Conf. Machine Learning(PMLR 70) 1321–1330 (2017) on Pith
  39. [59] Segura, S., Fraser, G., Sanchez, A. B. & Ruiz-Cortés, A. A survey on metamorphic testing.IEEE Trans. Softw. Eng.42, 805–824 (2016) on Pith
  40. [60] Ratner, A.et al.Snorkel: rapid training data creation with weak supervision.Proc. VLDB Endow.11, 269–282 (2017) on Pith
  41. [61] Kaufman, S., Rosset, S., Perlich, C. & Stitelman, O. Leakage in data mining: formulation, detection, and avoidance.ACM Trans. Knowl. Discov. Data6, 15 (2012) on Pith
  42. [62] Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science.Patterns 4, 100804 (2023) on Pith
  43. [63] Shrivastava, A., Gupta, A. & Girshick, R. Training region-based object detectors with online hard example mining. InProc. IEEE Conf. Computer Vision and Pattern Recognition761–769 (2016) on Pith
  44. [64] Soni, D. Falsifiable release gates for self-improving systems: standing invariants at scale. Preprint athttps: //arxiv.org/abs/2607.13070(2026) on PithPith's reviewarXiv
  45. [65] Chernozhukov, V.et al.Double/debiased machine learning for treatment and structural parameters.Econom. J. 21, C1–C68 (2018) on Pith
  46. [66] Wolpert, D. H. Stacked generalization.Neural Netw.5, 241–259 (1992) on Pith
  47. [67] Valverde-Albacete, F. J. & Peláez-Moreno, C. 100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox.PLoS ONE9, e84217 (2014) on Pith
  48. [68] Brodersen, K. H., Ong, C. S., Stephan, K. E. & Buhmann, J. M. The balanced accuracy and its posterior distribution. InProc. 20th Int. Conf. Pattern Recognition3121–3124 (2010) on Pith
  49. [69] SantaLucia, J.Jr&Hicks, D.ThethermodynamicsofDNAstructuralmotifs.Annu.Rev.Biophys.Biomol.Struct. 33, 415–440 (2004) on Pith
  50. [70] Mamanova, L.et al.Target-enrichment strategies for next-generation sequencing.Nat. Methods7, 111–118 (2010) on Pith
  51. [71] Gnirke, A.et al.Solution hybrid selection with ultra-long oligonucleotides for massively parallel targeted sequencing.Nat. Biotechnol.27, 182–189 (2009) on Pith
  52. [72] Samorodnitsky, E.et al.Evaluation of hybridization capture versus amplicon-based methods for whole-exome sequencing.Hum. Mutat.36, 903–914 (2015) on Pith
  53. [73] Mandelker, D.et al.Navigating highly homologous genes in a molecular diagnostic setting: a resource for clinical next-generation sequencing.Genet. Med.18, 1282–1289 (2016) on Pith
  54. [74] Chafin,T.K.,Douglas,M.R.&Douglas,M.E.MrBait: universalidentificationanddesignoftargeted-enrichment capture probes.Bioinformatics34, 4293–4296 (2018) on Pith
  55. [75] Li, H. Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM. Preprint athttps: //arxiv.org/abs/1303.3997(2013). 21 Operon-S1 Genoria AI on PithPith's reviewarXiv
  56. [76] Camacho, C.et al.BLAST+: architecture and applications.BMC Bioinformatics10, 421 (2009) on Pith
  57. [77] Church, D. M.et al.Modernizing reference genome assemblies.PLoS Biol.9, e1001091 (2011) on Pith
  58. [78] Zook,J.M.etal.IntegratinghumansequencedatasetsprovidesaresourceofbenchmarkSNPandindelgenotype calls.Nat. Biotechnol.32, 246–251 (2014) on Pith
  59. [79] S1-NexusAgent Team. S1-NexusAgent: a self-evolving agent framework for multidisciplinary scientific research. Preprint athttps://arxiv.org/abs/2602.01550(2026). Methods Slow system The slow system is the ProtoPilot agent runtime1 (frozen commits 7d92ccb, 7417042 and cb88472, e... on PitharXiv
  60. [80] Noeffectoftheeffortsettingonseqqa2wasdetectableatthissamplesize(5against3discordantitems), whereas the skills of the earlier configuration raised the counts from 33 and 31 to 58 and 54. No release containsitemidentifiers,sequences,answersorparaphraseditems. Aleakscanagainstall... on Pith
  61. [81] Robertson, S. & Zaragoza, H. The probabilistic relevance framework: BM25 and beyond.Found. Trends Inf. Retr.3, 333–389 (2009) on Pith
  62. [82] Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. Large language models are not robust multiple choice selectors. Preprint athttps://arxiv.org/abs/2309.03882(2023) on PithPith's reviewarXiv
  63. [83] Hanley, J. A. & McNeil, B. J. The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology143, 29–36 (1982) on Pith
  64. [84] Fleming, S. M. & Lau, H. C. How to measure metacognition.Front. Hum. Neurosci.8, 443 (2014) on Pith
  65. [85] Leinonen, R.et al.The European Nucleotide Archive.Nucleic Acids Res.39, D28–D31 (2011) on Pith
  66. [86] Clopper, C. J. & Pearson, E. S. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika26, 404–413 (1934) on Pith
  67. [87] Efraimidis, P. S. & Spirakis, P. G. Weighted random sampling with a reservoir.Inf. Process. Lett.97, 181–185 (2006) on Pith
  68. [88] Naeini, M. P., Cooper, G. F. & Hauskrecht, M. Obtaining well calibrated probabilities using Bayesian binning. In Proc. 29th AAAI Conf. Artificial Intelligence2901–2907 (2015) on Pith
  69. [89] Field, C. A. & Welsh, A. H. Bootstrapping clustered data.J. R. Stat. Soc. B69, 369–390 (2007) on Pith
  70. [90] Piaggio, G.et al.Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement.JAMA308, 2594–2604 (2012) on Pith
  71. [91] Holm, S. A simple sequentially rejective multiple test procedure.Scand. J. Stat.6, 65–70 (1979) on Pith
  72. [92] Quinlan,A.R.&Hall,I.M.BEDTools: aflexiblesuiteofutilitiesforcomparinggenomicfeatures.Bioinformatics 26, 841–842 (2010) on Pith
  73. [93] Pedersen, B. S. & Quinlan, A. R. Mosdepth: quick coverage calculation for genomes and exomes.Bioinformatics 34, 867–868 (2018) on Pith
  74. [94] Chen, S., Zhou, Y., Chen, Y. & Gu, J. fastp: an ultra-fast all-in-one FASTQ preprocessor.Bioinformatics34, i884–i890 (2018) on Pith
  75. [95] Broad Institute. Picard toolkit.https://broadinstitute.github.io/picard/(2019) on Pith
  76. [96] bamdst: a lightweight BAM depth statistics tool.https://github.com/shiquan/bamdst(2026). Acknowledgements We thank the developers of the benchmarks and open-source tools used in this work. Author contributions Genoria AI Team designed the study, built the system, ran the exper... on Pith
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment