Artificial Intelligence · arXiv:2610.41350 v1 · submitted 30 Oct 2026 · 96 references
Operon-S1: a fast decision controller for long-horizon biomedical agents
The paper says
“Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own.”
more of the abstract
Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Every decision is recorded with its consequence. Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. Task-level results in this study use the public checkpoint. The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 2 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A well-built, honest decision-layer paper whose load-bearing claim—that retraining the judge improves the agent—is not yet demonstrated at task level. This line is wrong
4 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 96 references say.
The paper says, at Results, 'These gains carry over to the agent's own states'; Fig. 4c; Table 3
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The headline analysis-family transfer (0.056→1.000, +0.944 [+0.908, +0.974]) is reported with the parenthetical 'one label value'. If the held-out seqqa2 states carry a single analysis-family label, the majority-class accuracy on those exact states is already 1.000, and rsi-v6's perfect score is consistent with learning the label prior rather than with meaningful transfer to the agent's own states.
more of objection 1
The paper reports majority-class and rule-based accuracies on pooled benchmark states (Extended Data Table 6) but not on the exact held-out state set used for this claim, so the reader cannot tell whether rsi-v6 exceeded the trivial baseline. Please report majority-class, rule-based, and balanced accuracy on the exact held-out analysis-family states, or replace this example with a multi-class judgement.
The paper says, at Results, 'These gains carry over to the agent's own states'; Extended Data Table 6; Eq. (6)
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The second transfer number, execution-labelled error-class accuracy 0.174→0.408 (item-level paired +0.186 [+0.045, +0.329]), is the only unconfounded held-out item-level evidence, and it is weak in absolute terms: 0.408 is far below the rule-based controller's 0.953 on benchmark states and only about 0.025 above the pooled majority rate of 0.383.
more of objection 2
Under the deployed action rule (Eq. 6, tau_act=0.6, delta=0.2), raw accuracy of 0.408 implies that most acted-on error-triage decisions would name the wrong class; the paper does not report the acted-on share or the acted-on accuracy for rsi-v6 under that policy. Additionally, rsi-v6's gate-split accuracy for error class is not reported anywhere, while for rsi-v2 the gate-split error-class accuracy was 0.91 (Fig. 4c); the implied curriculum-to-deployment drop is large and is neither quantified nor discussed.
The paper says, at Abstract; Results, 'Gated retraining'; Extended Data Table 2
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
No trained version is ever run as the controller in the main harness: every LAYA cell in Extended Data Table 2 is version 0, and the paper states that task-level results use the public checkpoint. The central claim that the judgement layer 'can be trained' therefore rests on decision-accuracy and replayed-state evidence only; whether a promoted version improves, leaves unchanged, or degrades task outcomes under the actual agent is untested.
more of objection 3
Given the paper's own demonstration that decision accuracy does not track task outcomes (version 0 has error-class accuracy 0.164 yet reduces steps on protocolqa2, while its seqqa2 score falls by 0.013), this is a substantive empirical gap rather than a formality. I recommend either running a promoted version in the harness on at least a subset of benchmarks (e.g., the held-out halves of seqqa2 and protocolqa2) or explicitly restating the trainability claim so that it excludes task-level consequences.
The paper says, at Abstract vs. Results, 'The local decision model costs little...'; Extended Data Table 2
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The paper's headline efficiency numbers (SeqQA cost −5.8%; protocolqa2 steps −10.3%) are single-benchmark effects, one on a benchmark added after the hypotheses were written, and the pre-registered criterion that an efficiency gain hold on at least four of five hypothesis benchmarks was not met by any arm ('at the data cutoff it holds for no arm').
more of objection 4
In addition, the LAYA version-0 score change on seqqa2 (−0.013 [−0.028, −0.001]) is a measurable degradation. The Results section is transparent about all of this, but the abstract presents the two favourable benchmark results without the H1 status and without the seqqa2 drop; I recommend reporting the H1 outcome and the seqqa2 degradation wherever the efficiency claims are summarized.
4 minor objections
The paper says, at Abstract and Results, controller latency
Pith says: minor objection 1 of 4
The abstract's '0.6 s at the median for an in-loop request' matches the error-triage point (P3, 576 ms), but in-loop medians range from 531 ms (P8) to 839 ms (P7); please report the range in the abstract or qualify the single number.
The paper says, at Results, injected-message audit
Pith says: minor objection 2 of 4
The audit statement 'no injected message contained the gold answer' applies only to the 2,361 runs whose answers can be matched as text; the remaining 3,382 runs cannot be checked. The main text discloses this immediately after the audit sentence, but the claim is repeated in the opening results section without the coverage caveat, and the coverage should travel with the claim.
show 2 more
The paper says, at Results, probe design
Pith says: minor objection 3 of 4
The sentence 'One rule over visible features, fitted to the labels, reproduces all 3,859 keep-or-redesign labels in-sample' describes an in-sample fit to the labels it reproduces; as the paper itself frames this as a check of the layer against a known answer, I suggest labelling it explicitly as a sanity check so that the in-sample reproduction is not read as validation.
The paper says, at Extended Data Table 2 and Methods, benchmarks
Pith says: minor objection 4 of 4
Table 2 lists substitute judge models for several benchmarks (BiomniBench-DA, BixBench range answers, protocolqa2), and the ratio of wall-clock changes on BiomniBench-DA is reported with a concurrency caveat. Within-arm comparisons are fair because the judge is identical across arms, but the manuscript should state once, in the main text, that scores are not directly comparable to published benchmark leaderboards because of the substitute judges and doubled time limits.
What would settle it
Run a promoted version, e.g. rsi-v6, as the LAYA controller on the held-out halves of seqqa2 and protocolqa2 under the same harness, with version 0 and no-op arms, and report paired score, cost and steps. In parallel, on the exact held-out states used for Table 3, report majority-class and rule-based accuracy for analysis family and error class alongside rsi-v6. If rsi-v6 does not beat v0/no-op on task outcomes, or if its held-out transfer accuracy is within the reported interval of the majority class or below the rule-based controller, the claim that the trainable layer improves the agent at the task level is not supported.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 2 Nov 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Here is my read of Operon-S1. The genuinely new thing is not any single mechanism but the full package: 13 fixed decision points, closed questions, fixed messages that never contain answers, per-decision provenance, a leakage firewall, and a promotion gate with majority-class and random-subset controls, with labels drawn only from construction, verifiers, or execution. That is a useful design for auditable, self-improving agents, and it is implemented carefully. The no-op control, the message audit, the non-inferiority framework, and the honest labeling of protocolqa2 as exploratory all show real discipline.
What holds up: a separate fast judge can be cheap and can leave scores mostly unchanged. With the public zero-shot checkpoint, the cost savings on SeqQA and the step savings on protocolqa2 are real, and no measurable score change appears on five of six benchmarks. Retraining does raise decision accuracy on gate splits across all trained versions. That part is solid.
The soft spot is exactly where the reader and stress-test put it: the abstract’s “gains carried over” is doing more work than two numbers can support. On the seqqa2 analysis-family held-out states, the paper itself reports “one label value.” If those states are majority-class one, the 0.056 to 1.000 jump is mostly learning the prior. The error-class transfer is item-level and better, but 0.408 is still far below the rule-based controller’s 0.953 on benchmark states and only slightly above majority. And no trained version is ever run as the controller in the main harness—task-level results all use version 0. So the trainable-controller claim is supported only on decision accuracy, not on task outcomes.
The authors disclose most of this gap somewhere in the text, which I credit. The failure of their pre-registered H1 for every arm is reported plainly, not buried. That is a point in their favor.
What I would want before trusting the stronger claim: one promoted version run end-to-end at task level, with the held-out states reporting majority-class accuracy alongside the model’s, and shipped code and data. Right now the artifacts are “available from Genoria AI,” which is not the same as available.
Who gets value: anyone building agent harnesses, decision layers, or release gates for self-improving systems. It deserves a serious referee; the gap between decision accuracy and task-level benefit is the thing to push on, and it is fixable in a revision. I would take it to our reading group.
The strongest objection
The central value claim is that the decision layer can be measured, replaced and trained. The main evaluation uses LAYA version 0 zero-shot, and the authors state that task-level results use the public checkpoint and that the quantity training targets is decision accuracy rather than task score. The load-bearing bridge is therefore the claim that training gains carry over to the agent's own decision states. That bridge rests on two heldout numbers in Table 3 and Fig. 4c: seqqa2 analysis-family accuracy 0.056->1.000, and execution-labelled error-class accuracy 0.174->0.408. The analysis-family result is accompanied by the text 'one label value'. If the heldout seqqa2 states carry a single analysis-family label, then the majority class on those same states already scores 1.000, so rsi-v6's perfect score demonstrates learning the label prior rather than meaningful transfer; the majority-class and rule-based accuracies on those exact heldout states are not reported. The error-class result is better because it is item-level paired, but 0.408 remains far below the rule-based controller's 0.953 on benchmark states and only about 0.025 above the majority-class rate of 0.383 (Extended Data Table 6). These numbers are consistent with real but functionally small transfer. No promoted version is ever run as the controller in the main harness, so the step from 'better decision accuracy on replayed states' to 'better or cheaper task outcomes under the same agent' is untested. The paper discloses part of this gap explicitly, but the abstract's 'gains carried over' formulation gives the transfer evidence more weight than the single-label and below-rule numbers can carry.
The reader's weakest assumption is that decision accuracy on synthetic curriculum states transfers to task-level benefit. My read agrees, and the same concern is load-bearing because the abstract and results highlight retraining gains as evidence for the trainable-layer claim. I sharpen the reader's point in two ways. First, the flagship transfer number, analysis-family 0.056->1.000 on seqqa2, is accompanied by 'one label value'; if that means all heldout states share one label, the majority class already achieves 1.000 on those states, so the result is not evidence of transfer beyond the label prior. Second, the other transfer number, error-class 0.174->0.408, is well below the rule-based controller's benchmark-state accuracy of 0.953 and only marginally above the majority rate of 0.383, so the demonstrated transfer is functionally small even if statistically real. The paper is transparent that task-level results use version 0 and that the gate certifies decision accuracy, and I do not treat that as a hidden flaw. But it means the central controller claim still depends on an untested link: a promoted version running in the loop and improving task outcomes. The proposed concrete test would settle that link. This does not change the reader's verdict: conditional acceptance remains appropriate, pending a task-level run of a promoted version and the missing majority/rule comparison on the same heldout states.
The referee's report
The conceptual contribution — separating an agent's recurring scientific judgements from its computation and treating them as a measured, replaceable, trainable layer whose labels no model produces — is valuable and timely, and the engineering is unusually careful. Strengths that deserve explicit credit include: a no-op arm with a regression test (Eq. 8) enforcing equivalence to the original agent; a leakage firewall (Eq. 11); a promotion gate (Eq. 14) with majority-class and random-subset controls and an explicit accuracy-paradox discussion; cross-fitting for any benchmark states entering training; an append-only registry with SHA-256 identities; a dated, itemized log of deviations from the evaluation plan; and transparent reporting of the H1 failure and of the seqqa2 score losses. If the transfer claims are adequately supported, the paper would provide a reference architecture for judgement-layer design in scientific agents. At present the task-level value of the retrained versions is unproven, and the two transfer numbers carry most of that load; one of them is potentially confounded by a single-label prior.
Pith's reading of the paper (a machine's summary, not the authors')
The paper claims that the limit on long-horizon biomedical agents has moved from computation to judgement: most of the original agent's wrong answers and surplus time came from choices about conventions, skills, error classes, evidence and stopping that were made inside the reasoning model and could not be measured or fixed on their own. Operon-S1 separates judging from doing by posing those recurring judgements as 13 closed questions during a run and handing them to a fast local decision model that acts only through pre-written messages that never contain an answer. With the public checkpoint of the local model as judge, this layer cut cost on SeqQA by 5.8% and agent steps on protocolqa2 by 10.3%, with no measurable score change on five of six benchmarks, at a median of about 0.6 seconds per in-loop request and no request cost. Retraining on labels drawn only from task construction, program verifiers, or code execution raised decision accuracy on every question in each of three trained versions, and the gains carried over to the agent's own decision states on held-out items. A promotion gate admits a new version only when it beats its parent, a rule-based controller, the majority class, and a random-subset control, so the layer's evidence of improvement never comes from a language model.
What carries the argument. The load-bearing object is the decision harness: 13 decision points, each a tuple of a trigger, a deterministic state renderer, and closed questions over finite option sets, mapped through one fixed policy to four actions (brief, remind, end the tool loop, request one recheck) delivered in pre-written messages that never contain an answer. States are rendered as pure functions of the run's facts — task prompt, workspace listing, environments, skill shortlist, budget fraction — so the agent never assembles its own context and any decision can be replayed; all controllers share the policy, thresholds, and messages and differ only in the probability distributions they return, making a comparison between controllers a comparison between judgements. Two properties make the layer trainable. Labels come only from construction, a verifier, or code execution, never from a model, and a promotion gate admits a candidate only when it beats its parent, the rule-based controller, the majority class, and a random-subset control of equal training budget on every question with headroom, with expected calibration error at most 0.10 and a lower cluster-bootstrap bound above zero. The gate is part of the scientific claim: an agent that improved on labels its own training loop generated could be fooling itself, and the paper cites the result that no automated gate can in general refuse every bad update and accept every good one, so the gate's guarantees are confined to a bounded, labelled split.
Core claim. The central claim is that an agent's scientific judgements form a layer of their own, distinct from the computation they steer: each judgement can be posed as a closed question, scored against a label that no model produces, and improved by retraining under a statistical gate. The paper builds this from a 13-point decision harness whose states are rendered deterministically from the facts of the run, so every decision can be replayed, and whose actions are four fixed message types — brief, remind, end the tool loop, request one recheck — that may name a convention, a file, or a quantity but never an answer value. Six controllers ran in the same harness on seven life-science benchmarks; the no-op control established that the harness itself leaves the agent unchanged, and an audit of 16,042 injected messages found none containing a gold answer in any run whose answer can be matched as text. As judge, the public LAYA checkpoint (a 322-million-parameter local encoder) accounted for 0.8% to 5.2% of wall-clock time at no request cost, lowered cost on SeqQA by 5.8% and steps on protocolqa2 by 10.3%, and changed no score outside its interval on five of six benchmarks, while the general language model driving the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Retraining on verifiable labels raised accuracy on every decision question in three trained versions, with transfer measured on the agent's own held-out states: analysis-family accuracy on seqqa2 rose from 0.056 to 1.000 and execution-labelled error-class accuracy from 0.174 to 0.408, and option-order answer changes fell from 37% to 6%. The same closed questions were extended through a domain profile to wet-laboratory protocol troubleshooting and to germline target-enrichment panel design, where per-probe keep-or-redesign decisions reduce to a single rule while route, parameter, and escalation choices remain open decisions.
The premise it rests on. The retraining claim assumes that decision accuracy measured on synthetic curriculum states carries over to the agent's real decision states, a transfer the paper demonstrates for only two of the thirteen judgements (analysis family and error class); the further claim that more accurate decisions improve task scores is not established by the reported task-level results.
If the paper is right (Pith's inferences, not the authors')
If the layer works as claimed, a wrong answer becomes attributable to a specific judgement: provenance records link each decision to whether the agent adopted it and whether the task was scored right, so losses can be located decision by decision rather than blamed on the run as a whole.
Fast local judging becomes a near-free substitute for asking the driver model: the same 13 questions cost 8.0% to 35.2% of task cost when answered by the general language model and no request cost when answered by the local model, at a median of 0.6 seconds per request.
Improvement no longer requires changing the agent, its prompts, or its skills: retraining only the decision model under the promotion gate raised accuracy on every trained question in each of three versions, so each behaviour change can be attributed to the decision layer alone.
The harness transfers across domains by adding a profile rather than a new architecture: a protocol profile turned the same closed questions into wet-laboratory diagnosis questions, and probe-panel design was divided into measured state plus eight closed design actions.
Because the local model abstains unless a top option is probable and clearly separated from its runner-up, most decisions leave the agent untouched; the measured cost and step savings therefore come from a small set of firing decisions, so the layer's value lies partly in posing the right question at the right point.
My inference: the architecture reads as a general recipe rather than a biomedical one — any long-horizon agent whose recurring judgements can be posed as closed questions with model-free labels could carry the same layer, and the binding constraint is finding labels that no model produces; that is exactly why conventions, the judgement where the original agent lost most items, were left untrained
My inference: the reported numbers suggest decision accuracy and task score are only loosely coupled — the zero-shot checkpoint scored 0.087 on analysis family and 0.164 on error class on real benchmark states while task scores barely moved, and the trained versions that fixed those accuracies are not reported to improve task outcomes, so the layer's measurable task value may concentrate in a few
My inference: a direct test of the transfer assumption would deploy a gated trained version as judge on held-out benchmark items and report task score, which the paper does not do; until then, the claim that better judgements improve task performance rests on decision accuracy alone.
My inference: the per-probe decision that reduces to a fitted rule reproducing all 3,859 keep-or-redesign labels in-sample is best read as a calibration check on the measuring apparatus itself, which suggests the layer's remaining, untested value sits in the decisions with no simple rule — route, parameter, and escalation choices.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own. Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Every decision is recorded with its consequence. Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. Task-level results in this study use the public checkpoint. The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-02
The authors' main claim, as the classifier picked it out 0.93
“Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.54 · human participants 0.08 · controlled comparison 0.75 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.83 · mentions code or data availability 0.37
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own. | background 0.60 | 0.08 | 0.04 | asserted 1.00 |
| Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. | contribution 1.00 | 0.86 | 0.17 | asserted 0.99 |
| A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. | method 0.96 | 0.87 | 0.10 | asserted 1.00 |
| Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. | result 1.00 | 0.96 | 0.31 | asserted 1.00 |
| Every decision is recorded with its consequence. | method 0.95 | 0.83 | 0.05 | asserted 1.00 |
| Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. | method 0.75 | 0.93 | 0.09 | asserted 1.00 |
| Task-level results in this study use the public checkpoint. | method 0.55 | 0.60 | 0.02 | asserted 0.98 |
| The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. | contribution 0.53 | 0.85 | 0.05 | asserted 1.00 |
| Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained. | interpretation 0.75 | 0.63 | 0.67 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.76
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-02: 1 request, 7,082 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (96)
- [1] Jiang, Y., Tang, W., Sun, H., Tang, Z., Hou, Y.et al.A self-evolving agentic system for automated generation and execution of biological protocols. Preprint athttps://arxiv.org/abs/2606.31763(2026) on PithPith's reviewarXiv
- [2] Huang, K.et al.Autonomous biomedical research with an artificial intelligence agent.Science393, eadz4351 (2026) on Pith
- [3] Gottweis, J.et al.Accelerating scientific discovery with Co-Scientist.Nature655, 487–496 (2026) on Pith
- [4] Ghareeb, A. E.et al.A multi-agent system for automating scientific discovery.Nature655, 497–505 (2026) on Pith
- [5] Li, O., Agarwal, V., Zhou, S., Gopinath, A. & Kassis, T. K-Dense Analyst: towards fully automated scientific analysis. Preprint athttps://arxiv.org/abs/2508.07043(2025) on PitharXiv
- [6] Swanson,K.,Wu,W.,Bulaong,N.L.,Pak,J.E.&Zou,J.TheVirtualLabofAIagentsdesignsnewSARS-CoV-2 nanobodies.Nature646, 716–723 (2025) on Pith
- [7] Boiko,D.A.,MacKnight,R.,Kline,B.&Gomes,G.Autonomouschemicalresearchwithlargelanguagemodels. Nature624, 570–578 (2023) on Pith
- [8] Bourgon, R., Gentleman, R. & Huber, W. Independent filtering increases detection power for high-throughput experiments.Proc. Natl Acad. Sci. USA107, 9546–9551 (2010) on Pith
- [9] Cuadron,A.etal.Thedangerofoverthinking: examiningthereasoning-actiondilemmainagentictasks.Preprint athttps://arxiv.org/abs/2502.08235(2025) on PithPith's reviewarXiv
- [10] Chen, X.et al.Do NOT think that much for 2+3=? On the overthinking of o1-like LLMs. Preprint at https://arxiv.org/abs/2412.21187(2024) on PithPith's reviewarXiv
- [11] Kahneman, D.Thinking, Fast and Slow(Farrar, Straus and Giroux, 2011) on Pith
- [12] Evans, J. St B. T. & Stanovich, K. E. Dual-process theories of higher cognition: advancing the debate.Perspect. Psychol. Sci.8, 223–241 (2013) on Pith
- [13] Nelson, T. O. & Narens, L. Metamemory: a theoretical framework and new findings.Psychol. Learn. Motiv.26, 125–173 (1990) on Pith
- [14] Almeida, D. Introducing System One models & Jev.TypeSafe AI Bloghttps://typesafe.ai/blog/intro ducing-system-one-models-and-jev(2026) on Pith
- [15] Deka, P. From retrieval to typed decisions: calibrated System One models from biomedical sentence encoders. Preprint athttps://arxiv.org/abs/2610.02486(2026) on PithPith's reviewarXiv
- [16] Li, J. Fast models, slow evidence: a paired and self-audited evaluation of System-1 decision models for LLM agent harnesses. Preprint athttps://arxiv.org/abs/2610.02267(2026) on PithPith's reviewarXiv
- [17] Chen,L.,Zaharia,M.&Zou,J.FrugalGPT:howtouselargelanguagemodelswhilereducingcostandimproving performance. Preprint athttps://arxiv.org/abs/2305.05176(2023) on PithPith's reviewarXiv
- [18] Ong, I.et al.RouteLLM: learning to route LLMs with preference data. Preprint athttps://arxiv.org/abs/ 2406.18665(2024) on PithPith's reviewarXiv
- [19] Ding, D.et al.Hybrid LLM: cost-efficient and quality-aware query routing. InProc. 12th Int. Conf. Learning Representations(2024) on Pith
- [20] Madras, D., Pitassi, T. & Zemel, R. Predict responsibly: improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems31(2018) on Pith
show 76 more
- [21] Lightman, H.et al.Let’s verify step by step. InProc. 12th Int. Conf. Learning Representations(2024) on Pith
- [22] Cobbe, K.et al.Training verifiers to solve math word problems. Preprint athttps://arxiv.org/abs/2110.1 4168(2021) on Pith
- [23] Uesato, J.et al.Solving math word problems with process- and outcome-based feedback. Preprint athttps: //arxiv.org/abs/2211.14275(2022) on PithPith's reviewarXiv
- [24] Wang, P.et al.Math-Shepherd: verify and reinforce LLMs step-by-step without human annotations. Preprint at 19 Operon-S1 Genoria AI https://arxiv.org/abs/2312.08935(2023) on PithPith's reviewarXiv
- [25] Zheng, L.et al.Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems36(Datasets and Benchmarks Track) (2023) on Pith
- [26] Bai, Y.et al.Constitutional AI: harmlessness from AI feedback. Preprint athttps://arxiv.org/abs/2212 .08073(2022) on Pith
- [27] McAleese, N.et al.LLM critics help catch LLM bugs. Preprint athttps://arxiv.org/abs/2407.00215 (2024) on PithPith's reviewarXiv
- [28] Geifman, Y. & El-Yaniv, R. Selective classification for deep neural networks. InAdvances in Neural Information Processing Systems30(2017) on Pith
- [29] Chow, C. K. On optimum recognition error and reject tradeoff.IEEE Trans. Inf. Theory16, 41–46 (1970) on Pith
- [30] Angelopoulos, A. N. & Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Preprint athttps://arxiv.org/abs/2107.07511(2021) on PithPith's reviewarXiv
- [31] Quach, V.et al.Conformal language modeling. InProc. 12th Int. Conf. Learning Representations(2024) on Pith
- [32] Wu, T. & Lim, W. Y. B. REFLEX with Jev for efficient selective control in LLM agents. Preprint athttps: //arxiv.org/abs/2609.26532(2026) on PithPith's reviewarXiv
- [33] Deng, B., Fan, S., Zhang, H. & Xie, X. Jev for scientific decisions: evaluating semantic choices and their consequences. Preprint athttps://arxiv.org/abs/2609.24965(2026) on PithPith's reviewarXiv
- [34] Cao, C.et al.SearchJev: a calibrated System-1 model for search agents. Preprint athttps://arxiv.org/abs/ 2610.05107(2026) on PithPith's reviewarXiv
- [35] Zhang, A.et al.A general-purpose System One decision model for clinical records. Preprint athttps: //arxiv.org/abs/2610.27099(2026) on PithPith's reviewarXiv
- [36] AIMultiple. Laya typed-decisions, fine-tuned on browser decisions. Model card athttps://huggingface.co /AIMultiple/laya-typed-decisions-browser-ft(2026) on Pith
- [37] Do Rosario, A. & Atef, K. Jev-as-a-Judge for RAG claim verification: confidence-gated cascading between a non-generative verifier and a frontier LLM. Preprint athttps://arxiv.org/abs/2610.20634(2026) on PithPith's reviewarXiv
- [38] Shinn, N.et al.Reflexion: language agents with verbal reinforcement learning. Preprint athttps://arxiv.or g/abs/2303.11366(2023) on PithPith's reviewarXiv
- [39] Madaan, A.et al.Self-Refine: iterative refinement with self-feedback. Preprint athttps://arxiv.org/abs/ 2303.17651(2023) on PithPith's reviewarXiv
- [40] Wang, G.et al.Voyager: an open-ended embodied agent with large language models. Preprint athttps: //arxiv.org/abs/2305.16291(2023) on PithPith's reviewarXiv
- [41] Zhang, J., Hu, S., Lu, C., Lange, R. & Clune, J. Darwin Gödel Machine: open-ended evolution of self-improving agents. Preprint athttps://arxiv.org/abs/2505.22954(2025) on PithPith's reviewarXiv
- [42] Zelikman, E., Lorch, E., Mackey, L. & Kalai, A. T. Self-taught optimizer (STOP): recursively self-improving code generation. InProc. 1st Conf. Language Modeling(2024) on Pith
- [43] Novikov, A.et al.AlphaEvolve: a coding agent for scientific and algorithmic discovery. Preprint athttps: //arxiv.org/abs/2506.13131(2025) on PithPith's reviewarXiv
- [44] Guo, Y. Evidence-gated research: statistically controlled model adoption in adaptive search. Preprint at https://arxiv.org/abs/2610.01751(2026) on PithPith's reviewarXiv
- [45] Moser, B. B., Frolov, S., Nauen, T. C., Raue, F. & Dengel, A. The self-improvement problem. Preprint at https://arxiv.org/abs/2610.21646(2026) on PithPith's reviewarXiv
- [46] Jacob, F. & Monod, J. Genetic regulatory mechanisms in the synthesis of proteins.J. Mol. Biol.3, 318–356 (1961) on Pith
- [47] Convai Innovations. Laya Multilingual (convaiinnovations/laya-multilingual) and laya software v0.4.15.https: //huggingface.co/convaiinnovations/laya-multilingual(2026) on Pith
- [48] Love, M. I., Huber, W. & Anders, S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2.Genome Biol.15, 550 (2014) on Pith
- [49] Laurent, J. M.et al.LAB-Bench: measuring capabilities of language models for biology research. Preprint at https://arxiv.org/abs/2407.10362(2024) on PithPith's reviewarXiv
- [50] Laurent, J. M., Bou, A., Pieler, M., Igoe, C.et al.LABBench2: an improved benchmark for AI systems 20 Operon-S1 Genoria AI performing biology research. Preprint athttps://arxiv.org/abs/2604.09554(2026) on PithPith's reviewarXiv
- [51] Mitchener, L.et al.BixBench: a comprehensive benchmark for LLM-based agents in computational biology. Preprint athttps://arxiv.org/abs/2503.00096(2025) on PitharXiv
- [52] Phylo. BixBench-Verified-50. Dataset athttps://huggingface.co/datasets/phylobio/BixBench-Ver ified-50(2026) on Pith
- [53] Liu, T., Wang, A. X., Panescu, A., Chen, L. X.et al.Benchmarking AI agents for addressing scientific challenges across scales. Preprint athttps://arxiv.org/abs/2606.12736(2026) on PithPith's reviewarXiv
- [54] Qu, Y., Lu, Y., Tu, X.et al.BiomniBench: process-level evaluation of LLM agents for real-world biomedical research. Preprint at bioRxivhttps://doi.org/10.64898/2026.05.12.724604(2026) on Pithdoi:10.64898/2026.05.12.724604(2026
- [55] Phylo. BiomniBench-DA (public release). Dataset athttps://huggingface.co/datasets/phylobio/Bio mniBench-DA(2026) on Pith
- [56] Shaked, G. & Nagisetty, S. OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study- scale bioinformatics. Preprint athttps://arxiv.org/abs/2610.21074(2026) on PithPith's reviewarXiv
- [57] McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika12, 153–157 (1947) on Pith
- [58] Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. InProc. 34th Int. Conf. Machine Learning(PMLR 70) 1321–1330 (2017) on Pith
- [59] Segura, S., Fraser, G., Sanchez, A. B. & Ruiz-Cortés, A. A survey on metamorphic testing.IEEE Trans. Softw. Eng.42, 805–824 (2016) on Pith
- [60] Ratner, A.et al.Snorkel: rapid training data creation with weak supervision.Proc. VLDB Endow.11, 269–282 (2017) on Pith
- [61] Kaufman, S., Rosset, S., Perlich, C. & Stitelman, O. Leakage in data mining: formulation, detection, and avoidance.ACM Trans. Knowl. Discov. Data6, 15 (2012) on Pith
- [62] Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science.Patterns 4, 100804 (2023) on Pith
- [63] Shrivastava, A., Gupta, A. & Girshick, R. Training region-based object detectors with online hard example mining. InProc. IEEE Conf. Computer Vision and Pattern Recognition761–769 (2016) on Pith
- [64] Soni, D. Falsifiable release gates for self-improving systems: standing invariants at scale. Preprint athttps: //arxiv.org/abs/2607.13070(2026) on PithPith's reviewarXiv
- [65] Chernozhukov, V.et al.Double/debiased machine learning for treatment and structural parameters.Econom. J. 21, C1–C68 (2018) on Pith
- [66] Wolpert, D. H. Stacked generalization.Neural Netw.5, 241–259 (1992) on Pith
- [67] Valverde-Albacete, F. J. & Peláez-Moreno, C. 100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox.PLoS ONE9, e84217 (2014) on Pith
- [68] Brodersen, K. H., Ong, C. S., Stephan, K. E. & Buhmann, J. M. The balanced accuracy and its posterior distribution. InProc. 20th Int. Conf. Pattern Recognition3121–3124 (2010) on Pith
- [69] SantaLucia, J.Jr&Hicks, D.ThethermodynamicsofDNAstructuralmotifs.Annu.Rev.Biophys.Biomol.Struct. 33, 415–440 (2004) on Pith
- [70] Mamanova, L.et al.Target-enrichment strategies for next-generation sequencing.Nat. Methods7, 111–118 (2010) on Pith
- [71] Gnirke, A.et al.Solution hybrid selection with ultra-long oligonucleotides for massively parallel targeted sequencing.Nat. Biotechnol.27, 182–189 (2009) on Pith
- [72] Samorodnitsky, E.et al.Evaluation of hybridization capture versus amplicon-based methods for whole-exome sequencing.Hum. Mutat.36, 903–914 (2015) on Pith
- [73] Mandelker, D.et al.Navigating highly homologous genes in a molecular diagnostic setting: a resource for clinical next-generation sequencing.Genet. Med.18, 1282–1289 (2016) on Pith
- [74] Chafin,T.K.,Douglas,M.R.&Douglas,M.E.MrBait: universalidentificationanddesignoftargeted-enrichment capture probes.Bioinformatics34, 4293–4296 (2018) on Pith
- [75] Li, H. Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM. Preprint athttps: //arxiv.org/abs/1303.3997(2013). 21 Operon-S1 Genoria AI on PithPith's reviewarXiv
- [76] Camacho, C.et al.BLAST+: architecture and applications.BMC Bioinformatics10, 421 (2009) on Pith
- [77] Church, D. M.et al.Modernizing reference genome assemblies.PLoS Biol.9, e1001091 (2011) on Pith
- [78] Zook,J.M.etal.IntegratinghumansequencedatasetsprovidesaresourceofbenchmarkSNPandindelgenotype calls.Nat. Biotechnol.32, 246–251 (2014) on Pith
- [79] S1-NexusAgent Team. S1-NexusAgent: a self-evolving agent framework for multidisciplinary scientific research. Preprint athttps://arxiv.org/abs/2602.01550(2026). Methods Slow system The slow system is the ProtoPilot agent runtime1 (frozen commits 7d92ccb, 7417042 and cb88472, e... on PitharXiv
- [80] Noeffectoftheeffortsettingonseqqa2wasdetectableatthissamplesize(5against3discordantitems), whereas the skills of the earlier configuration raised the counts from 33 and 31 to 58 and 54. No release containsitemidentifiers,sequences,answersorparaphraseditems. Aleakscanagainstall... on Pith
- [81] Robertson, S. & Zaragoza, H. The probabilistic relevance framework: BM25 and beyond.Found. Trends Inf. Retr.3, 333–389 (2009) on Pith
- [82] Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. Large language models are not robust multiple choice selectors. Preprint athttps://arxiv.org/abs/2309.03882(2023) on PithPith's reviewarXiv
- [83] Hanley, J. A. & McNeil, B. J. The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology143, 29–36 (1982) on Pith
- [84] Fleming, S. M. & Lau, H. C. How to measure metacognition.Front. Hum. Neurosci.8, 443 (2014) on Pith
- [85] Leinonen, R.et al.The European Nucleotide Archive.Nucleic Acids Res.39, D28–D31 (2011) on Pith
- [86] Clopper, C. J. & Pearson, E. S. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika26, 404–413 (1934) on Pith
- [87] Efraimidis, P. S. & Spirakis, P. G. Weighted random sampling with a reservoir.Inf. Process. Lett.97, 181–185 (2006) on Pith
- [88] Naeini, M. P., Cooper, G. F. & Hauskrecht, M. Obtaining well calibrated probabilities using Bayesian binning. In Proc. 29th AAAI Conf. Artificial Intelligence2901–2907 (2015) on Pith
- [89] Field, C. A. & Welsh, A. H. Bootstrapping clustered data.J. R. Stat. Soc. B69, 369–390 (2007) on Pith
- [90] Piaggio, G.et al.Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement.JAMA308, 2594–2604 (2012) on Pith
- [91] Holm, S. A simple sequentially rejective multiple test procedure.Scand. J. Stat.6, 65–70 (1979) on Pith
- [92] Quinlan,A.R.&Hall,I.M.BEDTools: aflexiblesuiteofutilitiesforcomparinggenomicfeatures.Bioinformatics 26, 841–842 (2010) on Pith
- [93] Pedersen, B. S. & Quinlan, A. R. Mosdepth: quick coverage calculation for genomes and exomes.Bioinformatics 34, 867–868 (2018) on Pith
- [94] Chen, S., Zhou, Y., Chen, Y. & Gu, J. fastp: an ultra-fast all-in-one FASTQ preprocessor.Bioinformatics34, i884–i890 (2018) on Pith
- [95] Broad Institute. Picard toolkit.https://broadinstitute.github.io/picard/(2019) on Pith
- [96] bamdst: a lightweight BAM depth statistics tool.https://github.com/shiquan/bamdst(2026). Acknowledgements We thank the developers of the benchmarks and open-source tools used in this work. Author contributions Genoria AI Team designed the study, built the system, ran the exper... on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment