{"id":"a1ddae29-5bc7-49e2-8841-45951d6b7acb","arxiv_id":"2607.23975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Plato-Bio is a reproducible, audit-friendly biology agent pipeline whose full test suite passes, whose one historical evidence-bridge task ranks before TF-IDF, and whose 15-protein structure screen finds 11 targets under 1 Å core RMSD.","lead":"Plato-Bio adds verification and provenance controls to an open research-agent codebase, then tests it on a single historical literature-search case and a 15-protein AlphaFold comparison. The value is a reproducible screening scaffold; the biological results are explicitly provisional and one-case-only.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Structural screen fits the same high-confidence core it audits, so the 27 discrepancy regions and core RMSD values are partially optimized residuals; a leave-one-out refit is required.","rationale":"The reader's weakest_assumption targeted the temporal benchmark, but the paper explicitly disclaims prospective discovery there and labels the task 'one manually curated task.' That concern is real but already well hedged. The structural screen, by contrast, is presented as an 'auditable screening baseline' with concrete outputs (core RMSD values and 27 discrepancy regions), yet the fitting and screening procedures share the same high-confidence residues. This is an unacknowledged circularity that can bias the headline numbers of the second benchmark. It is therefore the more load-bearing concern for the central claim about auditable screening baselines. The reader's rationale did mention 'a structural comparison that fits and evaluates on the same high-confidence core,' so there is partial agreement, but it was not the stated weakest assumption. My recommendation is UNCHANGED because the paper's core software-contract claim remains credible and the structural issue is testable and hedged as hypothesis-only; however, the structural baseline numbers should not be cited without the leave-one-out check.","tokens_in":9623,"tokens_out":9781,"duration_ms":107300,"concrete_test":"Re-run the diverse-structure analysis with a leave-one-out protocol: for each candidate discrepancy region, recompute the Kabsch fit on the pLDDT>=70 core excluding that region's residues, apply the resulting transform to all residues, and re-evaluate the region's core-aligned error; also run a whole-chain-fit variant. Compare the number, membership, and per-target classification of the 27 discrepancy regions and the four >2 Å targets against the shipped residue-level CSV. If the flagged set contracts substantially (for example, fewer than 20 regions or a target falls below 2 Å), the reported structural screening baseline is partly an artifact of fitting on the audited core.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Plato-Bio provides auditable screening baselines, including the 15-target structural screen. The screen computes a Kabsch fit using only residues with pLDDT >= 70, then flags discrepancy regions among residues with pLDDT >= 90 and core-aligned C alpha error >= 2 Å (§2.8, §3.5). Because the pLDDT>=90 residues are a subset of the fitting core, the rigid-body transform is least-squares optimized on exactly those residues whose residuals define the flags. Consequently, the reported core RMSD values (median 0.501 Å; 11/15 < 1 Å) and the list of 27 discrepancy regions are not independent of the alignment: residual errors in flagged regions can be reduced by the fitting procedure itself, and the triage queue may miss or reorder genuine discrepancies. The paper's limitation section (§4.1) notes that the mask is pLDDT-based rather than an independently selected structural domain, but it does not disclose the same-core fitting-and-screening circularity. This does not undermine the software-contract claim, but it weakens the structural screening baseline as an auditable measure of novelty triage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Plato-Bio, a biology-routed extension of the Plato/Denario agent architecture, and argues that its value lies in verification and auditability rather than in autonomous scientific discovery. It describes three source-level measurement-validity repairs, deterministic software validation (931 passes, 6 skips, no failures), a frozen single-task temporal rediscovery benchmark in which evidence-bridge ranking places the later-validated fish-oil/Raynaud relation first, and a 15-target AlphaFold-to-experiment structural screen reporting high-confidence-core Cα RMSD values and 27 hypothesis-only discrepancy regions. The central claim, stated in the conclusion, is strictly limited to reproducible software contracts and auditable screening baselines, with broader efficacy and novelty explicitly requiring preregistered evaluation and prospective validation.","tokens_in":9919,"tokens_out":4978,"duration_ms":51853,"significance":"The paper's strength is its disciplined claim boundary and its commitment to inspectable artifacts: exact test counts, a pinned commit hash, cached coordinates with source hashes, machine-readable validation manifests, and explicit abstention labels. If the software-contract claim holds, Plato-Bio is a useful template for verification-first agent evaluation. The temporal benchmark is an honest, well-documented single case, and the structural screen provides a reproducible pipeline for triage. The main technical weakness is the same-core fitting-and-screening circularity in the structural analysis, which weakens the benchmarking value of the RMSD and discrepancy-region outputs unless reframed or reanalyzed. Overall the paper is a solid contribution to evaluation infrastructure, but one load-bearing methodological point needs revision.","major_comments":[{"comment":"The structural screen fits the Kabsch transform to residues with pLDDT ≥ 70, then defines discrepancy regions on residues with pLDDT ≥ 90 and core-aligned Cα error ≥ 2 Å. Since the pLDDT ≥ 90 residues are a subset of the fitting core, the reported core RMSD values (median 0.501 Å; 11/15 < 1 Å) and the 27 discrepancy regions are residuals of a least-squares fit to the very residues whose errors define the flags. The SUMO1 reduction from 16.610 Å to 2.576 Å is therefore partly a mathematical artifact: fitting a subset always lowers the RMSD over that subset. The limitation section (§4.1) mentions that the mask is pLDDT-based rather than an independent structural domain, but it does not disclose this same-core circularity. Please add a split-core or leave-one-out analysis (e.g., fit on pLDDT 70–89 residues and evaluate on pLDDT ≥ 90 residues) or explicitly reframe the outputs as self-fit di","section":"§4.1, Table 2"},{"comment":"The temporal rediscovery pilot is a single curated task whose bridge records, concept annotations, and candidate were selected with knowledge of the 1989 validation result. The paper discloses this in §4.1, but the abstract and §3.3 describe the result as produced by 'independent pre-1986 literature bridges,' which can mislead readers into inferring evidential value that the design cannot support. Because the bridge path and concepts were hand-chosen to connect fish oil to Raynaud's via blood viscosity, the rank-1 outcome is encoded in the fixture. Please rephrase the claim to state explicitly that this is a regression-test-style illustration of the measurement pipeline, not evidence of rediscovery capability, and move the circularity disclosure into the abstract or the results section where the headline number first appears.","section":"§4.1, Table 2 and §3.3"}],"minor_comments":[{"comment":"'F AIR' should be 'FAIR' (spacing artifact).","section":"Abstract and throughout"},{"comment":"Spacing issues: 'T able 1' and 'Figure 2:Passed' should be fixed.","section":"Table 1 and Figure 2 captions"},{"comment":"LaTeX rendering artifacts: 'sub-˚angstr¨om' and similar should be formatted as 'sub-Å'.","section":"§3.4"},{"comment":"The evidence-aware score weights (0.45/0.25/0.20/0.10) and the direct prior-art penalty are free parameters with no sensitivity analysis. Given n=1, this is not fatal, but a sentence noting that the weights are arbitrary would be helpful.","section":"§2.7"},{"comment":"The text says 11/15 targets are below 1 Å and 4 are above 2 Å in the core comparison; this implies none fall between 1 and 2 Å. The authors may wish to confirm this is intended.","section":"§3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and well-scoped, but the structural-screen circularity is a genuine methodological flaw that appears in the headline SUMO1 masking result. I would accept a revision that either reanalyzes with a split-core fit or substantially softens the interpretation of the core RMSD and discrepancy regions. The temporal benchmark's circularity is acknowledged and bounded, so it does not block acceptance if rephrased correctly. The software-engineering contributions and reproducibility artifacts are solid and deserve credit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: it is a fork-specific engineering report that keeps its promises. The three defects it found and fixed — lost task domain, unscored method signals, and a JSONL sidecar that defaulted the unsupported-claim rate to 0.0 when the denominator was missing — are concrete, and the third is a good cautionary example for anyone building evaluation harnesses. The repo ships a commit hash, coordinate files, hashes, validation manifests, and exact test counts. That is real reproducible evidence, and the conclusion carefully refuses to claim efficacy.\n\nThe stress-test note is right about the structural screen. The Kabsch transform is fit to residues with pLDDT ≥ 70, and discrepancy flags are defined on pLDDT ≥ 90, a subset of that core. So the residual errors that define the flags are partly minimized by the same fit; the 27 regions and core RMSD values are not fully independent. That doesn't break the software-contract claim, but it weakens the structural screen as an auditable novelty triage measure. The limitation section is adjacent — it says the mask is pLDDT-based — but it does not name the same-core fitting. A leave-one-out or independent-domain fit would settle it.\n\nThe temporal result is honest but small: one curated task, with concept annotations made knowing the 1989 validation. The paper says so, so it is not a hidden flaw. It is a design demonstration, not a rediscovery result.\n\nThe paper is most useful for people building evaluation pipelines for scientific agents, particularly around claim–evidence links. It is not a biological finding or an AlphaFold benchmark. I would send it to peer review: the reproducible artifacts and the measurement-defect story deserve a serious referee. The referee should ask for a disclosure and stability check on the same-core fitting, and a clear statement that the temporal fixture is illustrative. Worth a short reading-group slot, mostly for the unsupported-claim trap.","headline":"Narrow, honestly scoped engineering report with three real measurement-defect fixes and shipped reproducible artifacts; the structural screen's same-core fitting is a genuine but disclosed-adjacent limitation, and the temporal rank-1 is a designed demonstration, not evidence of efficacy.","tokens_in":10369,"tokens_out":3283,"would_cite":true,"duration_ms":31862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A biology agent's evidence bridges outrank TF-IDF on a frozen rediscovery task, while every structural discrepancy stays an unvalidated hypothesis.","keywords":["scientific agents","literature-based discovery","temporal rediscovery","evidence bridge","AlphaFold","structural validation","reproducibility","verification"],"falsifier":"Run a preregistered version of the temporal benchmark with dozens of historical tasks where concept annotations are produced by annotators blind to the later validation paper and the PubMed corpus is fully frozen; if the evidence-bridge condition does not significantly beat TF-IDF on Recall@1 across tasks, the paper's core ranking claim collapses. Alternatively, re-annotate the existing six records with different synonym choices and check whether the bridge rank-1 persists.","tokens_in":9519,"feed_emoji":"🧬","tokens_out":4091,"duration_ms":39872,"temperature":0.7,"pith_summary":"The paper argues that coherent LLM output does not equal scientific validity, and that an agent's value lies in inspectable contracts: provenance, citation checks, claim-to-evidence links, and explicit abstention. It presents Plato-Bio, a biology-routed agent workflow, and shows that after fixing three measurement defects the full test suite passes. In a single frozen historical task, an A–B/B–C evidence-bridge ranker places the later-validated fish-oil/Raynaud relation first, ahead of TF-IDF and corpus frequency. In a 15-protein structural screen, confidence-masked superposition yields sub-angstrom core agreement for 11 targets and emits 27 discrepancy regions labeled as unvalidated hypotheses. The central claim is narrow: reproducible software contracts and auditable screening baselines, not demonstrated autonomous discovery.","feed_headline":"Evidence bridges put a hidden 1989 finding first","feed_subtitle":"All 27 structural discrepancy regions stay unvalidated hypotheses; the paper claims verifiable baselines, not discovery.","key_machinery":"The load-bearing mechanism is the explicit workflow state: a domain profile routes retrieval and scoring to biology sources, and an evidence-bridge scorer counts independent A–B and B–C pre-cutoff records (penalizing direct A–C prior art) to rank candidate relations. The second mechanism is the confidence-masked Kabsch superposition, which fits the rigid transform on pLDDT≥70 residues before computing residue-level error, so flexible termini and partial constructs do not dominate discrepancy triage.","core_discovery":"On the paper's own terms, the discovery is that verification-first design can be made concrete: a workflow state machine that separates sources, claims, evidence links, and outputs, plus a corrected evaluation harness that no longer hides unsupported claims behind a missing denominator. In the temporal pilot, two pre-1986 records bridge Raynaud phenomenon to blood viscosity to fish oil, and the evidence-aware ranker recovers the 1989 validation paper's relation at rank 1, while TF-IDF and frequency rank it second and third. In the structure screen, fitting predicted models to experimental coordinates on a pLDDT-masked core reduces whole-chain discrepancies, leaving 27 traceable candidate reg","pith_inferences":["A fair test of bridge-based discovery would be a preregistered set of dozens of historical tasks where annotators are blind to the later validation literature; the single curated task here is a mechanism demonstration, not a discovery result.","The rank-1 outcome may hinge on how concepts were annotated; re-running the bridge ranker on independently paraphrased records omitted from the frozen set would show whether the result is robust or an artifact of the chosen wording.","The three measurement defects the paper repairs suggest that other agent evaluation harnesses may harbour similar denominator or domain-routing bugs; the same audit pattern could be applied to those systems.","The 27 discrepancy regions are a candidate queue: a concrete next step is to check each against alternative experimental structures (e.g., ligand-bound or multimer states) and, where none match, design a prospective experiment."],"forward_implications":["If the contract fixes are real, unsupported-claim rates become meaningful: an evaluation cannot report a reassuring zero when claims were never persisted.","The temporal pilot suggests that explicit bridge support can outperform lexical baselines for literature-based discovery ranking, at least when the bridge records are present and correctly annotated.","The structural screen implies that confidence masking plus context stratification is a reproducible way to turn model-versus-experiment RMSD into a short review queue rather than a novelty detector.","Reproducibility artifacts (hashes, fixtures, manifests) allow third parties to re-run the exact analyses and audit the claims."],"fun_headline_variants":["Verification-first screen revives a 1989 link","Evidence-aware ranker recovers 1989 fish-oil clue","Audit fixes three flaws in agent evaluation harness","Evidence bridges beat frequency on 1989 Raynaud link","Structure screen leaves 27 discrepancies as hypotheses"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The temporal rediscovery result rests on six curator-written paraphrases of pre-1986 PubMed records, annotated with concepts and bridge choices made with knowledge of the 1989 validation; with different records or annotations, the fish-oil/Raynaud bridge may no longer rank first.","fun_headline_variants_meta":{"raw":{"variants":["Verification-first screen revives a 1989 link","Evidence-aware ranker recovers 1989 fish-oil clue","Audit fixes three flaws in agent evaluation harness","Evidence bridges beat frequency on 1989 Raynaud link","Structure screen leaves 27 discrepancies as hypotheses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001128,"raw_usage":{"total_tokens":4579,"prompt_tokens":849,"completion_tokens":3730,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":3652}},"tokens_in":593,"tokens_out":3730,"duration_ms":25551,"temperature":1.0,"reasoning_tokens":3652,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:21:43.715959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a preregistered version of the temporal benchmark with dozens of historical tasks where concept annotations are produced by annotators blind to the later validation paper and the PubMed corpus is fully frozen; if the evidence-bridge condition does not significantly beat TF-IDF on Recall@1 across tasks, the paper's core ranking claim collapses. Alternatively, re-annotate the existing six records with different synonym choices and check whether the bridge rank-1 persists.","supporting_citations":[],"review_version":1}