{"id":"feaa36d2-d8ff-41af-841b-9f3d428c7c49","arxiv_id":"2607.28990","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SciDisco trains scientific-analysis agents with turn-level RL rewards derived from hidden evidence-DAG verifiers, achieving SOTA on DiscoveryBench but not on DABStep.","lead":"This paper builds SciDisco, a training pipeline that turns open scientific datasets into interactive environments with hidden evidence graphs, verifies multi-step analyses, and uses turn-level RL rewards to train a 14B model for scientific hypothesis testing. The trained model beats prior open and closed models on the DiscoveryBench hypothesis-generation benchmark, though it lags on business-data benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is not protected against training/eval distribution overlap: the contamination audit is hash-only and explicitly excludes topical similarity, so SciDisco-14B's benchmark gains may reflect task-format familiarity rather than general scientific-discovery skill.","rationale":"The reader's CONDITIONAL verdict is appropriate. I considered two other candidate concerns: (i) the 0.5-point margin has no error bars; (ii) no artifacts are released. Both are real but secondary. The margin issue would only be decisive if we had reason to expect noise; the overlap issue is more fundamental because if eval tasks are inside the training distribution, the result is not a general capability even with perfect replication. The paper deserves credit for its honest Limitations, detailed visibility/leakage controls, verifier traces, and domain-expert review; these strengthen the internal validity of SciThèque as a training substrate. But internal validity is not external validity. The hash audit answers a narrower question than the one the SOTA claim needs. The concrete semantic-overlap test above would settle it. Until then, CONDITIONAL remains the right verdict rather than ACCEPT or REJECT.","tokens_in":19542,"tokens_out":6866,"duration_ms":74607,"concrete_test":"Run a semantic overlap audit: with a frozen sentence encoder (e.g., gtr-t5-xl or an embedding API), embed all 1,686 SciThèque task prompts and all DiscoveryBench/DataSciBench task prompts; compute pairwise cosine similarity and report the distribution (e.g., percentage of eval prompts with a training prompt above 0.7/0.8). Then retrain SciDisco-14B after excluding the training tasks in the top overlap quantile (or hold out entire domains absent from eval) and re-evaluate. If the DiscoveryBench margin over Intern-S1-Pro shrinks to within noise or DataSciBench gains disappear, the SOTA claim is confounded by distribution overlap; if the gains persist on held-out/least-overlapping tasks, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — state-of-the-art on hypothesis-driven scientific data analysis — rests on SciDisco-14B's DiscoveryBench HMS (35.2 vs 34.7 for Intern-S1-Pro) and DataSciBench gains. But SciThèque tasks are built from the same public data ecosystem (UCI, OpenML, FRED, CDC, USGS, Yelp) and the same hypothesis-template/statistical-pattern space that DiscoveryBench and DataSciBench also sample. Appendix A's benchmark-overlap audit compares only SHA-256 hashes of materialized data files and normalized source identifiers; it does not compare task prompts, hypothesis templates, variable/role assignments, or statistical patterns. The appendix explicitly states: 'The audit is scoped to exact file reuse and source-ID reuse, while broad topical similarity is expected across independently sourced scientific datasets.' Thus the audit rules out exact-file replication, not distributional overlap. If the 1,686 SciThèque task prompts are semantically close to eval prompts (same domains, same 'Does X predict Y after controlling for Z' phrasing), then SFT/RL optimizes the policy on the eval distribution. The 0.5-point SOTA margin and the DataSciBench gains could then reflect template familiarity rather than improved reasoning. This is a correctness risk for the empirical headline, not an internal inconsistency; the paper's own Limitations concede that environment-accepted progress is a proxy for scientific quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SciDisco, a three-stage post-training pipeline for data-driven scientific discovery agents. It introduces SciThèque, which compiles open scientific datasets and hypothesis templates into 1,686 sandboxed task environments, each with a hidden evidence DAG that defines prerequisite-ordered scientific analysis steps and a verifier that accepts a turn only when it produces evidence for exactly one frontier node. DAG-grounded trajectory synthesis generates 5,620 verifier-filtered multi-turn demonstrations used for SFT. DiscoPO extends GRPO with turn-level rewards computed from increases in the DAG-progress potential, assigning token-level advantages at each interaction turn. The paper reports experiments on DiscoveryBench, DABStep, and DataSciBench, claiming state-of-the-art on DiscoveryBench for SciDisco-14B, and an ablation showing that SFT+DiscoPO outperforms SFT+GRPO on DataSciBench.","tokens_in":20034,"tokens_out":5522,"duration_ms":51750,"significance":"If the empirical claims withstand scrutiny, the framework is a solid contribution: it provides a reusable, executable, and verifiable training substrate for scientific discovery agents, with a clear mechanism for process supervision and turn-level credit assignment. The ablation in Table 2 is clean and supports the central algorithmic claim. Strengths include a detailed environment contract, leakage controls, an explicit benchmark-overlap audit, and a verifier-grounded trajectory synthesis pipeline. However, the headline SOTA claim rests on a 0.5-point margin on DiscoveryBench, a single-run evaluation with unreported judge variance, and an overlap audit that explicitly excludes semantic/template overlap. These methodological weaknesses make the empirical conclusions provisional.","major_comments":[{"comment":"The abstract's 'state-of-the-art' claim rests on a 0.5-point margin over Intern-S1-Pro on DiscoveryBench (35.2 vs 34.7, Table 1). No error bars, multiple seeds, or judge-reliability statistics are reported, and the HMS judge is GPT-5 Mini with no human-agreement measure. Appendix A's overlap audit compares only SHA-256 hashes and normalized source identifiers, and explicitly states it is scoped to exact file reuse while 'broad topical similarity is expected.' Since SciThèque is built from the same public repositories (UCI, OpenML, FRED, CDC, etc.) and the same hypothesis-template space that DiscoveryBench and DataSciBench sample, the reported gains may reflect distributional familiarity rather than improved scientific reasoning. Please either add a semantic/template-level overlap analysis or a held-out source/template evaluation, and temper the SOTA claim accordingly.","section":"§5.1, Table 1, Appendix A"},{"comment":"The training reward is defined by the authors' hidden DAGs and verifier acceptance criteria, and the Limitations section concedes that 'environment-accepted progress is still a proxy for scientific quality.' The 10-expert review in Appendix B inspects verifier specifications, not rollout-level verifier decisions. No empirical evidence is given that accepted steps correlate with human-judged scientific quality or final-answer correctness. This is a correctness risk for the RL signal: the policy could optimize verifier compliance rather than scientific progress. Please report verifier precision/recall on a human-labeled rollout sample and analyze whether accepted intermediate turns predict final score on held-out benchmarks.","section":"§4.1, Appendix B, Limitations"},{"comment":"The DABStep average of 17.8% is far below DeepAnalyze-8B (38.9%) and several proprietary models. The paper attributes this to a training-distribution boundary, which is plausible but not tested. Given the abstract's broad 'scientific data analysis benchmarks' phrase, the authors should either restrict the headline claim to DiscoveryBench/DataSciBench or provide an analysis (e.g., per-task-type breakdown) showing the cause is domain shift rather than a general inability to do multi-step analysis.","section":"§5.4, Table 1"}],"minor_comments":[{"comment":"The pipeline diagram is dense, and the labels for DiscoPO's advantage calculation are difficult to read. Consider enlarging or splitting into separate figures.","section":"Figure 1"},{"comment":"The term 'materialization pool' is used before it is defined; define it at first use.","section":"§4.1"},{"comment":"The 'Stress variants' are counted separately from aggregate totals, but no further detail is given. A sentence describing what perturbations are applied would help reproducibility.","section":"Appendix A, Stress variants"},{"comment":"The VLM column is left empty with a note. Consider removing the column or adding 'n/a' to avoid confusion.","section":"Table 2"},{"comment":"Several 2026 preprints central to the baseline comparisons (e.g., Intern-S1-Pro Team, DeepSeek-V4-Flash) should have stable identifiers or URLs to allow verification. The 'slime' software is cited as a repository without a version.","section":"References"},{"comment":"The conclusion contains 'SciTh‘eque' with a stray quote; spelling of 'materialised/materialized' is inconsistent between appendix and main text.","section":"Typographical"}],"recommendation":"major_revision","confidential_remarks":"The framework contribution is solid, but the empirical headline is under-supported. The authors should be asked either to substantially strengthen the overlap audit and evaluation reliability or to reframe the abstract and SOTA claim. The reference list contains several 2026 preprints that appear central to the comparisons; their availability should be verified. If the venue values reusable environments for training agentic scientists, the paper is a strong candidate after the empirical issues are addressed; if it emphasizes demonstrated scientific discovery, the proxy concern is more serious."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper offers a real, reusable pipeline for training scientific-discovery agents, and the ablation is the cleanest part. SciThèque turns open datasets into hidden-DAG verifier environments; DAG-grounded synthesis gives you cold-start SFT; DiscoPO assigns turn-level credit from verifier-gated progress. None of the components is brand new, but the composition is, and the DataSciBench ablation (SFT 46.8 vs +GRPO 49.5 vs +DiscoPO 56.3 success rate) is a decent argument that turn-level process credit does more than trajectory-level GRPO. The authors also say plainly in the Limitations that environment-accepted progress is a proxy, not novel science. That honesty counts.\n\nThe soft spots are real but fixable. The headline SOTA on DiscoveryBench is 35.2 vs Intern-S1-Pro's 34.7 — a hair-thin margin, and the paper gives no error bars or judge-variance reporting. The contamination audit in Appendix A only checks exact file hashes and source IDs; it explicitly says broad topical similarity is expected. That means the benchmark gains could partly reflect template familiarity rather than general scientific reasoning. That is a correctness risk for the empirical claim, though not a demonstration that the pipeline is broken. The DABStep failure (17.8, below DeepAnalyze's 38.9) shows the training distribution doesn't transfer to business analytics, which the paper acknowledges. No artifacts are released, which also limits independent checking.\n\nThe stress-test note's concern is legitimate, but I'd stop short of calling it load-bearing for the whole paper. The central mechanism — hidden DAGs give process supervision without manual step labels — is sound and the ablation supports it, even if the benchmark numbers are shakier than the abstract suggests. The fix is straightforward: a smarter overlap audit comparing task prompts and template patterns, plus error bars and artifact release.\n\nWho gets value: people working on agentic RL, process reward models, and scientific-discovery agents. It deserves a serious referee — the idea is important enough, and the execution is coherent enough, that a referee can usefully push on the empirical claims without the whole thing collapsing. My recommendation: send it to peer review with a request for a stricter overlap audit, uncertainty quantification, and open artifacts. If those get added, this could be a solid contribution; as it stands, the empirical headline oversells the evidence.","headline":"A genuinely new pipeline for turn-level RL in scientific discovery, with a clean ablation and honest limitations, but the SOTA headline rests on a 0.5-point margin and a leaky overlap audit.","tokens_in":20455,"tokens_out":1432,"would_cite":true,"duration_ms":17959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 14B open-weight model can reach leading results on hypothesis-driven scientific data analysis if it is trained inside process-verifiable environments that reward each turn producing verifiable analytical evidence, ra","keywords":["scientific discovery agents","process-verifiable environments","hidden evidence DAGs","turn-level reinforcement learning","verifier-grounded credit assignment","hypothesis-driven data analysis","agentic RL for data science","synthetic trajectory filtering"],"falsifier":"Take SciDisco-14B and run it on a held-out evaluation set built by independent domain experts who define hypotheses, evidence prerequisites, and verifiers without using the SciThèque template library or reference analyses. If the model's accuracy on that set is no better than the SFT-only checkpoint, then the DiscoPO turn-level signal—rather than verifier content or corpus overlap—is not what produces the reported gains.","tokens_in":19481,"feed_emoji":"🔬","tokens_out":6953,"duration_ms":73177,"temperature":0.7,"pith_summary":"The paper claims that the missing ingredient for reliable data-driven scientific-discovery agents is process supervision over real scientific data, not larger models or better prompts. It introduces SciDisco, a three-stage pipeline: SciThèque compiles 1,686 open scientific datasets into sandboxed tasks where a hidden evidence DAG lets a verifier check intermediate progress; DAG-grounded trajectory synthesis produces 5,620 verifier-filtered multi-turn demonstrations for cold-start SFT; and DiscoPO assigns turn-level rewards equal to the increase in verified DAG progress, then applies group-relative policy optimization at the turn level. The reported result is that SciDisco-14B reaches 35.2% average HMS on DiscoveryBench, ahead of stronger proprietary and open-source baselines, and outperforms trajectory-level GRPO on DataSciBench. The authors present this as improved performance on executable hypothesis-driven analysis, and explicitly note that environment-accepted progress is a proxy for scientific quality rather than proof of novel discovery.","feed_headline":"Turn-level rewards land a 14B agent atop discovery benchmarks","feed_subtitle":"Hidden evidence DAGs reward each verified step, helping a 14B open model beat larger baselines.","key_machinery":"The hidden evidence DAG gj=(Vj,Aj) is the central object: each node is a verifiable scientific state transition typed by a process primitive (data inspection, model fitting, diagnostic checking, robustness analysis), and directed edges encode prerequisites so only frontier nodes are eligible at each turn. SciThèque uses the DAG to define tasks and verifiers; trajectory synthesis uses it to schedule and filter demonstrations; DiscoPO uses it to compute progress potential Φj(C)=|C∩Pj|/|Pj| and turn rewards as potential differences. The same graph thus carries task construction, imitation cold-start, and RL credit assignment.","core_discovery":"The central claim is that scientific discovery can be trained as an interactive process if the environment can verify each analytical step, not just the final answer. To that end, SciThèque materializes 1,686 tasks from open datasets, each with a hidden evidence DAG whose nodes are typed process primitives—data setup, EDA, feature construction, model fitting, diagnostics, uncertainty, robustness, and submission—and whose edges enforce a prerequisite order. A verifier accepts a turn only when it produces new evidence for exactly one eligible frontier node, refusing bundled, repeated, premature, or unverifiable actions. DAG-grounded trajectory synthesis uses the DAG to schedule multi-turn demo","pith_inferences":["Editorial inference: the same hidden-DAG credit mechanism could generalize to other long-horizon agentic domains—code repair, lab automation, literature search—wherever a verifier can define typed, prerequisite-ordered evidence nodes; the authors do not claim this extension.","The paper's own limitation section concedes that environment-accepted progress is a proxy, and its overlap audit checks exact file reuse only. A content-level audit of topical similarity between training sources and DiscoveryBench/DataSciBench would be the natural next test of whether the benchmark gains reflect general reasoning or corpus overlap.","Because hypotheses are generated from template libraries, the discovery space is bounded by the templates; replacing templates with open-ended hypothesis generation would test whether the pipeline can move beyond template-shaped discoveries toward genuinely novel claims."],"forward_implications":["If the reported numbers hold, a 14B open-weight model can outperform much larger closed models on hypothesis formation from scientific datasets, implying process-verifiable environments can substitute for model scale in this regime.","Turn-level verifier credit scales directly: the same environments that emit SFT demonstrations also supply RL rewards, so expanding the environment corpus expands the training signal without human trajectory annotation.","The ablation ordering SFT < SFT+GRPO < SFT+DiscoPO on DataSciBench indicates that fine-grained process reward, not just outcome verification, drives the largest gains, especially in data-exploration and data-modeling steps.","Because the training distribution is hypothesis-driven scientific analysis, the weaker DABStep result shows the gains will not automatically transfer to business-analytics workflows; transfer is selective.","The hidden-DAG design makes training auditable: leakage scans and verifier traces can be checked, and failure modes such as premature submission or bundled analyses are explicitly excluded from both SFT and RL signal."],"fun_headline_variants":["Verifiable steps make a 14B agent top discovery benchmarks","Process verifiers turn a 14B model into a discovery SOTA","Verifiable steps let a 14B agent beat bigger agents on science","SciDisco's turn-level verifiers push a 14B agent to SOTA","Evidence DAGs reward each step, training a 14B agent to SOTA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the hidden evidence DAGs and verifier acceptance criteria—built from the authors' template libraries and reference analyses—faithfully measure scientific progress; if they only encode the authors' task-specific heuristics, the training signal and benchmark gains may reflect conformance to those heuristics (or topical overlap with the benchmark data) rather than improved scientific reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Verifiable steps make a 14B agent top discovery benchmarks","Process verifiers turn a 14B model into a discovery SOTA","Verifiable steps let a 14B agent beat bigger agents on science","SciDisco's turn-level verifiers push a 14B agent to SOTA","Evidence DAGs reward each step, training a 14B agent to SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001493,"raw_usage":{"total_tokens":5789,"prompt_tokens":665,"completion_tokens":5124,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":5024}},"tokens_in":409,"tokens_out":5124,"duration_ms":32475,"temperature":1.0,"reasoning_tokens":5024,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T15:50:56.686359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take SciDisco-14B and run it on a held-out evaluation set built by independent domain experts who define hypotheses, evidence prerequisites, and verifiers without using the SciThèque template library or reference analyses. If the model's accuracy on that set is no better than the SFT-only checkpoint, then the DiscoPO turn-level signal—rather than verifier content or corpus overlap—is not what produces the reported gains.","supporting_citations":[],"review_version":1}