{"id":"f27795ee-4bea-49d8-8b62-7f4540fa848c","arxiv_id":"2607.05762","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Quantity-grounded multi-agent decomposition makes LLM-generated collider analysis code inspectable and reliable with 14B-scale models, outperforming prior single-prompt approaches.","lead":"A multi-agent system splits LLM scientific code generation into helper selection, coding, execution repair, quantity tracing, and critique so researchers can see exactly which physics quantities and assumptions the code implements. It also flags ambiguous instructions before generation and runs the full pipeline on models as small as 14B parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Reliability vs single-prompt is under-measured: Table 3 shows 14B helper selection still implements the wrong composite Δφ, with no multi-case semantic pass-rate comparison to CoLLM.","rationale":"The reader correctly flags CONDITIONAL and points at tracer completeness as a soft spot for critique. That matters for inspection quality, but the load-bearing gap for the stated strongest claim is broader: reliability is asserted relative to single-prompt CoLLM without a controlled semantic correctness comparison, while Table 3 already shows that enabling the 14B workflow can still compute the wrong composite quantity when helper selection is weak. Transparency via intermediate artifacts is well motivated by design; the reliability and “usable at 14B” halves are only partially evidenced (executability + limited manual review on five LHCO cases). That reinforces CONDITIONAL rather than moving the verdict. No internal inconsistency or derivation error; the architecture and GitHub release remain a real systems contribution. A quantitative multi-case pass-rate table would settle whether decomposition improves analysis fidelity or mainly pipeline completion.","tokens_in":17320,"tokens_out":639,"duration_ms":32425,"concrete_test":"On all five Table 1 task cards, run single-prompt CoLLM-style generation and the full multi-agent pipeline with Qwen3-14B and a 32B model (n≥5 seeds each). Score every main selection cut and every composite/validation observable pass/fail with the Table 3 protocol (or reference scripts). If multi-agent does not clearly raise the composite-observable pass rate at 14B over single-prompt, the reliability half of the claim does not hold as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim couples two results: (1) modular decomposition enhances reliability relative to single-prompt CoLLM, and (2) 14B–32B models can run the full workflow that previously needed ~70B. Section 4.2 supports (2) mainly as executability—helper selection makes the generation–fixer loop produce runnable code at 14B where the prior single-agent setup “rarely” did. That is not the same as reliable physics quantities. Table 3 (Case 5) is the only detailed semantic scorecard: with Qwen-14B helper selection, both 14B and 70B generators fail composite Δφ(ττ,jj), using leading τ vs leading jet instead of ditau vs dijet systems—the exact quantity-level error the framework is meant to expose. Llama-70B helpers pass the same composite. There is no head-to-head, multi-case tally of selection-cut / composite / cutflow pass rates for multi-agent vs single-prompt at matched model sizes. Tracer+critique (the transparency path) are only qualitatively shown; §4.5 explicitly skips a statistical study of tracing. So “reliability” currently rests on pipeline completion plus one-case manual scoring, not on demonstrated gains in analysis fidelity.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper presents a multi-agent framework for LLM-generated scientific analysis code that decomposes the pipeline into helper selection, code generation, execution-based repair, quantity-grounded tracing, post-generation critique, and a pre-generation ambiguity module (Golden Axe Oracle). Domain-specific assets (generation profile, helper registry, critique/oracle contexts) are isolated behind a fixed interface so the orchestration can be reused across domains. The system is instantiated for LHCO collider analyses and evaluated on five structured task cards. The authors report that modular decomposition improves transparency and reliability relative to their prior single-prompt CoLLM approach and that Qwen-family models at the 14B–32B scale can complete a workflow that previously required ~70B-scale models.","tokens_in":17705,"tokens_out":1390,"duration_ms":21109,"significance":"If the claims hold, the work is a useful systems contribution for scientific software engineering: it treats LLM code generation as an auditable multi-stage process rather than a single opaque inference step, and it shows a practical path to running such pipelines with smaller, locally hostable models. Strengths include the explicit provenance design (run directories with intermediate artifacts), line-referenced tracer outputs, a domain-asset interface that separates orchestration from HEP-specific resources, and a public GitHub package. The Oracle’s pre-generation ambiguity protocol and the emphasis on composite-object construction in task cards are concrete, transferable practices. The significance is currently limited by the narrow empirical base (one domain, five cases) and by incomplete measurement of semantic reliability versus the single-prompt baseline.","major_comments":[{"comment":"The central claim that modular decomposition “enhances … reliability relative to the previous single prompt approach” is not supported by a head-to-head, multi-case semantic evaluation. Section 4.2 and the surrounding text mainly establish that helper selection makes the generation–fixer loop produce runnable code at 14B where CoLLM “rarely” did—an executability result, not a fidelity result. Table 3 provides the only detailed semantic scorecard and covers only Case 5; with Qwen-14B helper selection, both 14B and 70B generators fail the composite Δφ(ττ,jj) quantity that the framework is designed to protect. There is no multi-case tally of selection-cut / composite / cutflow pass rates for multi-agent vs single-prompt at matched model sizes. Either add that comparison or narrow the abstract/conclusion claim to transparency plus executability at smaller scale.","section":"Abstract; §4.2–4.3; Table 3"},{"comment":"The load-bearing assumption that the tracer’s as-implemented specification is complete and accurate enough for critique to surface all important semantic mismatches is asserted but not measured. Section 2.1 states that the tracer prompt was “extensively validated” on LHCO examples for O(10)B models, yet §4.5 explicitly declines a statistical study of tracing and offers only qualitative excerpts (Case 3 and Case 5). Without a controlled check—e.g., injected definition mismatches / composite violations and measured recall of tracer+critique across the five cases—the claim that quantity-grounded semantic differencing reliably exposes implementation assumptions remains under-supported. A modest injection or multi-run agreement study would address this.","section":"§2.1 (Implementation tracing); §4.5"},{"comment":"Helper-selection stability (Table 2) is reported carefully, but the paper itself notes that stability is not correctness and that multiple helper sets may be compatible with the same task card. Table 3 then shows that the choice of helper set (Qwen-14B vs Llama-70B) changes whether composite observables are implemented correctly. The manuscript needs a clearer statement of how a “correct” or “acceptable” helper set is defined for evaluation, and whether downstream semantic pass rates should be conditioned on a fixed gold helper list versus the selector’s own output. Without that, the reliability of the full pipeline under realistic selector noise is hard to interpret.","section":"§4.2 Table 2; §4.3 Table 3"}],"minor_comments":[{"comment":"Table 3 footnote states code generation used Qwen3-14B because Qwen2.5-14B API access became unavailable, while helper-selection tables use Qwen2.5-14B. Please state model identities consistently in all tables and note any cross-version caveats in one place.","section":"§4.3 Table 3"},{"comment":"Figure 1 is helpful; a short caption note on which stages are domain-dependent vs domain-independent would help readers planning ports to other fields.","section":"Figure 1; §2.1"},{"comment":"Oracle evaluation (Table 4) uses manual inspection “using ChatGPT” to judge valid ambiguities. Briefly describe the adjudication criteria and whether a second human rater was involved, so the useful/off-target counts are reproducible.","section":"§4.4 Table 4"},{"comment":"Several compound words appear without spaces in the PDF source (e.g., “accompaniedbyanaccountofwhatwascomputed”, “missingtransverse”). Clean typesetting before camera-ready.","section":"§1 Introduction"},{"comment":"The package is said to be on GitHub, but the manuscript text only has the placeholder “Package: GitHub”. Provide a stable URL or DOI for the artifact.","section":"Title page; §5"},{"comment":"Clarify default values of free parameters (retry limit N, regeneration temperature 0.3, oracle max three ambiguities) in one configuration table so experiments can be reproduced without reading the code.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems/SE contribution with a clear HEP use case, but the abstract overclaims “reliability” relative to CoLLM. If the authors reframe around transparency, provenance, and smaller-model executability—or add the missing multi-case semantic comparison—the work is appropriate for a methods-oriented venue. Scope fit depends on whether the journal wants agentic scientific-software systems papers; the novelty is primarily architectural and empirical rather than theoretical."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a practical systems paper that decomposes LLM scientific code generation into helper selection, generation, repair, quantity-grounded tracing, critique, and a pre-generation ambiguity oracle. The real win is inspectability plus the fact that the pipeline can complete with ~14B models where their prior CoLLM single-prompt setup needed ~70B.\n\nWhat is new and solid: the helper registry + selector, the tracer that rebuilds as-implemented specs with upstream chains and line refs, and the Golden Axe Oracle that forces ambiguity choices before code is written. Domain assets are cleanly separated from orchestration, so the design is portable in principle. Task cards that force composite objects (diphoton, ditau, dijet) as reusable four-vectors clearly stabilize generation. They ship GitHub code, report helper-selection stability, ambiguity recall by model scale, and give concrete tracer/critique excerpts. That is honest engineering, not vapor.\n\nSoft spots, in proportion: the abstract’s “enhances reliability relative to single-prompt CoLLM” is under-measured. Section 4.2 mainly shows executability at 14B once helpers are selected. Table 3 (Case 5) is the only detailed semantic scorecard: with Qwen-14B helpers, both 14B and 70B generators still get composite Δφ(ττ,jj) wrong—leading τ vs leading jet instead of ditau vs dijet—the exact quantity error the framework is meant to catch. Llama-70B helpers pass. There is no multi-case head-to-head pass-rate table for selection cuts, composites, and cutflow vs single-prompt at matched sizes. Tracer/critique are qualitative; they skip a statistical study of tracing. So transparency is demonstrated; fidelity gains are only partly shown. Evaluation stays on five LHCO-style cards and partly manual scoring. Free knobs (retry N, regen temperature, oracle max ambiguities) are minor.\n\nWho it is for: people building agentic analysis pipelines in HEP or other domains that need auditable LLM code, not people looking for a new physics result. Math and citations look fine; circularity is low. I would send it to peer review. Engage if you care about inspectable scientific codegen; treat the reliability claim as provisional until they add multi-case semantic tallies against the baseline.","headline":"Useful multi-agent workbench that makes LLM HEP analysis code inspectable and runnable at 14B; the reliability-vs-CoLLM claim is thinner than the abstract suggests.","tokens_in":18237,"tokens_out":582,"would_cite":true,"duration_ms":11029,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Decomposing LLM scientific analysis into specialized agents makes what the code computes inspectable and lets much smaller models finish the job.","keywords":["LLM code generation","scientific computing","multi-agent systems","semantic differencing","reproducibility","collider physics","task decomposition","ambiguity detection"],"falsifier":"Run the five collider benchmarks repeatedly under both the modular pipeline and a single-prompt baseline at matched model sizes; if expert review finds the modular tracer-plus-critique misses a comparable fraction of serious semantic errors, or if 14B-scale models still fail to produce usable end-to-end implementations, the central claim does not hold.","tokens_in":18234,"feed_emoji":"🔬","tokens_out":751,"duration_ms":27154,"temperature":0.7,"pith_summary":"Language-model code for scientific analyses can run successfully yet still hide which quantities were computed and which assumptions shaped the numbers. This paper introduces a multi-agent workflow that splits the problem into helper selection, code generation, execution-and-repair, quantity-grounded tracing of how outputs are built, and critique against the original request. A pre-generation module also surfaces ambiguities in the user instruction and offers alternative rewrites. On representative collider-physics analyses the modular design is more transparent and reliable than a single-prompt baseline, and it lets models at the 14B–32B scale complete a workflow that previously required models near 70B parameters.","feed_headline":"Smaller AIs write transparent physics code when the job is split","feed_subtitle":"Specialized agents reconstruct what quantities the code computes and catch hidden assumptions before results are trusted.","key_machinery":"Quantity-grounded semantic differencing: a multi-agent pipeline whose tracer follows each requested quantity through variable definitions and dependency chains back to object collections and helper calls, then feeds the reconstructed “as-implemented” specification to a critique agent that compares it with the original task card.","core_discovery":"The paper establishes that quantity-grounded semantic differencing—assigning helper selection, generation, execution repair, implementation tracing, and critique to separate agents—reconstructs how key scientific outputs are produced and surfaces mismatches between the intended analysis and the implemented code. Validated on collider analyses, this task decomposition improves transparency and reliability relative to the prior single-prompt approach while enabling substantially smaller models to execute the complete pipeline.","pith_inferences":["The same tracing-and-critique layer could audit any LLM-generated scientific pipeline where silent redefinitions of objects or cuts matter.","Forcing reusable composite objects in the task card may matter as much as model scale for semantic fidelity.","If tracer completeness is tuned mainly to the validated domain patterns, new analysis styles will need fresh prompt hardening before critique can be trusted.","Groups without large-model API access could draft production analyses locally with 14B–32B models under this architecture."],"forward_implications":["Selected helpers, tracer chains, and critique reports form an auditable record of the assumptions behind each numerical result.","Domain packages can be swapped so the same orchestration applies outside collider physics without rewriting the pipeline.","Explicit composite-object construction in the task card reduces implementation variability across repeated generations.","Ambiguity detection before code generation moves interpretation choices from the model to the researcher.","Models at the 14B scale become usable for full generation-plus-review when the task is decomposed this way."],"fun_headline_variants":["Task split lets smaller models generate transparent collider code","Multi-agent breakdown surfaces hidden assumptions in AI science code","Quantity-grounded agents reconstruct outputs from LLM physics analyses","Decomposed workflow improves reliability over single-prompt AI coding","Modular agents enable smaller models for full scientific analysis pipelines"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes the tracer fully reconstructs every load-bearing definition and dependency chain so the critique can catch semantic mismatches, even when the code is long and the models are only around 14 billion parameters.","fun_headline_variants_meta":{"raw":{"variants":["Task split lets smaller models generate transparent collider code","Multi-agent breakdown surfaces hidden assumptions in AI science code","Quantity-grounded agents reconstruct outputs from LLM physics analyses","Decomposed workflow improves reliability over single-prompt AI coding","Modular agents enable smaller models for full scientific analysis pipelines"]},"model":"grok-4.5","effort":"low","cost_usd":0.004764,"raw_usage":{"total_tokens":1324,"prompt_tokens":747,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":47640000,"prompt_tokens_details":{"text_tokens":747,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":516,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":747,"tokens_out":61,"duration_ms":5096,"temperature":1.0,"reasoning_tokens":516,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T02:19:37.940667+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the five collider benchmarks repeatedly under both the modular pipeline and a single-prompt baseline at matched model sizes; if expert review finds the modular tracer-plus-critique misses a comparable fraction of serious semantic errors, or if 14B-scale models still fail to produce usable end-to-end implementations, the central claim does not hold.","supporting_citations":[],"review_version":1}