{"id":"970c72cb-15ee-4d69-bb5b-a3f42845f83e","arxiv_id":"2508.02721","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A source-code-driven 'blueprint first' agent framework reports strong τ-bench results, but the paper contains an abstract/body mismatch and releases no artifacts.","lead":"This paper proposes a code-first architecture for LLM agents: a human-written script, called an Execution Blueprint, controls every step of the workflow, and the large language model is called only for narrow sub-tasks such as parsing or summarizing. The authors report large gains on the τ-bench agent benchmark, but the submitted abstract describes different experiments and no code or data is included.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on undisclosed, hand-written blueprints with benchmark-specific components; without a held-out authoring procedure, the 10.1-point gain may measure encoded rules rather than the framework.","rationale":"The paper's contribution is an empirical claim, so the key question is whether the reported 10.1-point τ-bench advantage is a property of the architecture. The ablation data make the alternative reading vivid: adding the bare Source Code Agent to the Function Call baseline moves Pass^1 from 48.2 to 49.3, while adding the hand-coded Double-Check module contributes +11.7, and Retail Tools contribute the rest. Those components are not produced by the framework; they are described as coded workflow steps and domain-specific tools. The Conclusion's admission that implementation relies on manual blueprint creation is therefore not a peripheral limitation but the crux: if an expert hand-encodes the policy rules, the resulting system is a bespoke τ-bench solution, and LLM invocation at selected nodes does not need to be a general framework to achieve the number. The absence of released code/blueprints and the abstract-level mismatch between TravelPlanner and τ-bench prevent an independent check. I credit the design idea: decoupling workflow from model is coherent and the implementation details (sandbox, session control, resource governance) are plausible engineering. But those details cannot rescue the central empirical claim. The reader's weakest assumption already identifies the manual-encoding risk; I agree, and I would move the verdict to UNVERDICTED rather than CONDITIONAL because no amount of internal consistency checking can settle whether the framework generalizes until the blueprints and a held-out evaluation are available.","tokens_in":11344,"tokens_out":9449,"duration_ms":99634,"concrete_test":"Ask the authors to release the exact Execution Blueprints and to evaluate the framework on a third, held-out procedural domain (or held-out τ-bench subset) with blueprints frozen before seeing the evaluation set, comparing against the same Function Call/ReAct baselines on the same backbone. The decisive outcome is whether the Pass^1 margin over Function Call is preserved; if the held-out margin is materially smaller than the reported 10.1 points, the SOTA claim is attributable to manual encoding of τ-bench rules rather than to the architectural separation.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The load-bearing condition is that the τ-bench gains come from the Blueprint First, Model Second architecture rather than from hand-written, benchmark-specific logic. The paper's own evidence points the other way: Ablation Table 3 attributes +11.7 points to the hand-coded Double-Check module and a further gain to specialized Retail Tools, and the Conclusion admits that the current implementation relies on manual blueprint creation. The paper does not describe how blueprints were authored, whether they were iterated on τ-bench, or how Retail Tools were designed, and no code or blueprints are released. A hand-coded workflow with LLM calls inserted at selected points would beat generic ReAct/FC baselines even if it contributed nothing reusable. This is compounded by an internal inconsistency: the separately presented abstract reports TravelPlanner 35.56% while the body reports τ-bench average Pass^1 67.7%; the community cannot tell which evaluation supports the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Source Code Agent framework, built on a \"Blueprint First, Model Second\" philosophy in which an expert-defined operational procedure is codified as a source-code Execution Blueprint and executed by a deterministic engine, with LLM invocations confined to bounded sub-tasks. The manuscript claims a new state-of-the-art average Pass^1 of 67.7% on τ-bench with Claude-3.5-Sonnet, a 10.1-point gain over the strongest baseline, as well as large efficiency improvements and an ablation study attributing the gains to the Double-Check module and specialized Retail Tools. A separately presented abstract additionally claims a 35.56% final pass rate on TravelPlanner, production deployments, and results on ScienceWorld and ALFWorld, but these evaluations are not reported in the body.","tokens_in":11493,"tokens_out":4374,"duration_ms":45534,"significance":"The proposed architectural pattern is coherent and practically relevant: encoding workflow control flow in deterministic code while using the LLM only for bounded sub-tasks is a plausible way to improve procedural fidelity. If the empirical claims were fully supported, the paper would make a useful contribution to the deployment of LLM agents in structured operational environments. However, the current evidence is not yet sufficient to support the headline state-of-the-art claim. The paper does not release code, blueprints, or tools; it does not describe the blueprint authoring process; and the submitted abstract contradicts the body on which benchmark supports the central claim. These gaps are load-bearing because the reported gains could plausibly come from hand-encoding benchmark-specific rules rather than from the framework itself. The architecture description and the efficiency analysis are strengths, but the empirical evaluation needs substantial reworking before the central claim can be assessed.","major_comments":[{"comment":"The central SOTA claim is not currently supported because the reported gains are attributed to manually authored, benchmark-specific components, while no artifacts or authoring protocol are provided. Table 3 attributes +11.7 points to the hand-coded Double-Check module and a further gain to the specialized Retail Tools, and the Conclusion states that the current implementation relies on manual blueprint creation. The paper does not report how the blueprints were authored, whether they were iterated against τ-bench, or whether any held-out domain was used. Without releasing the blueprints and tools and describing the authoring process, the 10.1-point improvement over baselines may measure hand-encoded rules rather than a property of the framework.","section":"Experiment Result / Ablation Study (Table 3)"},{"comment":"There is an internal inconsistency in the evaluation claim: the separately presented abstract reports a TravelPlanner final pass rate of 35.56% (a 97.6% improvement over ATLAS on Claude-Sonnet-4), constraint violations reduced to 11 vs. 275, two production incident-diagnosis deployments, and additional ScienceWorld and ALFWorld results, while the full-text abstract and the body report a ν-bench average Pass^1 of 67.7% and contain no TravelPlanner, ATLAS, ScienceWorld, ALFWorld, or production-deployment results. The reader cannot determine which benchmark supports the paper's central claim, and the contradiction must be resolved before the results can be evaluated.","section":"Abstract vs. full text"},{"comment":"The numerical reporting is internally inconsistent and not reproducible. In Table 3, the row with Retail 74.0 and Airline 56.0 is labeled \"✓ ✓ ×\" (SC and DC without RT), and the final row labeled \"✓ ✓ ✓\" gives Retail 77.0 and Airline 56.0; these two rows yield simple averages of 65.0 and 66.5, respectively, not the reported 66.0 and 67.7. The table footnote says the average is \"weighted by domain rather than by individual task,\" but the weighting is not defined or justified. Because the headline 10.1-point margin depends on this average, the authors must provide the exact per-domain task counts or explicit weights and correct the repeated/incorrect row labels.","section":"Table 1 and Table 3 (metric computation)"},{"comment":"No measure of variability or statistical significance is reported for any pass rate. The τ-bench evaluation uses a GPT-4-based user simulator, so the agent's temperature setting of 0.0 does not make the evaluation deterministic; user-simulator randomness can affect pass rates. The paper reports single pass rates without standard errors, confidence intervals, or repeated evaluations, which is particularly problematic because the reported SOTA margin (10.1 points) is not accompanied by any uncertainty estimate. At minimum, the authors should report variance or rerun statistics for the headline comparisons.","section":"Experiment Setup / Metrics (variability)"}],"minor_comments":[{"comment":"There is a typo \"pandsas\" that should read \"pandas\" in the list of dependencies.","section":"Run Environment (Proposed Method)"},{"comment":"The header contains \"Sence\", which should be \"Scenario\" or \"Domain\".","section":"Table 2 heading"},{"comment":"The sentence \"As discussed in Section .\" contains an incomplete cross-reference that should be resolved to the relevant section on tool consolidation.","section":"Ablation Study"},{"comment":"The sentence beginning \"This methodology significantly reduces the risk of unexpected behaviors...\" is duplicated verbatim two paragraphs apart.","section":"User-Defined Configuration"},{"comment":"The sentence \"EBF is designed to fill this precise gap...\" uses the abbreviation EBF, which is never introduced; it should refer to the Source Code Agent framework.","section":"Background and Related Work"},{"comment":"The final row contains stray \"x\" characters in the numerical cells, and the row labels are misaligned, making the ablation table difficult to parse.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The mismatch between the separately presented abstract and the full-text body is serious enough that I would ask the authors to reconcile it before further consideration. I also recommend that the editor require release of code, blueprints, and tools, or at least a detailed authoring protocol, because the current evidence cannot distinguish framework-level gains from benchmark-specific manual rule encoding. The architecture itself is plausible and the production-deployment claim is potentially valuable, but the paper as submitted does not yet support its headline SOTA statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on 2508.02721. The architectural idea is real: run the workflow as deterministic code, call the LLM only for bounded sub-tasks. That separation is not brand-new—code-defined orchestration exists in various tools—but this paper states it cleanly and builds a full system around it. The ablation story is coherent: raw function calling 48.2, plus code structure 49.3, plus a double-check validation module 61.0, plus domain-specific retail tools 67.7. The efficiency numbers are plausible. I credit the group for saying plainly in the conclusion that blueprint creation is manual.\n\nNow the problems, in rough order of severity. First, the paper as written does not support its own headline. The submitted abstract reports TravelPlanner 35.56% and a 97.6% improvement over ATLAS; the body reports τ-bench average pass@1 67.7%. That's not a minor wording slip, it's two different evaluations. A reader cannot tell what was actually run. Second, the ablation table (Table 3) has duplicate rows: two rows both marked '✓ ✓ ×' with different numbers. Looks like a copy/paste error, but it undermines confidence in the numbers. Third, no code, no blueprints, no data, no variance. The largest gains come from the hand-written Double-Check module and Retail Tools, which are exactly the kind of benchmark-specific components that could be tuned to τ-bench. The paper does not say how blueprints were authored, whether they were iterated against the benchmark, or whether anything generalizes to a held-out domain. So the 10.1-point SOTA claim may measure encoded rules rather than the framework. The stress-test note is right about that.\n\nThe counterargument is that the ablation trend does show structure alone gives little, and the headline gain is mostly from the validation step, which is a generalizable pattern. But without the release and a held-out evaluation, it is a promising hypothesis, not a verified result.\n\nWho's this for: anyone building deterministic agent workflows in enterprise or operations. It's a worthwhile read for system builders.\n\nRecommendation: send to peer review. The idea deserves scrutiny and the evaluation is fixable—release blueprints, align abstract, add variance and at least one held-out domain. As-is, it's a conditional reject/major revision. My own verdict is skeptical but engaged.","headline":"Serious idea, sloppy write-up: the blueprint-first pattern is worth a careful look, but the current evaluation and internal inconsistencies make the SOTA claim unverifiable.","tokens_in":12046,"tokens_out":2190,"would_cite":false,"duration_ms":22009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that moving workflow decisions out of the generative model—encoding operational procedures as executable source-code blueprints run by a deterministic engine—turns the LLM from an unpredictable planner into a bounded…","keywords":["deterministic LLM agents","Execution Blueprint","procedural fidelity","τ-bench","Double-Check validation","tool consolidation","source code agent framework"],"falsifier":"Run the framework on a fresh, procedurally governed benchmark whose policy documents are withheld while the blueprints are written; if a developer who has never seen the test policies still scores near the reported 67.7% average $Pass^{1}$, the determinism claim is supported, and if the advantage collapses, the reported gains are manual rule-encoding rather than a property of the architecture.","tokens_in":11151,"feed_emoji":"🤖","tokens_out":14279,"duration_ms":124139,"temperature":0.7,"pith_summary":"The paper is trying to establish that the unreliability of LLM agents in rule-governed work is a structural flaw, not a model flaw: letting one generative model both plan and act makes the execution path unpredictable, so the remedy is to move workflow logic into deterministic code. It proposes a 'Blueprint First, Model Second' framework in which an expert-authored source-code blueprint is executed by a deterministic engine and the LLM is called only at bounded sub-task nodes. The body of the paper reports this framework reaching a 67.7% average $Pass^{1}$ on τ-bench with the strongest model tested, 10.1 points above the best baseline, with the largest gain coming from a coded Double-Check validation step. The abstract reports a different evaluation on the TravelPlanner benchmark, so the paper presents two sets of headline numbers for the same architecture.","feed_headline":"Let code decide the workflow: agents hit 67.7% on τ-bench","feed_subtitle":"Coding the workflow path makes rule-governed agents auditable and cuts mistakes and tool calls.","key_machinery":"The load-bearing object is the Execution Blueprint: a user-written, source-code-level procedure (such as a Python script) that fixes the workflow's states, conditionals, retry logic, and decision points, rather than letting a language model generate them step by step. It is executed by a sandboxed Source Code Executor, and the LLM enters only through SDK APIs at predetermined nodes. The Double-Check node is the component the ablation singles out as carrying the largest accuracy gain: a codified validation step that re-asserts task constraints and business rules at critical junctures before a proposed action is committed. The Retail Tools are consolidated, source-coded operations that replace sequences of fine-grained tool calls with a single call, which the paper argues improves both accuracy and efficiency.","core_discovery":"On the paper's own terms, the discovery is that an agent's adherence to procedural rules does not have to be elicited from a language model; it can be compiled. The framework's Execution Blueprint is a normal program that fixes each branch, retry, and validation, and the deterministic engine executes it exactly, so every deployed agent follows the same path for the same input. The LLM's role is deliberately surgical: it handles intent recognition, output parsing, and summarizing at points the blueprint marks, but never selects the next step. In evaluation, the strongest configuration (with the Double-Check node and consolidated Retail Tools) is what yields the reported 67.7% average $Pass^{1}$ on τ-bench, and the ablation traces the gain mainly to the Double-Check node's programmatic reassertion of constraints.","pith_inferences":["A reader should note that the abstract and the body report different benchmark evaluations for the same architecture: the abstract gives TravelPlanner numbers while the body's headline result is on τ-bench, and reconciling these is a prerequisite for treating the measured advantage as a single established fact.","A clean test of the determinism thesis would be a held-out, policy-heavy domain in which blueprint authors never see the test policy documents; credible gains there would separate architectural benefit from manual overfitting to a benchmark.","The Double-Check ablation result suggests that explicit, code-enforced validation is a more reliable carrier of rule adherence than prompt-based instruction, a component other agent architectures could adopt without committing to the full blueprint design.","Automating blueprint authoring from policy documents is the missing link that would turn this from an engineering practice into a generative pipeline; the paper itself names manual blueprint creation as its current limitation and semi-automation as future work."],"forward_implications":["Agents can be deployed in workflows where every step must be auditable, because a run's path is a deterministic trace through code with model-generated text confined to known nodes.","Rule adherence stops depending on the model remembering a long policy document, since the blueprint re-asserts constraints at decision points through the Double-Check mechanism.","Consolidating multi-step tool chains into single coded tools reduces conversational turns, token consumption, and latency, making the framework attractive for high-volume operational settings.","Because workflow logic lives in code rather than in prompts, swapping or shrinking the underlying model at selected nodes should not change the agent's procedural behavior."],"supporting_citations":[{"why":"Defines τ-bench, the benchmark's policy documents, user simulator, and Pass^1 metric, so the paper's headline comparison is measured against this benchmark.","marker":"Yao et al. 2024"},{"why":"Introduces chain-of-thought prompting, the representative generative-reasoning approach the paper argues does not guarantee external execution determinism.","marker":"Wei et al. 2022"},{"why":"Tree of Thoughts represents multi-path generative search that the paper contrasts with a single deterministic execution path.","marker":"Yao et al. 2023a"},{"why":"ReAct defines the reason-and-act architecture around which the paper's baselines and its 'other agents' contrast are built.","marker":"Yao et al. 2023b"},{"why":"Reflexion is the self-correcting agent baseline whose probabilistic revision the coded Double-Check validation is designed to replace.","marker":"Shinn et al. 2023"},{"why":"Plan-and-Solve prompting is an upfront-planning baseline that still relies on generative plans rather than an enforced code path.","marker":"Wang et al. 2023"}],"fun_headline_variants":["Blueprint-first agents cut rule breaks by 96%","Deterministic LLM agents: code the workflow path","LLM as tool, not planner: 97.6% better on travel","Compile agent logic to cut mistakes and tool calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-written blueprints and the domain-specific pieces (Double-Check and Retail Tools) are fair instantiations of a general framework rather than rules tuned against the τ-bench policy documents, and the paper does not report how they were authored or whether they were revised during evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Blueprint-first agents cut rule breaks by 96%","Deterministic LLM agents: code the workflow path","LLM as tool, not planner: 97.6% better on travel","Compile agent logic to cut mistakes and tool calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3566,"prompt_tokens":1011,"completion_tokens":2555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2485}},"tokens_in":627,"tokens_out":2555,"duration_ms":17621,"temperature":1.0,"reasoning_tokens":2485,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:13:32.746510+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on a fresh, procedurally governed benchmark whose policy documents are withheld while the blueprints are written; if a developer who has never seen the test policies still scores near the reported 67.7% average $Pass^{1}$, the determinism claim is supported, and if the advantage collapses, the reported gains are manual rule-encoding rather than a property of the architecture.","supporting_citations":[],"review_version":1}