{"id":"b53ba11a-b498-4466-8a44-819442779203","arxiv_id":"2607.06764","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Explorer-Definer and Reflective Orchestrator harnesses raise DeepSeek V3.2 from 15.5% to 67.25% pass@2 on ARC-AGI-1 public eval at $0.25–$0.62 per task without ARC-specific training.","lead":"Agent harnesses that split pattern discovery from program synthesis lift an open-weight model from 15.5% to 67% on ARC-AGI-1 at under $1 per task, without ARC fine-tuning. The work shows a large share of abstract-reasoning performance can come from architecture and adaptive re-exploration rather than scale or specialized training.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The generation-bound diagnostic and orchestrator confirmation rest on a single unreplicated M=5 candidate pool per architecture, so the near-equality of lifts could be run-specific rather than structural.","rationale":"The Reader correctly identifies public-set evaluation and config selection on the eval set as real caveats that keep the verdict CONDITIONAL rather than ACCEPT; those issues primarily threaten absolute accuracy transfer and the precise magnitude of the ~52-point lift. The more load-bearing soft spot for the paper's distinctive contribution (the diagnostic that the pipeline is generation-bound and that the orchestrator is the architectural test of that claim) is the unreplicated single-run candidate pools on which both the ~95% ceiling recovery and the +9.81 vs +9.75 near-equality rest. The paper already flags single-run variance and a ~±2 pp noise floor in §6, and the bimodality it reports makes the unbiased estimator especially sensitive to which tasks fall into the zero-correct bin. A multi-seed recomputation of exactly those two numbers would settle whether the confirmation is structural or run-specific. Because the Reader already chose CONDITIONAL for related reasons, the appropriate adjustment is UNCHANGED in label but with a sharper focus on the diagnostic's statistical fragility; I therefore leave the verdict CONDITIONAL and mark agreement partial.","tokens_in":23178,"tokens_out":698,"duration_ms":7180,"concrete_test":"Re-run the canonical pipeline and orchestrator (N=5, t=0.5, M=5) on the same 400 tasks for 2–3 independent seeds, recompute unbiased pass@1 and the fraction of the pass@M ceiling recovered by train-score selection for each seed, and check whether the orchestrator–pipeline unbiased-pass@1 delta remains within ~1 pp of the naive pass@2 delta in every seed (and whether recovered ceiling stays ≥90%). If the near-equality or the ~95% figure fails to hold across seeds, the generation-bound confirmation is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest interpretive claim is not merely the accuracy numbers but that the pipeline is generation-bound (train-score selection recovers ~95% of the candidate ceiling) and that the orchestrator's near-equal lifts (+9.75 pp naive pass@2 vs +9.81 pp unbiased pass@1) confirm generation-side gains from adaptive re-exploration (Abstract; §4.7). Both the ceiling fraction and the unbiased estimator are computed from one M=5 candidate distribution per architecture on the public set (Limitations §6 notes single-run point estimates and a ~±2 pp noise floor). Because the per-task correct-candidate counts are described as strongly bimodal, modest run-to-run variation in which tasks land in the 'none correct' bin can move the unbiased pass@1 and the recovered-ceiling percentage by several points, enough to make the near-equality of the two lifts appear tighter or looser than a stable structural property. The public-set / config-selection issues the Reader flags affect absolute levels; this concern is more load-bearing for the diagnostic-then-test narrative that justifies the orchestrator design itself.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper studies a third regime for ARC-AGI-1: an open-weight model (DeepSeek V3.2, non-thinking) under a strict budget with no ARC-specific fine-tuning, asking how much accuracy is recoverable through agent harness architecture alone. It introduces an Explorer-Definer Pipeline that separates pattern discovery from executable transformation synthesis (57.50% pass@2 at $0.25/task) and a Reflective Orchestrator that adds mid-loop adaptive re-exploration when hypotheses fail on training pairs (67.25% pass@2 at $0.62/task), lifting a 15.50% one-shot baseline by ~52 points. A generation-bound diagnostic (unbiased pass@k; train-score selection recovers ~95% of the candidate ceiling) motivates the orchestrator; near-equal lifts on naive pass@2 (+9.75 pp) and unbiased pass@1 (+9.81 pp) are presented as confirmation that gains are generation-side. Controlled ablations (think tool −5.75 pp; refinement −0.75 pp), an N×t×M Pareto surface, paired bootstrap CIs, and a 99-task Qwen3-235B cross-model check support the architectural claims.","tokens_in":23586,"tokens_out":1787,"duration_ms":30058,"significance":"If the within-model lifts hold, the work occupies a sparsely reported sub-$1/task region of the ARC cost–accuracy plane and shows that decomposition, a think scratchpad, and adaptive re-exploration can recover a large fraction of the gap between cheap one-shot calls and heavy test-time or ARC-specialized systems without benchmark-specific training. Strengths include: (i) tightly controlled within-model comparisons with token-normalized costs and paired bootstrap deltas; (ii) a falsifiable generation-vs-selection diagnostic that is then architecturally tested; (iii) load-bearing ablations (especially the think tool); (iv) public code and prompts; and (v) honest public-set and configuration-selection caveats. The diagnostic-then-test framing is a useful methodological contribution for agentic systems papers more generally.","major_comments":[{"comment":"§4.7 (and Abstract): The generation-bound claim and the orchestrator confirmation rest on a single unreplicated M=5 candidate pool per architecture. Unbiased pass@1 (46.40% pipeline; 56.21% orchestrator), the ~95% ceiling-recovery figure, and the near-equality of +9.75 pp naive pass@2 vs +9.81 pp unbiased pass@1 are all computed from one draw. Given the strongly bimodal correct-candidate counts and the paper’s own ~±2 pp single-run noise floor (§6), modest run-to-run movement of tasks into/out of the ‘none correct’ bin can change both the ceiling fraction and the lift equality by several points. This is load-bearing for Contribution 2 and for the claim that the orchestrator ‘confirms’ a generation-side prediction. Please either (a) report multi-seed M=5 pools (or a substantial subset) with uncertainty on unbiased pass@k and ceiling recovery, or (b) substantially soften the confirmation l","section":"§4.7 Generation-bound diagnostic"},{"comment":"§4.8 / §7: Spawn is presented as the dominant mechanism behind the orchestrator lift (69–75% of unique solves use explore_new_patterns), but there is no spawn-off ablation. The correlational task analysis and Appendix A case study are informative, yet they do not isolate spawn from the rest of the agentic loop (done exit, longer iteration budget, different prompting). Because the paper’s design narrative is that spawn is the relief valve for wrong-abstraction failures, a controlled spawn-off run (even on a fixed subset) is needed to make that causal claim, or the text should consistently frame the 75% figure as correlational evidence only and move stronger causal language to future work.","section":"§4.8 Behavioral characterization"},{"comment":"§6 Configuration selection: The canonical (N=5, t=0.5, M=5) point was chosen by inspecting the Pareto surface on the same 400-task public evaluation set used for all headline comparisons. The three mitigations offered (smooth surface; deltas hold across much of the surface; Qwen3 ordering) reduce but do not remove selection bias on absolute levels and on which cell is starred. At minimum, report the full surface numbers for nearby cells as primary results (not only the starred cell), and state clearly that absolute pass@2 at the chosen operating point is not an independent holdout estimate. Preferable: re-select on a validation split or pre-register the cell and re-evaluate the remainder.","section":"§6 Limitations; §4.4 Pareto surface"}],"minor_comments":[{"comment":"Public-evaluation caveat is well written but should appear earlier in the abstract or as a footnote on the headline numbers so readers do not treat 57.50%/67.25% as leaderboard-comparable without reading §6.","section":"Abstract / §1"},{"comment":"Table 1 reports pipeline pass@1 and pass@2 but only orchestrator pass@2. For symmetry and for the unbiased-pass@1 discussion in §4.7, include orchestrator pass@1 (or state why it is omitted).","section":"Table 1"},{"comment":"The 50% train-accuracy refinement threshold (§3.4) is described as probe-chosen and not rigorously tuned; a one-sentence sensitivity note (e.g., 40%/60%) would help readers judge robustness of the small refinement ablation.","section":"§3.4 Train-feedback refinement"},{"comment":"Figure 4/5 cost axes and AtlasCloud FP8 rates are clear; please also report mean tokens (prompt/completion) per architecture so costs remain interpretable if list prices change.","section":"§3.1 Cost accounting; Figures 4–7"},{"comment":"Cross-model Qwen3 result (n=99, M=3) is valuable; the CI for Qwen3 pipeline→orchestrator touches zero—state more explicitly in the main text that this is underpowered rather than evidence of a weaker lift.","section":"§4.9 / Table 4"},{"comment":"Minor: arXiv ID and some reference years in the provided text look placeholder-like (e.g., 2026 dates); verify bibliography consistency before camera-ready.","section":"References"},{"comment":"Appendix A task 3a301edc is excellent for intuition; a short pointer in §4.8 to the figure/table numbers would help readers find it from the main text.","section":"Appendix A; §4.8"}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical systems paper with unusually careful cost accounting and a genuine diagnostic-then-test arc. The main risk for the journal is over-reading absolute public-set accuracy as semi-private SOTA; the authors already caveat this, but the diagnostic confirmation language in the abstract is stronger than the single-run evidence supports. I would not reject on that basis if they soften or multi-seed. Fit is good for a methods/systems venue that values compute-efficiency claims; less so if the venue requires private-holdout leaderboard numbers."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this is a careful systems paper, not another leaderboard flex. Same open-weight model (DeepSeek V3.2 non-thinking), no ARC fine-tuning, token-priced costs, and three stacked levers—CoT, explorer–definer decomposition, then adaptive re-exploration—move pass@2 from 15.5% to 57.5% ($0.25) to 67.25% ($0.62) on the public 400. That sub-$1 region is sparsely reported; the numbers are new.\n\nWhat they do well is the discipline. Same rendering, scoring, and cost code across conditions; paired bootstrap deltas; N×t×M Pareto showing explorer count dominates and best-of-M saturates; think-tool ablation (−5.75 pp) and a weak refinement effect; unbiased pass@k showing train-score selection already recovers ~95% of the candidate ceiling. The orchestrator is not a random add-on: it is the design that follows that diagnostic, and the near-equal +9.75 / +9.81 lifts are exactly the confirmation they claim. Code is shipped. Trace stats and the Appendix A geometric-vs-arithmetic trap make the spawn story concrete. Qwen3 on 99 tasks preserves the ordering, so the harness is not pure V3.2 magic.\n\nSoft spots, in proportion. Public eval only, and the canonical (N,t,M) was chosen by looking at the Pareto on that same set—they say so. Absolute levels may not transfer to semi-private; deltas are still interpretable because everything is held fixed. Single-run point estimates and a ~±2 pp noise floor mean the exact equality of the two lifts could move a bit; the stress-test is fair that the diagnostic rests on one M=5 pool per architecture. Spawn-off ablation is deferred. None of that collapses the generation-bound argument or the controlled lifts.\n\nMath and citations look fine: standard unbiased pass@k, honest placement against Berman/Pang/TRM as regime context, not fake head-to-heads. For anyone working on test-time compute, agent harnesses, or ARC cost–accuracy, this is useful. I would send it to peer review; the caveats belong in revision, not desk rejection. Engage.","headline":"Solid agent-harness paper: controlled ~52-point lift on ARC public eval under a real budget, with a generation-bound diagnostic that the orchestrator is built to test; public-set and single-run caveats are real but do not erase the within-model evidence.","tokens_in":24159,"tokens_out":591,"would_cite":true,"duration_ms":8727,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Agent harnesses alone can lift a general open-weight model by ~52 points on ARC-AGI-1, under a dollar per task and with no ARC fine-tuning.","keywords":["ARC-AGI-1","agent harnesses","test-time compute","program synthesis","pattern exploration","adaptive re-exploration","pass@k","cost-accuracy frontier"],"falsifier":"Run the identical harnesses, prompts, and scoring code on the official semi-private ARC-AGI-1 holdout (or a true held-out split) with the same model and budget; if the ~52-point total lift or the pipeline-to-orchestrator generation-side lift collapses or shrinks dramatically, the central claim about transferable architectural recovery fails.","tokens_in":24088,"feed_emoji":"🧩","tokens_out":1098,"duration_ms":10263,"temperature":0.7,"pith_summary":"The paper argues that a large share of ARC-AGI-1 performance is available through architecture rather than through expensive test-time search over frontier models or through training small models on ARC data. On the public 400-task set, a one-shot call to DeepSeek V3.2 in non-thinking mode scores 15.5%. A structured chain-of-thought call reaches 30%. An Explorer-Definer Pipeline that separates pattern discovery from executable program synthesis reaches 57.5% pass@2 at $0.25 per task. A Reflective Orchestrator that can spawn fresh exploration mid-loop when a hypothesis fails training pairs reaches 67.25% pass@2 at $0.62. Unbiased pass@k analysis shows the pipeline is generation-bound: selection already recovers about 95% of the candidate ceiling, so further gains require broader generation. The orchestrator implements that prediction and confirms it with nearly equal lifts on naive pass@2 and unbiased pass@1. A sympathetic reader cares because the result claims that careful agent design recovers most of the gap between cheap single-call baselines and expensive leaderboard systems without specializing the model to the benchmark.","feed_headline":"Agent design alone lifts ARC scores 52 points under $1/task","feed_subtitle":"No ARC fine-tuning, no heavy search: pipeline and orchestrator recover most of the gap on a general open-weight model.","key_machinery":"The Explorer-Definer Pipeline (parallel PatternExplorers producing compressed natural-language findings, then TransformationDefiners that emit and refine executable Python scored on training pairs) plus the Reflective Orchestrator’s explore_new_patterns tool, which mid-loop spawns focused explorers under negative guidance when the current frame fails.","core_discovery":"Three architectural levers—within-call deliberation, across-call decomposition of pattern discovery from transformation synthesis, and adaptive re-exploration when a hypothesis fails—lift the same non-thinking open-weight model from 15.5% one-shot to 67.25% pass@2 on the ARC-AGI-1 public set at under a dollar per task, with no ARC-specific fine-tuning. The pipeline is generation-bound rather than selection-bound, and the orchestrator’s near-equal lifts on selection-free and selection-mediated metrics confirm that adaptive re-exploration supplies the missing generation diversity.","pith_inferences":["If generation-bound failures dominate, tools that let synthesis stages request fresh upstream abstractions under negative guidance may transfer to other program-induction or puzzle domains where multiple rules fit the examples.","Dedup-by-test-prediction plus mid-loop spawn together act as an implicit diverse-voting scheme; similar selection-plus-re-exploration pairs could be tested on other multi-hypothesis agent systems.","A controlled spawn-off ablation and full-scale multi-model panel would convert the correlational evidence that spawn drives unique solves into tighter causal and generality claims."],"forward_implications":["A meaningful fraction of ARC-AGI-1 accuracy is recoverable on open-weight models by agent design alone, without ARC fine-tuning or multi-dollar test-time search.","When unbiased pass@k shows selection already near the candidate ceiling, further budget should be spent on broader generation (more explorers or mid-loop re-exploration) rather than better ranking.","A private think scratchpad is a measurable load-bearing component of the pipeline; removing it costs roughly 6 pass@2 points.","The same three-lever progression and generation-bound diagnosis can be used as a design loop for other few-shot abstract-reasoning settings that supply training-pair verifiers."],"fun_headline_variants":["Agent harnesses alone lift ARC 52 points under $1/task","Pipeline plus orchestrator hit 67% ARC pass@2 at $0.62/task","Architecture alone: non-thinking model to 67.25% ARC without fine-tune","Decompose and re-explore: +52 points ARC from open-weight agents","Explorer-Definer then Reflective Orchestrator close ARC gap under budget"],"cache_read_input_tokens":12544,"weakest_assumption_plain":"That the absolute scores and the size of the architectural lifts measured on the public 400-task set, with the main configuration chosen by inspecting that same set, will transfer to the semi-private holdout and are not inflated by public-set artifacts or residual ARC exposure in the model’s general post-training.","fun_headline_variants_meta":{"raw":{"variants":["Agent harnesses alone lift ARC 52 points under $1/task","Pipeline plus orchestrator hit 67% ARC pass@2 at $0.62/task","Architecture alone: non-thinking model to 67.25% ARC without fine-tune","Decompose and re-explore: +52 points ARC from open-weight agents","Explorer-Definer then Reflective Orchestrator close ARC gap under budget"]},"model":"grok-4.5","effort":"low","cost_usd":0.006802,"raw_usage":{"total_tokens":1855,"prompt_tokens":986,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":68020000,"prompt_tokens_details":{"text_tokens":986,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":779,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":986,"tokens_out":90,"duration_ms":15300,"temperature":1.0,"reasoning_tokens":779,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T21:48:38.830954+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the identical harnesses, prompts, and scoring code on the official semi-private ARC-AGI-1 holdout (or a true held-out split) with the same model and budget; if the ~52-point total lift or the pipeline-to-orchestrator generation-side lift collapses or shrinks dramatically, the central claim about transferable architectural recovery fails.","supporting_citations":[],"review_version":1}