{"id":"31ee6b49-9781-4aae-a378-2a9374ca8fd9","arxiv_id":"2507.14393","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Automated workflow generation and iterative prompt refinement let a standard GPT-4.1 model outperform state-of-the-art reasoning models on a revised riddle benchmark.","lead":"Nexus Architect automatically builds a multi-agent reasoning workflow, then tunes the agents' instructions with feedback, so that an off-the-shelf GPT-4.1 model can answer riddle-style logic questions. On a custom 158-question benchmark it reports higher pass rates than several large reasoning models, including Gemini 2.5 Flash Preview and DeepSeek-R1.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IPR tuning examples are included in the final ArcBench pass rate, so the reported gains may reflect fitting to the evaluation set rather than generalization; a held-out split is needed.","rationale":"The reader identified the same load-bearing weakness: the IPR loop tunes on a subset of ArcBench that is also included when computing the final pass rate. This is the single most important threat to the paper's central claim because the paper's motivation is generalization, and the evaluation protocol conflates fitting to the tuning examples with generalization. I considered other candidate concerns, including the internal inconsistency between the text (Gemini best 44.94%) and Figure 2 (Gemini 39.24%), and the absence of a GPT-4.1 standalone baseline. Those are real issues, but the leakage is more fundamental because it directly undermines the interpretation of every reported pass rate. The proposed test is straightforward: evaluate on the 148 items not used in IPR. If the held-out numbers remain high, the concern is resolved and the conditional verdict can be upgraded. If they drop, the claim needs substantial revision. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not change it.","tokens_in":9183,"tokens_out":5359,"duration_ms":70594,"concrete_test":"Re-run the five Nexus Architect runs with a strict split: sample 10 QA pairs per run, run IPR only on those, freeze the resulting workflows and system prompts, then evaluate on the remaining 148 ArcBench items, excluding the 10 from the denominator entirely. Report mean, min, and max held-out pass rates alongside Gemini 2.5 Flash Preview and GPT-4.1 standalone evaluated on the same 148 items. If the mean held-out pass rate falls below the best LRM, or if GPT-4.1 alone matches the Architect's performance, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central generalization claim rests on an evaluation protocol that does not separate tuning from testing. Section 4.1 states that for each of five runs, 10 ArcBench question-answer pairs are 'instrumental for the IPR phase,' and Figure 3 shows the final pass rate as a solid line over the 'entire dataset.' Since the same 10 pairs are part of that dataset, the IPR loop has direct access to the exact questions and answers that later contribute to the reported score. Section 4.3 argues that using only 6% of samples demonstrates generalizability, but that argument is weakened because those 6% are scored in the final metric. Direct leakage is bounded at about 6.3 percentage points, so it cannot by itself explain the roughly 20–35 percentage point gap over the best LRM, but prompt overfitting to the sampled questions could inflate performance on other items that share superficial patterns. The paper also omits a GPT-4.1 standalone baseline, so the contribution of the multi-agent workflow over the raw model is unquantified. The load-bearing condition for the abstract's claim is that the reported pass rates measure performance on unseen ArcBench items; the current protocol does not establish that.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Nexus Architect, an extension of the Nexus multi-agent framework that automatically synthesizes a multi-agent reasoning workflow from a user prompt and a small set of problem-solution pairs, and then iteratively refines agent system prompts through an Iterative Prompt Refinement (IPR) loop. The authors evaluate the system on ArcBench, a revised English subset of the RoR-Bench riddle benchmark, using GPT-4.1 as the underlying non-reasoning LLM, and compare pass rates against Llama 4 Scout/Maverick, DeepSeek-R1, Claude 3.5 Sonnet/Sonnet 4/Opus 4, and Gemini 2.5 Flash Preview. They report that Architect consistently outperforms all baselines, with a best-case pass rate of 74.68% versus 44.94% for Gemini 2.5 Flash Preview, and interpret this as evidence that automated workflow generation and prompt refinement improve the generalization capabilities of standard LLMs.","tokens_in":9420,"tokens_out":5306,"duration_ms":56497,"significance":"If the result holds on a properly held-out evaluation, the paper would make a useful empirical contribution: it demonstrates an automated alternative to fine-tuning for reasoning tasks, and it ships open-source code and a benchmark. The five independent runs and the detailed configuration table are positive features. However, the central generalization claim is currently not established because the IPR tuning examples are included in the final pass rate, and the contribution of the multi-agent workflow over raw GPT-4.1 is not quantified. The open-source release and the clear description of the workflow synthesis mechanism are strengths that support reproducibility.","major_comments":[{"comment":"The final pass rate is computed over the entire ArcBench dataset, which includes the 10 question-answer pairs used for IPR tuning in each run. Section 4.1 states that 10 pairs are 'instrumental for the IPR phase,' and Figure 3 shows the final pass rate as a solid line over the whole dataset. Since the system prompts are refined on the exact questions and answers that later count toward the reported score, the reported 62.74% mean and 74.68% max are not measurements of generalization to unseen problems. Direct leakage is bounded at about 6.3 percentage points, so it cannot by itself explain the gap over the best baseline, but the paper's central claim that IPR improves generalizability (Section 4.3) requires reporting pass rates on the held-out subset (the 148 items not used in IPR) or using a separate tuning set. Without this, the abstract's phrase 'generalize to novel, unseen problems' is unsupported.","section":"Section 4.1, Figure 3"},{"comment":"There is no standalone GPT-4.1 baseline. The comparison in Figure 2 is between Architect, which runs GPT-4.1, and other models, but the paper never reports how GPT-4.1 alone performs on ArcBench with equivalent prompting and decoding settings. As a result, the contribution of the multi-agent workflow, the automated workflow synthesis, and the IPR loop over the raw model cannot be separated. The authors should add a GPT-4.1 zero-shot baseline, and ideally a GPT-4.1 baseline that receives the same 10 examples as in-context exemplars without the IPR loop.","section":"Section 4.2, Table 1"},{"comment":"The five Architect runs produce pass rates from 51.27% to 74.68%, yet all baseline models are reported as single point estimates without confidence intervals or significance tests. The claim 'consistently outperforms' and the precise multipliers (2.5x, 3x) are therefore only point estimates under unknown sampling variability. The authors should report per-run standard deviations or bootstrapped confidence intervals and, where differences are considered, a paired significance test across the 158 items.","section":"Section 4.2, Figure 2"},{"comment":"ArcBench is a revised version of RoR-Bench in which the authors 'revised several questions and answers to enhance their accessibility' and translated the set into English. Because the benchmark is authored and modified by the same group that evaluates its own system, the revision criteria are a free parameter that could bias results in favor of the proposed pipeline. The authors should justify the revisions with explicit criteria, report results on the original RoR-Bench items as a sanity check, and release the mapping between original and revised items.","section":"Section 4.1, ArcBench construction"}],"minor_comments":[{"comment":"The labels 'mean=62.74 max=74.68 min=51.27' appear in the plot area without a legend entry explaining that they refer to the five Nexus Architect runs; this should be clarified in the caption.","section":"Figure 2"},{"comment":"The pass rate metric is defined only as 'the ratio of correct answers over the total number of questions.' The paper should specify how answers are extracted and scored (exact match, LLM judge, or human review), how many samples per question were used, and whether the baselines were evaluated with greedy decoding or sampling.","section":"Section 4.1"},{"comment":"The configuration table lists temperature and top_p for each model, but the paper does not state whether the reported pass rates are averaged over multiple samples; given that GPT-4.1 is used with temperature and top_p both set to 1, stochasticity should be quantified.","section":"Appendix A, Table 1"},{"comment":"The revised benchmark name ArcBench is introduced in Section 4.1, but the introduction (list item 3) says it is 'derived from RoR-Bench'; the appendix or repository should include the full revision log to support reproducibility.","section":"Section 4.1, Appendix B"},{"comment":"Appendix B refers to the IPR examples as 'training example' and 'representative training example'; this terminology should be reconciled with the claim that the evaluation measures generalization, since the examples are drawn from the same benchmark.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical extension of the authors' own Nexus framework. The main risk is that the evaluation protocol conflates tuning and testing, and the absence of a raw GPT-4.1 baseline leaves the contribution of the workflow itself unquantified. These issues are fixable within the manuscript's scope by re-analyzing on a held-out split and adding baselines. The open-source release is a genuine asset, and I do not see grounds for rejection if the evaluation is corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it builds something real and then evaluates it in a way that cannot carry the abstract's claim. Nexus Architect automatically synthesizes a multi-agent workflow from a task description plus a few examples, and an iterative prompt-refinement loop tunes the agents' system prompts. That combination is genuinely new relative to the cited frameworks, and the open-source code and benchmark are a plus. The system description is clear, and the Appendix B walkthrough shows exactly how the IPR loop works, which I found helpful.\n\nThe problems are in Section 4. For each of five runs, ten ArcBench pairs are used for IPR and those same pairs are included when computing the final full-dataset pass rate. Direct leakage is bounded at about 6.3 percentage points, so it cannot by itself explain the 20–35 point gap over the best LRM, but the stress-test note is right: the protocol does not establish that the reported scores measure performance on unseen items. The paper's claim that using only 6% of samples demonstrates generalizability is weakened because those 6% are scored in the final metric.\n\nThe bigger gap is the missing GPT-4.1 standalone baseline. Without it, we cannot tell how much of the gain comes from the multi-agent workflow versus the underlying model. There are also no confidence intervals or significance tests across the five runs, and the authors revised the benchmark themselves to make it more accessible, which makes the comparison to LRMs harder to interpret. They also do not compare against prior automated agent design approaches, only against static frameworks like AutoGen and LangGraph.\n\nThese are fixable, but they are load-bearing for the headline. The paper is honest about its method, and the system is worth a serious referee; it just needs a clean held-out split, a base-model baseline, variance estimates, and ideally a comparison on the original benchmark. I would not cite the empirical result as it stands, but I would bring the paper to a reading group as a good case study in why evaluation protocols matter for agentic systems.\n\nRecommendation: send it to peer review with a request for major revision focused on the evaluation.","headline":"A real systems contribution with a suggestive but not yet supported headline claim: the evaluation mixes IPR tuning examples into the final score and omits the raw GPT-4.1 baseline.","tokens_in":9932,"tokens_out":1628,"would_cite":false,"duration_ms":23084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that automatically generating a multi-agent workflow—complete with roles, tools, and iterative prompt refinement—can turn an off-the-shelf non-reasoning model into a better solver of novel logic riddles than…","keywords":["multi-agent systems","automated workflow synthesis","iterative prompt refinement","large reasoning models","logical reasoning","riddle benchmark","generalization","GPT-4.1"],"falsifier":"Hold out every question used during IPR (or all ten tuning examples per run) from the final scored set and recompute the pass rate; if the advantage over Gemini 2.5 Flash Preview shrinks to near zero, the reported 62.74% mean is driven by prompt fitting rather than transferable reasoning.","tokens_in":8993,"feed_emoji":"🤖","tokens_out":9157,"duration_ms":611890,"temperature":0.7,"pith_summary":"Nexus Architect is an automated system that tries to establish that an ordinary, non-reasoning language model can outperform dedicated reasoning models on hard logic puzzles if it is embedded in a self-generated multi-agent workflow. Given a task description and ten example question–answer pairs, the system decomposes the task, designs a team of supervisor and worker agents with tools, validates the workflow on the examples, and then iteratively rewrites each agent's system prompt until failures are corrected. On ArcBench, a curated set of 158 translated and revised riddles, the resulting workflows running GPT-4.1 achieve a mean pass rate of 62.74% and a best of 74.68%, against 44.94% for the strongest tested reasoning model, Gemini 2.5 Flash Preview; the paper reports up to a 66% relative improvement over that model and $2.5\\times$ or better over several others. The importance, if the result is right, is that test-time orchestration and automated prompt engineering can substitute for expensive reasoning-specific training.","feed_headline":"Auto-built agent workflows out-reason top reasoning models","feed_subtitle":"GPT-4.1, orchestrated by Nexus Architect, beats Gemini 2.5 Flash and others on 158 logic riddles without fine-tuning.","key_machinery":"The machinery that carries the argument is the automated workflow synthesis pipeline closed by the Iterative Prompt Refinement (IPR) loop. The pipeline starts from a user prompt and example pairs, decomposes the task into requirements, designs a multi-agent blueprint, instantiates supervisors, workers, and tools through component builders, and validates the running workflow on the supplied examples. When validation fails, failure cases are analyzed and fed back to a Prompt Engineering stage, which rewrites the agents' system prompts for the next iteration; this reinforcement-style loop continues until performance criteria are met. IPR is the load-bearing part for the generalization claim because it improves behavior by changing instructions rather than model weights, and the paper reports that pass rates rise across five iterations despite only 10 of 158 questions being used for tuning.","core_discovery":"The discovery, stated on the paper's own terms, is that an automatically synthesized, validated, and iteratively refined multi-agent workflow makes an off-the-shelf non-reasoning model (GPT-4.1, used without fine-tuning) exceed the pass rate of state-of-the-art LRMs on a benchmark of novel logical riddles. The Architect generates the whole reasoning pipeline from a prompt and ten examples: it decomposes the task, selects strategies, tools, and adversarial techniques, builds the supervisor/worker topology, and then applies the IPR feedback loop, which converts validation failures into targeted modifications of system messages. Across five runs this yields a mean pass rate of 62.74% and a best of 74.68% on ArcBench, versus 44.94% for Gemini 2.5 Flash Preview, about 30.38% for Claude Sonnet 4 and DeepSeek-R1, and 24.68% for Llama 4 Scout. The paper reads this as evidence that robust reasoning can be democratized through workflow design and agentic automation rather than through ever-larger or reasoning-specialized models.","pith_inferences":["The IPR examples are sampled from the same 158-question pool used to compute final pass rates, so part of the reported advantage may reflect fitting system prompts to the test distribution; a held-out tuning set would separate transferable gains from fitting.","The appendix's digital-watch example suggests the IPR gains concentrate on riddle conventions—recognizing trick or punchline answers—so the margin over reasoning models could be smaller on formal, multi-step mathematical reasoning.","A direct test of transferability would synthesize a workflow on one family of problems (say arithmetic word problems), then deploy it unchanged on a disjoint family; the paper reports no such cross-family evaluation."],"forward_implications":["An off-the-shelf non-reasoning model can beat dedicated reasoning models on a logic-riddle benchmark when the workflow around it is generated automatically.","Reasoning gains can be obtained at test time through orchestration and prompt refinement, without fine-tuning or training a larger model.","Because each run tunes on only 10 of 158 questions yet raises final pass rates, the paper argues the improvement reflects generalization rather than simple memorization of the ArcBench items.","Open-source release of Nexus Architect and ArcBench allows the workflow-synthesis and IPR results to be reproduced and applied to other problem classes."],"supporting_citations":[{"why":"supplies the original riddle benchmark suite RoR-Bench from which ArcBench is derived","marker":"[8]"},{"why":"supplies the base Nexus multi-agent system that Architect builds on","marker":"[12]"},{"why":"supplies the reinforcement-style prompt-rewriting idea that IPR builds on","marker":"[23]"},{"why":"supplies a system-prompt optimization approach related to the IPR refinement loop","marker":"[24]"},{"why":"provides the Llama 4 Scout and Maverick baselines that the Architect is compared against","marker":"[25]"},{"why":"provides the DeepSeek-R1 baseline that the Architect is compared against","marker":"[26]"},{"why":"provides the Claude 3.5 Sonnet baseline that the Architect is compared against","marker":"[27]"},{"why":"provides the Claude Sonnet 4 and Claude Opus 4 baselines that the Architect is compared against","marker":"[28]"},{"why":"provides the Gemini 2.5 Flash Preview baseline, the strongest competitor in the comparison","marker":"[29]"}],"fun_headline_variants":["Automated workflow synthesis lets plain LLMs beat reasoning models","Nexus Architect auto-builds reasoning workflows that outperform LRMs","Non-reasoning GPT-4.1 with auto workflows surpasses top reasoning models","Workflow generation: off-the-shelf LLMs surpass specialized reasoners","Auto-generated reasoning pipelines outdo fine-tuned reasoning LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that tuning system prompts on ten examples drawn from the same ArcBench pool that is later scored still measures generalization to unseen problems, rather than fitting the workflow to the test set.","fun_headline_variants_meta":{"raw":{"variants":["Automated workflow synthesis lets plain LLMs beat reasoning models","Nexus Architect auto-builds reasoning workflows that outperform LRMs","Non-reasoning GPT-4.1 with auto workflows surpasses top reasoning models","Workflow generation: off-the-shelf LLMs surpass specialized reasoners","Auto-generated reasoning pipelines outdo fine-tuned reasoning LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00057,"raw_usage":{"total_tokens":2753,"prompt_tokens":1060,"completion_tokens":1693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":1600}},"tokens_in":676,"tokens_out":1693,"duration_ms":12393,"temperature":1.0,"reasoning_tokens":1600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:56:52.521445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out every question used during IPR (or all ten tuning examples per run) from the final scored set and recompute the pass rate; if the advantage over Gemini 2.5 Flash Preview shrinks to near zero, the reported 62.74% mean is driven by prompt fitting rather than transferable reasoning.","supporting_citations":[{"cited_title":"Llama 4 scout and maverick: Mixture-of-experts multimodal models,","cited_arxiv_id":null,"evidence_quote":"provides the Llama 4 Scout and Maverick baselines that the Architect is compared against"},{"cited_title":"Claude 3.5 Sonnet,","cited_arxiv_id":null,"evidence_quote":"provides the Claude 3.5 Sonnet baseline that the Architect is compared against"},{"cited_title":"Introducing Claude 4,","cited_arxiv_id":null,"evidence_quote":"provides the Claude Sonnet 4 and Claude Opus 4 baselines that the Architect is compared against"},{"cited_title":"Gemini 2.5 flash preview – model card,","cited_arxiv_id":null,"evidence_quote":"provides the Gemini 2.5 Flash Preview baseline, the strongest competitor in the comparison"}],"review_version":1}