{"id":"529a0044-3428-424f-9d7e-3c064891357c","arxiv_id":"2607.03601","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM agents beat architecture baselines with full simulator harnesses, but only one configuration stays above baseline without feedback, and performance modeling remains weak.","lead":"ArchEval is a 20-challenge, eight-simulator benchmark that tests whether LLM agents can act as computer architects under three levels of tool support. It finds agents improve real designs with a prepared harness, but mostly fail once simulator feedback is removed.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The L3 \"autonomous architect\" claim rests on a single-seed, harness-confounded snapshot that may not generalize beyond the four tested agent configurations.","rationale":"The reader's weakest_assumption correctly identifies the single-seed, lightweight-suite, mixed-baseline, LLM-judge design as the soft spot under the strongest claim. I agree: the L1 success and L3 collapse are the paper's load-bearing empirical spine, and they are currently a preliminary snapshot of four agent configurations, not a settled map of \"current agents.\" The paper is careful in abstract and conclusion, and the L1/L2/L3 ladder plus multi-simulator platform remain a real contribution, so the verdict should stay CONDITIONAL rather than move to REJECT or ACCEPT. A multi-seed re-run of the outcome tables is the cleanest falsifier; if the boundary holds, the claim strengthens without changing the paper's own scoping. No stronger internal inconsistency (e.g., metric definition error or verifier leakage) is evident from the text.","tokens_in":38098,"tokens_out":663,"duration_ms":5552,"concrete_test":"Re-run the full 20-challenge L1/L2/L3 suite for GPT-5.5 + Codex and Gemini 3.5 Flash under three independent seeds (or temperature-0 fixed decoding if available), reporting geomean, win rate, and L3 modeling pass rate with 95% intervals. If any non-Codex agent exceeds 1.0× L3 geomean on ≥2 seeds, or if GPT-5.5's L3 geomean drops below 1.0× on any seed, the assistant-vs-architect boundary is not yet robust enough for the paper's strongest claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central framing—that current agents are useful optimization assistants rather than autonomous architects—depends on the L1-to-L3 collapse (Table 10: all agents ≥ baseline at L1; only GPT-5.5 + Codex at 1.21× / 65% win rate at L3; 15% performance-modeling pass rate). That collapse is measured under three conditions the authors themselves flag in §5.1 and §8.3: (1) one run per agent–challenge–setting, (2) agent configurations that confound model with harness (GPT-5.5 uses Codex CLI; the other three use MiniSWE), and (3) 20 intentionally lightweight, budget-bounded challenges with mixed-strength baselines. If multi-seed variance is large, or if a stronger MiniSWE-style harness for Gemini/Gemma closes much of the L3 gap, the \"sharp boundary\" and the assistant-vs-architect framing become configuration-specific rather than a general capability map. Trajectory labels that use Gemma 4 31B (itself an evaluated agent) for grounding/novelty further soften process claims, but the load-bearing issue is the outcome table itself.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"ArchEval proposes a simulator-grounded benchmark and platform for evaluating LLM agents as computer architects. It defines 20 design challenges across CPU, system, memory, accelerator, and CIM domains, backed by eight simulators behind a common connector. Each challenge is run under three support levels—L1 full harness with repeated verifier feedback, L2 simulator-source container without a prepared loop, and L3 agent-only with no runnable feedback—while scoring baseline-normalized performance and recording full trajectories. Evaluating four agent configurations, the paper reports that all agents reach or exceed baselines under L1, but performance collapses without support: only GPT-5.5 + Codex remains above baseline at L3 (1.21× geomean, 65% win rate), with a 15% performance-modeling pass rate. The authors conclude that current agents are useful optimization assistants rather than autonomous architects, and identify missing capabilities in simulator-tool use, calibrated prediction, pre-feedback judgment, and mechanism discovery.","tokens_in":38432,"tokens_out":1578,"duration_ms":19102,"significance":"If the empirical pattern holds, this is a substantial contribution to both computer architecture and agent evaluation. Architecture has long measured artifacts (SPEC, MLPerf, etc.) rather than the designer; ArchEval makes the design process itself measurable under controlled experimental support. The L1/L2/L3 protocol is a clean methodological idea: it holds the task and verifier fixed while removing harness and feedback, separating assisted DSE from tool assembly and pre-feedback judgment. The multi-simulator platform (isolation, typed outcomes, baseline normalization, trajectory logging) is serious systems work and is reusable community infrastructure. Strengths include external, simulator-backed scoring; hard-failure typing; concrete case studies (AllReduce DSE, MNSIM L2 sweep, 256 B metadata budget, ASTRA-sim workload-follow); and an unusually candid limitations section. The paper’s main value is as a capability map and diagnostic platform, not as a final leaderboard.","major_comments":[{"comment":"§5.1 and Table 10: the central “sharp boundary” claim (all agents ≥ baseline at L1; only GPT-5.5 + Codex above baseline at L3) rests on one run per agent–challenge–setting. With n=1, geomean/win-rate differences cannot be distinguished from run-to-run variance, especially on stochastic agent systems. §8.3 acknowledges this as preliminary, but the abstract and §6 still generalize to “current agents.” At minimum, report multi-seed results for a subset of challenges (or bootstrap/confidence intervals), or reframe all suite-level claims as a single-seed snapshot of specific configurations rather than a general capability map.","section":"§5.1, Table 10"},{"comment":"§5.1: agent configurations confound model capability with harness infrastructure—GPT-5.5 uses the Codex CLI backend, while the other three use MiniSWE. L2 tool-use analysis (§5.4, Table 12) even excludes GPT-5.5 because its shell/file actions are not recorded in the MiniSWE stream. The L1→L3 collapse and “assistant vs. architect” framing may therefore be harness-specific. Either evaluate at least one shared harness across models, or systematically qualify every cross-agent comparison as configuration-level rather than model-level (the paper sometimes does this, but not consistently in the abstract and findings).","section":"§5.1, §5.4"},{"comment":"§6.1 (Tables 15–16) and §6.4 (Table 20): workload-grounding, design-following, and novelty labels are produced by Gemma 4 31B, which is itself one of the four evaluated agents. This creates a circularity risk for process claims (an evaluated system judging peers and, in part, itself). Primary verifier scores remain external and non-circular, but Findings 1 and 5 depend on these labels. Use an independent judge model (or human audit on a sample) and report inter-rater agreement; until then, treat those findings as provisional relative signals, as §8.3 already half-suggests.","section":"§6.1, §6.4"},{"comment":"§3.2 and Appendix A: baselines are of mixed provenance and strength (LRU, stock bimodal, hand-written references, one published Gibbon design). Beating a weak baseline is not the same as architectural competence. The paper reports win rate and hard failures, which helps, but does not quantify baseline strength (e.g., gap to a known strong human or published design where available). For the L3 “below baseline” narrative especially, a short baseline-strength audit would make the assistant-vs-architect claim more robust.","section":"§3.2, Appendix A"}],"minor_comments":[{"comment":"Table 5 marks ArchEval L1 as lacking workload analysis and performance prediction; this is protocol coverage, not agent inability, but the table can be misread as capability results. Clarify the caption that checkmarks are “exposed by the protocol,” not “demonstrated by agents.”","section":"Table 5"},{"comment":"Figure 5(c) L3/L1 retained geomean is useful; add absolute hard-failure counts beside win rates in Table 10 so readers can see whether L3 losses are invalid submissions or valid-but-weak designs (partially addressed in §5.5 text).","section":"Figure 5, Table 10"},{"comment":"The performance-modeling pass criterion (Kendall τ ≥ 0.6, executable design-sensitive model, ≥3 measured candidates) is a free parameter. Briefly justify the 0.6 threshold or show sensitivity at nearby cutoffs.","section":"§5.5, Table 13"},{"comment":"Novelty criterion (“beyond recombining ≤3 known policies”) is reasonable but underspecified for hybrid designs; a short appendix example of borderline cases would help reproducibility of Table 20.","section":"§6.4, Table 20"},{"comment":"Several model names (GPT-5.5, Gemini 3.5 Flash, Gemma 4 31B) will date quickly; pin exact API/model identifiers and dates in §5.1 for reproducibility.","section":"§5.1"},{"comment":"Appendix trajectory excerpts are valuable; consider moving one fully annotated L1 vs L3 pair into the main body to illustrate the protocol for readers who skip the appendix.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"This is a strong systems/benchmark paper with real engineering and a useful conceptual frame. The empirical study is honestly preliminary; I would not reject on that basis if the authors tighten claim language and add either multi-seed evidence or clearer configuration-level framing. Fit is good for an architecture venue that values measurement infrastructure. Main editorial risk is over-reading a single-seed, harness-confounded snapshot as a definitive map of “AI architects.”"},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is the protocol, not the leaderboard. ArchEval holds the same architecture task fixed while removing experimental support: L1 prepared harness with repeated verifier feedback, L2 simulator source without a ready loop, L3 static evidence only. That ladder, plus a connector over eight real simulators and 20 challenges spanning CPU, memory, accelerators, collectives, and CIM, is the actual contribution. Prior work had software agents, architecture QA, or LLM-in-the-loop DSE inside a fixed harness. This is the first setup I have seen that cleanly separates assisted optimization from simulator-tool use and pre-feedback judgment.\n\nWhat they measure is also honest. Primary scores come from isolated verifiers and baseline-normalized metrics, not self-report. The L1 result is solid: all four configurations reach or beat baselines and improve real designs. The L2 process tables and L3 prediction diagnostics (high self-eval error, overestimation, 15% modeling pass rate even for the best system) match the trajectories and case studies. The paper does not overclaim autonomy; it frames agents as optimization assistants and names the missing skills. That is the right conclusion from the evidence they have.\n\nSoft spots are real but mostly the ones the authors already flag. One seed per cell, four agent configurations that confound model with harness (Codex vs MiniSWE), lightweight budget-bounded tasks, mixed baseline strength, and Gemma-as-judge on some process labels. The stress-test is fair: if multi-seed variance is large or a better harness closes the L3 gap for the weaker systems, the sharp boundary is configuration-specific rather than a general map of “current agents.” That weakens the grand framing more than the platform itself. Novelty labeling is thin; one “novel” policy that loses is a nice anecdote, not a discovery claim.\n\nThis is for people building AI-for-architecture systems and for anyone who needs a shared way to measure agent design loops. It deserves a serious referee. I would use the suite and cite the L1/L2/L3 distinction; I would not treat the single-seed numbers as settled capability science. Engage, release, multi-seed, broaden the agents.","headline":"Real multi-simulator architecture-agent benchmark with a useful L1/L2/L3 support ladder; the assistant-vs-architect framing is directionally right but rests on a single-seed, harness-confounded snapshot.","tokens_in":39119,"tokens_out":566,"would_cite":true,"duration_ms":6459,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Current AI agents improve real computer-architecture designs only when humans supply the harness and simulator feedback; without that support they fail as autonomous architects.","keywords":["computer architecture","LLM agents","design-space exploration","simulator benchmarks","performance prediction","ArchEval","agent evaluation","hardware design"],"falsifier":"A multi-seed re-run in which several independent agent configurations, without the L1 harness, consistently beat the same baselines with calibrated predictions (performance-modeling pass rate well above 15%) across the twenty challenges—or L1 gains vanishing once stronger baselines or longer industrial-scale verifiers replace the lightweight ones.","tokens_in":38947,"feed_emoji":"🖥️","tokens_out":934,"duration_ms":14532,"temperature":0.7,"pith_summary":"ArchEval asks whether language-model agents can do the work of computer architects: read workloads, choose mechanisms, drive simulators, predict performance before measurement, and stay inside hard constraints. It poses twenty design problems spanning CPU cores, memory, accelerators, distributed training, and compute-in-memory, each under three support levels—from a prepared optimization loop with repeated feedback, through raw simulator source the agent must turn into experiments, to static evidence only with no runnable feedback before submission. With the full harness, all four tested agents reach or beat the challenge baselines and improve real designs. When that support is removed, only one configuration stays above baseline, and even that system ranks candidate designs correctly before measurement only about fifteen percent of the time. The paper therefore frames today’s agents as useful optimization assistants inside human-built workflows, not as stand-alone architects, and names the missing skills: simulator-tool use, calibrated prediction, pre-feedback judgment, and useful mechanism discovery.","feed_headline":"AI agents beat architecture baselines only with a full harness","feed_subtitle":"Strip simulator feedback and most fall below baseline; even the best rarely predicts performance","key_machinery":"The L1/L2/L3 support ladder: the same architecture task under full harness with repeated verifier feedback (L1), simulator source the agent must assemble into experiments (L2), and static workload evidence with no runnable feedback before submission (L3). Paired with baseline-normalized verifier scores and full trajectory logs, the ladder separates assisted search from simulator-tool use and pre-feedback design judgment.","core_discovery":"With a prepared harness and repeated simulator feedback, current LLM agents already improve real architecture designs across diverse simulators and meet or exceed challenge baselines; once that support is stripped away, most fall below baseline, and even the strongest configuration reaches only about 1.21× geomean baseline-normalized performance with a 15% performance-modeling pass rate—so today’s agents function as optimization assistants rather than autonomous architects.","pith_inferences":["The same support ladder could diagnose agents in other slow-to-measure design fields, such as chip floorplanning or network-protocol design.","Decomposing workload analysis, surrogate modeling, and constraint checking into specialist modules may close the L3 gap faster than scaling one monolith.","If performance-modeling pass rates remain near 15%, autonomous architecture research will stay gated by human-built oracles for a long time.","As the suite becomes public, held-out or newly contributed challenges will be needed to keep the capability map free of contamination."],"forward_implications":["Near-term use of agents should keep them inside prepared verifier-simulator loops rather than unsupervised early-stage design.","Progress should be measured on simulator-tool use, calibrated prediction, and pre-feedback judgment, not final score alone.","Workload analyses should be audited against whether they actually change the submitted design.","Resource limits work when stated in program-checkable form and enforced by the verifier; feasibility is not design quality.","Next agents need mechanism discovery that survives workload, constraint, and simulator checks, not only recombination of known policies."],"fun_headline_variants":["Agents beat arch baselines only with full simulator harness","Strip the harness and most AI architecture agents fall below baseline","LLM agents improve real designs only with repeated simulator feedback","Without feedback, even top agents rarely model performance correctly","ArchEval: agents act as optimizers, not autonomous architects"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The general map of “current agents as architects” rests on a single-seed evaluation of twenty intentionally lightweight, budget-bounded challenges with mixed-strength baselines and some process labels produced by one of the agents under test.","fun_headline_variants_meta":{"raw":{"variants":["Agents beat arch baselines only with full simulator harness","Strip the harness and most AI architecture agents fall below baseline","LLM agents improve real designs only with repeated simulator feedback","Without feedback, even top agents rarely model performance correctly","ArchEval: agents act as optimizers, not autonomous architects"]},"model":"grok-4.5","effort":"low","cost_usd":0.00492,"raw_usage":{"total_tokens":1468,"prompt_tokens":876,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":49200000,"prompt_tokens_details":{"text_tokens":876,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":512,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":876,"tokens_out":80,"duration_ms":4197,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T01:15:45.725695+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A multi-seed re-run in which several independent agent configurations, without the L1 harness, consistently beat the same baselines with calibrated predictions (performance-modeling pass rate well above 15%) across the twenty challenges—or L1 gains vanishing once stronger baselines or longer industrial-scale verifiers replace the lightweight ones.","supporting_citations":[],"review_version":1}