Pith. sign in

REVIEW 4 major objections 6 minor 77 references

Current AI agents improve real computer-architecture designs only when humans supply the harness and simulator feedback; without that support they fail as autonomous architects.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:15 UTC pith:LOTMTVYJ

load-bearing objection Real multi-simulator architecture-agent benchmark with a useful L1/L2/L3 support ladder; the assistant-vs-architect framing is directionally right but rests on a single-seed, harness-confounded snapshot. the 4 major comments →

arxiv 2607.03601 v1 pith:LOTMTVYJ submitted 2026-07-03 cs.AR

ArchEval: Measuring AI Agents as Computer Architects

Chenyu Wang , Zishen Wan , Jeffrey Ma , Shvetank Prakash , Zhenting Qi , Haebin Do , Andy Cheng , Arya Tschand
show 3 more authors
Jiahe Shi Yilun Du Vijay Janapa Reddi
This is my paper
classification cs.AR
keywords computer architectureLLM agentsdesign-space explorationsimulator benchmarksperformance predictionArchEvalagent evaluationhardware design
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ArchEval asks whether language-model agents can do the work of computer architects: read workloads, choose mechanisms, drive simulators, predict performance before measurement, and stay inside hard constraints. It poses twenty design problems spanning CPU cores, memory, accelerators, distributed training, and compute-in-memory, each under three support levels—from a prepared optimization loop with repeated feedback, through raw simulator source the agent must turn into experiments, to static evidence only with no runnable feedback before submission. With the full harness, all four tested agents reach or beat the challenge baselines and improve real designs. When that support is removed, only one configuration stays above baseline, and even that system ranks candidate designs correctly before measurement only about fifteen percent of the time. The paper therefore frames today’s agents as useful optimization assistants inside human-built workflows, not as stand-alone architects, and names the missing skills: simulator-tool use, calibrated prediction, pre-feedback judgment, and useful mechanism discovery.

Core claim

With a prepared harness and repeated simulator feedback, current LLM agents already improve real architecture designs across diverse simulators and meet or exceed challenge baselines; once that support is stripped away, most fall below baseline, and even the strongest configuration reaches only about 1.21× geomean baseline-normalized performance with a 15% performance-modeling pass rate—so today’s agents function as optimization assistants rather than autonomous architects.

What carries the argument

The L1/L2/L3 support ladder: the same architecture task under full harness with repeated verifier feedback (L1), simulator source the agent must assemble into experiments (L2), and static workload evidence with no runnable feedback before submission (L3). Paired with baseline-normalized verifier scores and full trajectory logs, the ladder separates assisted search from simulator-tool use and pre-feedback design judgment.

Load-bearing premise

The general map of “current agents as architects” rests on a single-seed evaluation of twenty intentionally lightweight, budget-bounded challenges with mixed-strength baselines and some process labels produced by one of the agents under test.

What would settle it

A multi-seed re-run in which several independent agent configurations, without the L1 harness, consistently beat the same baselines with calibrated predictions (performance-modeling pass rate well above 15%) across the twenty challenges—or L1 gains vanishing once stronger baselines or longer industrial-scale verifiers replace the lightweight ones.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Near-term use of agents should keep them inside prepared verifier-simulator loops rather than unsupervised early-stage design.
  • Progress should be measured on simulator-tool use, calibrated prediction, and pre-feedback judgment, not final score alone.
  • Workload analyses should be audited against whether they actually change the submitted design.
  • Resource limits work when stated in program-checkable form and enforced by the verifier; feasibility is not design quality.
  • Next agents need mechanism discovery that survives workload, constraint, and simulator checks, not only recombination of known policies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same support ladder could diagnose agents in other slow-to-measure design fields, such as chip floorplanning or network-protocol design.
  • Decomposing workload analysis, surrogate modeling, and constraint checking into specialist modules may close the L3 gap faster than scaling one monolith.
  • If performance-modeling pass rates remain near 15%, autonomous architecture research will stay gated by human-built oracles for a long time.
  • As the suite becomes public, held-out or newly contributed challenges will be needed to keep the capability map free of contamination.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ArchEval proposes a simulator-grounded benchmark and platform for evaluating LLM agents as computer architects. It defines 20 design challenges across CPU, system, memory, accelerator, and CIM domains, backed by eight simulators behind a common connector. Each challenge is run under three support levels—L1 full harness with repeated verifier feedback, L2 simulator-source container without a prepared loop, and L3 agent-only with no runnable feedback—while scoring baseline-normalized performance and recording full trajectories. Evaluating four agent configurations, the paper reports that all agents reach or exceed baselines under L1, but performance collapses without support: only GPT-5.5 + Codex remains above baseline at L3 (1.21× geomean, 65% win rate), with a 15% performance-modeling pass rate. The authors conclude that current agents are useful optimization assistants rather than autonomous architects, and identify missing capabilities in simulator-tool use, calibrated prediction, pre-feedback judgment, and mechanism discovery.

Significance. If the empirical pattern holds, this is a substantial contribution to both computer architecture and agent evaluation. Architecture has long measured artifacts (SPEC, MLPerf, etc.) rather than the designer; ArchEval makes the design process itself measurable under controlled experimental support. The L1/L2/L3 protocol is a clean methodological idea: it holds the task and verifier fixed while removing harness and feedback, separating assisted DSE from tool assembly and pre-feedback judgment. The multi-simulator platform (isolation, typed outcomes, baseline normalization, trajectory logging) is serious systems work and is reusable community infrastructure. Strengths include external, simulator-backed scoring; hard-failure typing; concrete case studies (AllReduce DSE, MNSIM L2 sweep, 256 B metadata budget, ASTRA-sim workload-follow); and an unusually candid limitations section. The paper’s main value is as a capability map and diagnostic platform, not as a final leaderboard.

major comments (4)
  1. [§5.1, Table 10] §5.1 and Table 10: the central “sharp boundary” claim (all agents ≥ baseline at L1; only GPT-5.5 + Codex above baseline at L3) rests on one run per agent–challenge–setting. With n=1, geomean/win-rate differences cannot be distinguished from run-to-run variance, especially on stochastic agent systems. §8.3 acknowledges this as preliminary, but the abstract and §6 still generalize to “current agents.” At minimum, report multi-seed results for a subset of challenges (or bootstrap/confidence intervals), or reframe all suite-level claims as a single-seed snapshot of specific configurations rather than a general capability map.
  2. [§5.1, §5.4] §5.1: agent configurations confound model capability with harness infrastructure—GPT-5.5 uses the Codex CLI backend, while the other three use MiniSWE. L2 tool-use analysis (§5.4, Table 12) even excludes GPT-5.5 because its shell/file actions are not recorded in the MiniSWE stream. The L1→L3 collapse and “assistant vs. architect” framing may therefore be harness-specific. Either evaluate at least one shared harness across models, or systematically qualify every cross-agent comparison as configuration-level rather than model-level (the paper sometimes does this, but not consistently in the abstract and findings).
  3. [§6.1, §6.4] §6.1 (Tables 15–16) and §6.4 (Table 20): workload-grounding, design-following, and novelty labels are produced by Gemma 4 31B, which is itself one of the four evaluated agents. This creates a circularity risk for process claims (an evaluated system judging peers and, in part, itself). Primary verifier scores remain external and non-circular, but Findings 1 and 5 depend on these labels. Use an independent judge model (or human audit on a sample) and report inter-rater agreement; until then, treat those findings as provisional relative signals, as §8.3 already half-suggests.
  4. [§3.2, Appendix A] §3.2 and Appendix A: baselines are of mixed provenance and strength (LRU, stock bimodal, hand-written references, one published Gibbon design). Beating a weak baseline is not the same as architectural competence. The paper reports win rate and hard failures, which helps, but does not quantify baseline strength (e.g., gap to a known strong human or published design where available). For the L3 “below baseline” narrative especially, a short baseline-strength audit would make the assistant-vs-architect claim more robust.
minor comments (6)
  1. [Table 5] Table 5 marks ArchEval L1 as lacking workload analysis and performance prediction; this is protocol coverage, not agent inability, but the table can be misread as capability results. Clarify the caption that checkmarks are “exposed by the protocol,” not “demonstrated by agents.”
  2. [Figure 5, Table 10] Figure 5(c) L3/L1 retained geomean is useful; add absolute hard-failure counts beside win rates in Table 10 so readers can see whether L3 losses are invalid submissions or valid-but-weak designs (partially addressed in §5.5 text).
  3. [§5.5, Table 13] The performance-modeling pass criterion (Kendall τ ≥ 0.6, executable design-sensitive model, ≥3 measured candidates) is a free parameter. Briefly justify the 0.6 threshold or show sensitivity at nearby cutoffs.
  4. [§6.4, Table 20] Novelty criterion (“beyond recombining ≤3 known policies”) is reasonable but underspecified for hybrid designs; a short appendix example of borderline cases would help reproducibility of Table 20.
  5. [§5.1] Several model names (GPT-5.5, Gemini 3.5 Flash, Gemma 4 31B) will date quickly; pin exact API/model identifiers and dates in §5.1 for reproducibility.
  6. [Appendix C] Appendix trajectory excerpts are valuable; consider moving one fully annotated L1 vs L3 pair into the main body to illustrate the protocol for readers who skip the appendix.

Circularity Check

1 steps flagged

Main L1–L3 performance claims rest on independent simulator verifiers and baselines; only minor non-load-bearing circularity in using an evaluated agent (Gemma 4 31B) as trajectory judge for auxiliary process labels.

specific steps
  1. other [Section 6.1 / Table 15; also Tables 16, 20 and Section 8.3]
    "Grounded and Guides design are labeled by Gemma 4 31B. ... In this version, the trajectory rubric uses Gemma 4 31B for semantic questions such as workload grounding, originality, and artifact-rationale consistency, so those judgments should be treated as relative signals that require further calibration."

    Process diagnostics that support Findings 1 (workload analyses often ungrounded or unused) and 5 (little genuine mechanism novelty) are produced by Gemma 4 31B, which is itself one of the four agents under evaluation. For that agent the labels are therefore partly self-referential, and the judge may share architectural or training biases with the systems being scored. The paper acknowledges the need for further calibration; the labels remain non-load-bearing because the primary L1/L2/L3 performance numbers and the assistant-vs-architect framing rest on independent simulator verifiers, not on these semantic judgments.

full rationale

ArchEval is an empirical benchmark paper, not a first-principles derivation. Its central claims (all agents ≥ baseline under L1 full harness; only GPT-5.5 + Codex remains above baseline at L3 with 1.21× geomean / 65% win rate and 15% performance-modeling pass rate) are produced by isolated canonical verifier simulators that score submitted artifacts against per-challenge baselines via typed outcomes (SIM_OK, BUILD_FAIL, etc.) and baseline-normalized metrics. Those measurements do not reduce to the paper’s own inputs by construction, nor do they rely on fitted parameters renamed as predictions or on uniqueness theorems imported from the authors. The sole residual circularity is auxiliary: Gemma 4 31B (itself one of the four evaluated agent configurations) supplies the semantic labels for workload grounding, design-following, and novelty that support Findings 1 and 5 and Tables 15–16/20. The paper itself flags these as “relative signals that require further calibration” (Section 8.3). Because the labels are not load-bearing for the headline geomean/win-rate results, the circularity is minor and non-central, corresponding to score 2 rather than 0 or higher.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

ArchEval is an empirical systems/benchmark paper. Its claims rest on domain conventions of simulator-grounded architecture evaluation, a constructed challenge suite and baselines, and operational definitions of agent capability—not on free physical constants or new physical entities. The load-bearing modeling choices are the L1/L2/L3 support ladder, baseline-normalized scoring, and the trajectory rubric.

free parameters (3)
  • performance-modeling pass threshold (Kendall τ ≥ 0.6 plus executable design-sensitive model and ≥3 measured candidates) = τ ≥ 0.6
    The 15% pass-rate headline depends on this hand-chosen agreement threshold and gate structure (Table 13 / Section 5.5).
  • per-challenge verifier-attempt caps and runtime budgets = challenge-specific (e.g., 10/1/1 or 5/5/1)
    Caps such as L1/L2/L3 attempt limits and ~2-hour verifier budgets shape which designs can be explored and which challenges enter the suite (Appendix A, Section 4.3).
  • novelty criterion (beyond recombining ≤3 known policies with parameter tuning) = ≤3 known policies
    The novelty finding is defined by this audit rule on six code-authoring challenges (Table 20).
axioms (4)
  • domain assumption Simulator-backed, baseline-normalized metrics are a valid proxy for architecture design quality within each challenge.
    Stated throughout Sections 1 and 3; architecture quality is measured by ChampSim/gem5/etc. rather than silicon.
  • ad hoc to paper Holding the task fixed while removing prepared harness and simulator feedback isolates architect-relevant capabilities (assisted DSE, tool use, pre-feedback judgment).
    Core methodological claim of the L1/L2/L3 protocol (Section 3.3, Table 4).
  • domain assumption Lightweight challenges with bounded runtimes still require real architecture decisions rather than toy puzzles.
    Explicit design choice in Section 3.2; needed to make agent evaluation practical.
  • domain assumption Complete agent configurations (model + harness) are the right unit of comparison, not base models alone.
    Section 5.1; GPT-5.5 uses Codex while others use MiniSWE.
invented entities (3)
  • ArchEval L1/L2/L3 evaluation settings independent evidence
    purpose: Vary experimental support while keeping task, workload, verifier, and baseline fixed.
    Central invented protocol construct; not a physical entity, but the paper’s main measurement device.
  • Architecture-agent capability map (prediction, optimization, generation) with trajectory rubric dimensions no independent evidence
    purpose: Interpret final scores in terms of workload analysis, tool use, performance judgment, constraints, and integrity.
    Organizing framework in Section 2 and Table 6; operationalized by process labels.
  • Connector-mediated multi-simulator verifier platform independent evidence
    purpose: Present heterogeneous simulators through a uniform agent tool interface with isolation and logging.
    Platform contribution in Section 4; engineering entity enabling the benchmark.

pith-pipeline@v1.1.0-grok45 · 42165 in / 3579 out tokens · 31474 ms · 2026-07-12T01:15:45.725695+00:00 · methodology

0 comments
read the original abstract

Computer architecture has long used benchmarks to make progress measurable. LLM agents create a different measurement problem: success is not merely writing code or tuning parameters. The agent must interpret workloads, choose mechanisms, use simulators, predict performance, satisfy hard constraints, and decide which feasible design is worth evaluating. This paper introduces ArchEval, a benchmark and platform for evaluating LLM agents on computer architecture design and optimization. It contains 20 challenges across CPU core mechanisms, system architecture, memory systems, accelerators, and compute-in-memory, backed by eight simulators. Each challenge is posed under three settings: L1 full harness, with repeated simulator feedback; L2 simulator-code container, where simulator source is available but the agent must assemble its own workflow; and L3 agent-only, with no runnable feedback before submission. Each run reports baseline-normalized verifier performance and records the full trajectory, connecting results to workload analysis, simulator-tool use, prediction, constraint handling, and artifact integrity. Initial results show a sharp boundary in current agents. With L1 support, all four evaluated agents reach or exceed baseline and improve real designs across diverse simulators. Removing support exposes weaknesses: many agents fail to turn simulator source into useful experiments, and L3 predictions often disagree with verifier results. In L3, only GPT-5.5 + Codex remains above baseline, reaching 1.21x geomean performance and a 65% win rate; the other three fall below baseline. Even GPT-5.5 + Codex has only a 15% performance-modeling pass rate. ArchEval frames today's agents as useful optimization assistants rather than autonomous architects, and identifies capabilities needed next: simulator-tool use, calibrated prediction, pre-feedback judgment, and useful mechanism discovery.

Figures

Figures reproduced from arXiv: 2607.03601 by Andy Cheng, Arya Tschand, Chenyu Wang, Haebin Do, Jeffrey Ma, Jiahe Shi, Shvetank Prakash, Vijay Janapa Reddi, Yilun Du, Zhenting Qi, Zishen Wan.

Figure 1
Figure 1. Figure 1: ArchEval evaluates LLM agents as computer archi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: ArchEval evaluates agents through a controlled architecture-design workflow. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: One task, three evaluation settings. A worked example: the same cache replacement task is posed under L1 to L3 using the same agent. What changes is the prompt’s requirement, the experimental support available to the agent (L2 gets simulator source; L3 gets only static workload evidence), and how verifier feedback is used. L1 can iterate on repeated measurements; L2 and L3 stop at the first valid verifier … view at source ↗
Figure 4
Figure 4. Figure 4: Challenge-suite composition. Distribution of the 20 final challenges by architecture domain. changes across evaluation settings and distinguish performance failures from simulator-tool, constraint, and shortcut failures. 5 Experiments In this section we discuss early evaluation results from ArchEval. We first describe the experimental setup (Section 5.1), then report the main results across the three evalu… view at source ↗
Figure 5
Figure 5. Figure 5: Performance drops most sharply when agents lose simulator feedback. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: L1 agents improve with repeated verifier-simulator [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 21 linked inside Pith

  1. [1]

    Yash Akhauri, Bryan Lewandowski, Cheng-Hsi Lin, Adrian N Reyes, Grant C Forbes, Arissa Wongpanich, Bangding Yang, Mohamed S Abdelfattah, Sagi Perel, and Xingyou Song. 2025. Performance prediction for large systems via text-to- text regression.arXiv preprint arXiv:2506.21718(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC benchmark suite: Characterization and architectural implications. In 18 ArchEval: Measuring AI Agents as Computer Architects Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques. 72–81

    2008

  3. [3]

    Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, Tushar Krishna, Somayeh Sardashti, et al. 2011. The gem5 simulator.ACM SIGARCH Computer Architecture News39, 2 (2011), 1–7

    2011

  4. [4]

    Alexander Blasberg, Vasilis Kypriotis, and Dimitrios Skarlatos. 2026. Agentic Architect: An Agentic AI Framework for Architecture Design Exploration and Optimization.arXiv preprint arXiv:2604.25083(2026)

    Pith/arXiv arXiv 2026

  5. [5]

    Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2023. Chip-Chat: Challenges and Opportunities in Conversational Hardware Design. arXiv:2305.13243 [cs.AR] https://arxiv.org/abs/2305.13243

    Pith/arXiv arXiv 2023

  6. [6]

    Kistowski

    James Bucek, Klaus-Dieter Lange, and Jóakim v. Kistowski. 2018. SPEC CPU2017: Next-generation compute benchmark. InCompanion of the 2018 ACM/SPEC International Conference on Performance Engineering. 41–42

    2018

  7. [7]

    Weinan Dai, Hanlin Wu, Qiying Yu, Huan ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yufan Song, Hongli Yu, Jiaze Chen, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Mingxuan Wang, Xin Liu, and Hao Zhou. 2026. CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation. arXiv:2602.24286 [cs.LG] https://arxiv.org/abs/2602.24286

    arXiv 2026

  8. [8]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.arXiv preprint arXiv:2010.11929(2020)

    Pith/arXiv arXiv 2020

  9. [9]

    Yilun Du and Leslie Kaelbling. 2024. Compositional generative modeling: A single model is not all you need.arXiv preprint arXiv:2402.01103(2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Michael Ferdman, Almutaz Adileh, Onur Kocberber, Stavros Volos, Mohammad Alisafaee, Djordje Jevdjic, Cansu Kaynak, Adrian Daniel Popescu, Anastasia Ailamaki, and Babak Falsafi. 2012. Clearing the clouds: a study of emerging scale-out workloads on modern hardware.ACM SIGPLAN Notices47, 4 (2012), 37–48

    2012

  11. [11]

    Nathan Gober, Gino Chacon, Lei Wang, Paul V Gratz, Daniel A Jimenez, Elvira Teran, Seth Pugsley, and Jinchun Kim. 2022. The championship sim- ulator: Architectural simulation for education and competition.arXiv preprint arXiv:2210.14324(2022)

    Pith/arXiv arXiv 2022

  12. [12]

    Raghav Gupta, Akanksha Jain, Abraham Gonzalez, Alexander Novikov, Po-Sen Huang, Matej Balog, Marvin Eisenberger, Sergey Shirobokov, Ngân V ˜u, Mar- tin Dixon, et al. 2026. ArchAgent: Agentic AI-driven Computer Architecture Discovery.arXiv preprint arXiv:2602.22425(2026)

    arXiv 2026

  13. [13]

    Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149(2015)

    Pith/arXiv arXiv 2015

  14. [14]

    Milad Hashemi, Kevin Swersky, Jamie Smith, Grant Ayers, Heiner Litz, Jichuan Chang, Christos Kozyrakis, and Parthasarathy Ranganathan. 2018. Learning memory access patterns. InInternational Conference on Machine Learning. PMLR, 1919–1928

    2018

  15. [15]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Resid- ual Learning for Image Recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778

    2016

  16. [16]

    Charles Hong, Qijing Huang, Grace Dinh, Mahesh Subedar, and Yakun Sophia Shao. 2023. DOSA: Differentiable Model-Based One-Loop Search for DNN Ac- celerators. InProceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture. 209–224

    2023

  17. [17]

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2023. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation.arXiv preprint arXiv:2310.03302(2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues?. InInternational Conference on Learning Representations, Vol. 2024. 54107–54157

    2024

  19. [19]

    Matthias Jung, Christian Weis, and Norbert Wehn. 2015. DRAMSys: A flexible DRAM subsystem design space exploration framework.IPSJ Transactions on System and LSI Design Methodology8 (2015), 63–74

    2015

  20. [20]

    Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, and Onur Mutlu. 2014. Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors. ACM SIGARCH Computer Architecture News42, 3 (2014), 361–372

    2014

  21. [21]

    Yoongu Kim, Weikun Yang, and Onur Mutlu. 2015. Ramulator: A fast and extensible DRAM simulator.IEEE Computer architecture letters15, 1 (2015), 45–49

    2015

  22. [22]

    Srivatsan Krishnan, Natasha Jaques, Shayegan Omidshafiei, Dan Zhang, Izzeddin Gur, Vijay Janapa Reddi, and Aleksandra Faust. 2022. Multi-agent reinforce- ment learning for microprocessor design space exploration.arXiv preprint arXiv:2211.16385(2022)

    Pith/arXiv arXiv 2022

  23. [23]

    Srivatsan Krishnan, Zishen Wan, Kshitij Bhardwaj, Paul Whatmough, Aleksandra Faust, Sabrina Neuman, Gu-Yeon Wei, David Brooks, and Vijay Janapa Reddi

  24. [24]

    In2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)

    Automatic domain-specific soc design for autonomous unmanned aerial vehicles. In2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 300–317

  25. [25]

    Srivatsan Krishnan, Amir Yazdanbakhsh, Shvetank Prakash, Jason Jabbour, Ikechukwu Uchendu, Susobhan Ghosh, Behzad Boroujerdian, Daniel Richins, Devashree Tripathy, Aleksandra Faust, et al. 2023. Archgym: An open-source gymnasium for machine learning assisted architecture design. InProceedings of the 50th Annual International Symposium on Computer Architec...

    2023

  26. [26]

    Alex Krizhevsky, Geoffrey Hinton, et al . 2009. Learning Multiple Layers of Features from Tiny Images. (2009)

    2009

  27. [27]

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient- Based Learning Applied to Document Recognition.Proc. IEEE86, 11 (1998), 2278–2324

    1998

  28. [28]

    Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023. VerilogEval: Evaluating Large Language Models for Verilog Code Generation. arXiv:2309.07544 [cs.PL] https://arxiv.org/abs/2309.07544

    Pith/arXiv arXiv 2023

  29. [29]

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. 2025. General-Reasoner: Advancing LLM Reasoning Across All Domains. arXiv:2505.14652 [cs.CL] https://arxiv.org/abs/2505.14652

    Pith/arXiv arXiv 2025

  30. [30]

    1993.Combining Branch Predictors

    Scott McFarling. 1993.Combining Branch Predictors. Technical Report. Technical Report TN-36, Digital Western Research Laboratory

    1993

  31. [31]

    Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin. 2019. Ithemal: Accurate, portable and fast basic block throughput estimation using deep neural networks. InInternational Conference on machine learning. PMLR, 4505–4515

    2019

  32. [32]

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al

  33. [33]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868(2026)

    Pith/arXiv arXiv 2026

  34. [34]

    OpenAI. 2026. An OpenAI model has disproved a central conjecture in dis- crete geometry. https://openai.com/index/model-disproves-discrete-geometry- conjecture/. Accessed: 2026-06-16

    2026

  35. [35]

    Zhang, William Hu, Christo- pher Ré, and Azalia Mirhoseini

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christo- pher Ré, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs Write Efficient GPU Kernels? arXiv:2502.10517 [cs.LG] https://arxiv.org/abs/2502.10517

    Pith/arXiv arXiv 2025

  36. [36]

    Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W Keckler, and Joel Emer. 2019. Timeloop: A systematic approach to dnn accelerator evaluation. In2019 IEEE international symposium on performance analysis of systems and software (ISPASS). IEEE, 304–315

    2019

  37. [37]

    Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rang- harajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W Keckler, and William J Dally. 2017. SCNN: An accelerator for compressed-sparse convo- lutional neural networks.ACM SIGARCH Computer Architecture News45, 2 (2017), 27–40

    2017

  38. [38]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative LLM inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132

    2024

  39. [39]

    Shvetank Prakash, Andrew Cheng, Arya Tschand, Mark Mazumder, Varun Go- hil, Jeffrey Ma, Jason Yik, Zishen Wan, Jessica Quaye, Elisavet Lydia Alvanaki, et al. 2025. Quarch: A benchmark for evaluating llm reasoning in computer architecture.arXiv preprint arXiv:2510.22087(2025)

    arXiv 2025

  40. [40]

    Zhenting Qi, Huangyuan Su, Ao Qu, Chenyu Wang, Yu Yao, Han Zheng, Kushal Chattopadhyay, Guowei Xu, Zihan Wang, Weirui Ye, et al. 2026. Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactions.arXiv preprint arXiv:2606.02859(2026)

    Pith/arXiv arXiv 2026

  41. [41]

    Ritik Raj, Sarbartha Banerjee, Nikhil Chandra, Zishen Wan, Jianming Tong, Ananda Samajdhar, and Tushar Krishna. 2025. SCALE-Sim v3: A modular cycle- accurate systolic accelerator simulator for end-to-end system analysis. In2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 186–200

    2025

  42. [42]

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al . 2020. MLPerf inference benchmark. In2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 446–459

    2020

  43. [43]

    Hanbo Sun, Chenyu Wang, Zhenhua Zhu, Xuefei Ning, Guohao Dai, Huazhong Yang, and Yu Wang. 2022. Gibbon: Efficient Co-Exploration of NN Model and Processing-in-Memory Architecture. In2022 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 867–872

    2022

  44. [44]

    Rajeev Thakur, Rolf Rabenseifner, and William Gropp. 2005. Optimization of Collective Communication Operations in MPICH.The International Journal of High Performance Computing Applications19, 1 (2005), 49–66

    2005

  45. [45]

    Shailja Thakur, Jason Blocklove, Hammond Pearce, Benjamin Tan, Siddharth Garg, and Ramesh Karri. 2023. AutoChip: Automating HDL Generation Using LLM Feedback. arXiv:2311.04887 [cs.AR] https://arxiv.org/abs/2311.04887 19 Wang, Wan, et al

    Pith/arXiv arXiv 2023

  46. [46]

    Arya Tschand, Chenyu Wang, Zishen Wan, Andrew Cheng, Ioana Cristescu, Kevin He, Howard Huang, Alexander Ingare, Akseli Kangaslahti, Sara Kan- gaslahti, et al . 2026. GenAI for Systems: Recurring Challenges and Design Principles from Software to Silicon.arXiv preprint arXiv:2602.15241(2026)

    arXiv 2026

  47. [47]

    Boning, Zishen Wan, Yilun Du, and Vijay Janapa Reddi

    Chenyu Wang, Jiahe Caroline Shi, David Kong, Duane S. Boning, Zishen Wan, Yilun Du, and Vijay Janapa Reddi. 2026. AgentDSE: Reasoning-Augmented Architectural Design Space Exploration. arXiv:2606.21836 [cs.AR] https://arxiv. org/abs/2606.21836

    Pith/arXiv arXiv 2026

  48. [48]

    Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie, Tushar Krishna, Vijay Janapa Reddi, and Yilun Du. 2025. Slm-mux: Orchestrating small language models for reasoning.arXiv preprint arXiv:2510.05077(2025)

    arXiv 2025

  49. [49]

    Hanyu Wang, Xinrui Wu, Zijian Ding, Su Zheng, Chengyue Wang, Neha Prakriya, Tony Nowatzki, Yizhou Sun, and Jason Cong. 2025. LLM-DSE: Searching Accel- erator Parameters with LLM Agents.arXiv preprint arXiv:2505.12188(2025)

    arXiv 2025

  50. [50]

    William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2023. Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale. In2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 283–294

    2023

  51. [51]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer In- terfaces Enable Automated Software Engineering. arXiv:2405.15793 [cs.SE] https://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  52. [52]

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. 2026. ProgramBench: Can Language Models Rebuild Programs From Scratch?arXiv preprint arXiv:2605.03546(2026)

    Pith/arXiv arXiv 2026

  53. [53]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629(2022)

    Pith/arXiv arXiv 2022

  54. [54]

    Amir Yazdanbakhsh, Christof Angermueller, Berkin Akin, Yanqi Zhou, Albin Jones, Milad Hashemi, Kevin Swersky, Satrajit Chatterjee, Ravi Narayanaswami, and James Laudon. 2021. Apollo: Transferable architecture exploration.arXiv preprint arXiv:2102.01723(2021)

    Pith/arXiv arXiv 2021

  55. [55]

    Marilyn Zhang, Tianfeng Chen, Fabián Barzuna, Ankita Rathod, and Mark E. Whiting. 2026. LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design. arXiv:2605.15341 [cs.LG] https://arxiv.org/abs/2605.15341

    Pith/arXiv arXiv 2026

  56. [56]

    Zhenhua Zhu, Hanbo Sun, Tongxin Xie, Yu Zhu, Guohao Dai, Lixue Xia, Dimin Niu, Xiaoming Chen, Xiaobo Sharon Hu, Yu Cao, et al . 2023. Mnsim 2.0: A behavior-level modeling tool for processing-in-memory architectures.IEEE transactions on computer-aided design of integrated circuits and systems42, 11 (2023), 4112–4125. Appendix A The Challenge Suite Section ...

    2023

  57. [57]

    Workload:3 SPEC CPU2017 traces [6] (perlbench, gcc, xalancbmk); 3M-instr warmup + 7M-instr measure

    Branch Direction Predictorbranch_predictor(CPU core, ChampSim) Objective:Implement thecandidatetaken/not-taken predictor class (BTB and pipeline fixed) to minimize average MPKI, following classic branch-prediction design practice [29]. Workload:3 SPEC CPU2017 traces [6] (perlbench, gcc, xalancbmk); 3M-instr warmup + 7M-instr measure. Metric:MPKI ↓ vs stoc...

  58. [58]

    Workload:3 SPEC CPU2017 traces (perlbench, gcc, xalancbmk); 3M + 7M instr

    Branch Target Bufferbtb(CPU core, ChampSim) Objective:Design thecandidateBTB minimizing 3-trace average MPKI under a tight metadata budget. Workload:3 SPEC CPU2017 traces (perlbench, gcc, xalancbmk); 3M + 7M instr. Metric:MPKI ↓ vs stock basic_btb.Constraint: ≤16 KB metadata; code ≤1500 lines. Deliverables:candidate.{h,cc}; three design docs; pytest surro...

  59. [59]

    Workload:6 SPEC CPU2017 traces (sphinx3, mcf×2, omnetpp,

    Cache Replacement Policycache_replacement(CPU core, ChampSim) Objective:Design an LLC replacement policy maximizing cycle-weighted IPC (geomean speedup) over six workloads. Workload:6 SPEC CPU2017 traces (sphinx3, mcf×2, omnetpp,. . .); 3M + 7M instr. Metric:IPC↑vs LRU.Constraint:≤256 B total replacement-policy metadata across the whole LLC; code≤1000 lin...

  60. [60]

    Workload:3 memory-intensive SPEC traces (mcf, libquantum, mcf); 3M + 7M instr

    L1D Prefetcherl1d_prefetcher(CPU core, ChampSim) Objective:Design the candidate L1D prefetcher maximizing average IPC on memory-intensive traces. Workload:3 memory-intensive SPEC traces (mcf, libquantum, mcf); 3M + 7M instr. Metric:IPC↑vs next-line L1D prefetcher.Constraint:no storage cap; code≤1500 lines. Deliverables:candidate.{h,cc}; three design docs;...

  61. [61]

    Workload:3 SPEC CPU2017 traces (perlbench, gcc, xalancbmk); 3M + 7M instr

    Front-End Co-Design: BP + BTBcompose_bp_btb(CPU core, ChampSim) Objective:Co-design a branch predictor and a BTB that jointly minimize average MPKI under a shared storage budget. Workload:3 SPEC CPU2017 traces (perlbench, gcc, xalancbmk); 3M + 7M instr. Metric:MPKI ↓ vs reference BP+BTB.Constraint:joint ≤32 KB storage; code≤2500 lines. Deliverables:candid...

  62. [62]

    Workload:3 SPEC CPU2017 traces (mcf×2, omnetpp); 3M + 7M instr

    LLC Co-Design: Replacement + Prefetcher compose_replacement_prefetcher(CPU core, ChampSim) Objective:Co-design the LLC replacement policy and LLC prefetcher under a single shared storage budget to maximize IPC. Workload:3 SPEC CPU2017 traces (mcf×2, omnetpp); 3M + 7M instr. Metric:IPC ↑ vs reference repl.+pf..Constraint:shared ≤64 KB storage; code≤2000 li...

    2000

  63. [63]

    Workload:Single-CPU gem5 SE-mode workload

    Cache Hierarchy + Prefetchergem5_cache(System, gem5) Objective:Design the full on-die cache subsystem (L1I/L1D, optional L2/LLC) plus prefetcher for a single-CPU SE-mode system as a config.py. Workload:Single-CPU gem5 SE-mode workload. Metric:IPC ↑ vs reference config.Constraint:on-die cache area budget; code≤400 lines. Deliverables:config.py; workload/pr...

  64. [64]

    Workload:Fixed DDR4 access trace

    DDR4 Memory Controller Tuningdramsys_ddr4(Memory, DRAMSys) Objective:Design a DDR4 controller configuration maximizing sustained read+write bandwidth, subject to a latency deadline. Workload:Fixed DDR4 access trace. Metric:GB/s↑vs reference config.Constraint:latency deadline; code ≤200 lines. Deliverables:config.json, mc_config.json; three design docs. Ve...

  65. [65]

    Workload:Fixed memory access trace

    RowHammer Mitigation Overheadramulator_rowhammer (Memory, Ramulator 2.0) Objective:Pick and tune a RowHammer mitigation [20] for least performance overhead while staying provably secure at a fixed threshold. Workload:Fixed memory access trace. Metric:overhead %↓vs reference mitigation.Constraint:provably secure at𝑇 RH=100; code≤120 lines. Deliverables:con...

  66. [66]

    Workload:All-Reduce, 8 NPUs, fixed topology + workload

    Collective: All-Reduceastrasim_collective(System, ASTRA-sim) Objective:Configure ASTRA-sim’s System layer (collective algorithm + parameters) to minimize completion time on fixed hardware, following collective-communication optimization practice [42]. Workload:All-Reduce, 8 NPUs, fixed topology + workload. Metric:cycles↓vs reference algorithm+config.Const...

  67. [67]

    Workload:All-Gather, 8 NPUs, fixed topology + workload

    Collective: All-Gatherastrasim_allgather(System, ASTRA-sim) Objective:Configure the System layer to minimize All-Gather completion time on fixed hardware [42]. Workload:All-Gather, 8 NPUs, fixed topology + workload. Metric:cycles↓vs reference algorithm+config.Constraint:8 NPUs, fixed topology; code≤60 lines. Deliverables:system.json; three design docs. Ve...

  68. [68]

    Workload:All-to-All, 8 NPUs, fixed topology + workload

    Collective: All-to-All (8 NPUs) astrasim_alltoall (System, ASTRA-sim) Objective:Configure the System layer to minimize All-to-All completion time on fixed hardware [42]. Workload:All-to-All, 8 NPUs, fixed topology + workload. Metric:cycles↓vs reference algorithm+config.Constraint:8 NPUs, fixed topology; code≤60 lines. Deliverables:system.json; three desig...

  69. [69]

    Workload:All-to-All, 16 NPUs, fixed topology + workload

    Collective: All-to-All (16 NPUs)astrasim_alltoall_16 (System, ASTRA-sim) Objective:Configure the System layer to minimize All-to-All completion time at larger scale [42]. Workload:All-to-All, 16 NPUs, fixed topology + workload. Metric:cycles↓vs reference algorithm+config.Constraint:16 NPUs, fixed topology; code≤60 lines. Deliverables:system.json; three de...

  70. [70]

    Workload:Reduce-Scatter, 8 NPUs, fixed topology + workload

    Collective: Reduce-Scatterastrasim_reducescatter (System, ASTRA-sim) Objective:Configure the System layer to minimize Reduce-Scatter completion time on fixed hardware [42]. Workload:Reduce-Scatter, 8 NPUs, fixed topology + workload. Metric:cycles↓vs reference algorithm+config.Constraint:8 NPUs, fixed topology; code≤60 lines. Deliverables:system.json; thre...

  71. [71]

    Workload:ViT-Base [8] (14 layers); 4 cores, each 128×128 PE array, 256 KB SRAMs, BW=10

    Multi-Core Partitioning (128 ×128) scalesim_mc (Accelerator, SCALE-Sim v3) Objective:Design a per-layer partitioning policy for a 4-core systolic-array accelerator running ViT-Base [8]. Workload:ViT-Base [8] (14 layers); 4 cores, each 128×128 PE array, 256 KB SRAMs, BW=10. Metric:cycles↓vs reference schedule.Constraint:fixed 4×128×128 hardware; code≤2000 ...

    2000

  72. [72]

    Workload:ViT-Base [8] (14 layers); 4 cores, each 256×256 PE array, 256 KB SRAMs, BW=10

    Multi-Core Partitioning (256×256)scalesim_256array (Accelerator, SCALE-Sim v3) Objective:Design a per-layer partitioning policy for a 4-core accelerator with larger arrays. Workload:ViT-Base [8] (14 layers); 4 cores, each 256×256 PE array, 256 KB SRAMs, BW=10. Metric:cycles↓vs reference schedule.Constraint:fixed 4×256×256 hardware; code≤2000 lines. Delive...

    2000

  73. [73]

    Workload:ViT-Base [8] (14 layers); 4 cores, each 64×64 PE array, 256 KB SRAMs, BW=10

    Multi-Core Partitioning (64×64)scalesim_64array (Accelerator, SCALE-Sim v3) Objective:Design a per-layer partitioning policy for a 4-core accelerator with smaller arrays. Workload:ViT-Base [8] (14 layers); 4 cores, each 64×64 PE array, 256 KB SRAMs, BW=10. Metric:cycles↓vs reference schedule.Constraint:fixed 4×64×64 hardware; code≤2000 lines. Deliverables...

    2000

  74. [74]

    Workload:ResNet-50 [15] (24 unique layers) on one fixed DNN accelerator

    Per-Layer Mapping (DOSA [ 16]) timeloop_dosa (Accelerator, Timeloop+Accelergy) Objective:Choose the Timeloop mapping (loop order, tiling, spatial parallelism) for every ResNet-50 layer [15], minimizing geomean EDP. Workload:ResNet-50 [15] (24 unique layers) on one fixed DNN accelerator. Metric:geomean EDP↓vs reference mapping.Constraint:fixed Gemmini-styl...

  75. [75]

    Workload:MNSIM ReRAM CIM design point

    ReRAM CIM Design Searchmnsim_pim(CIM, MNSIM 2.0) Objective:Search MNSIM’s device / crossbar / interface knobs for one ReRAM CIM accelerator (SimConfig.ini) minimizing EDP under an area cap. Workload:MNSIM ReRAM CIM design point. Metric:EDP↓vs reference design.Constraint:area cap; code≤2000 lines. Deliverables:SimConfig.ini; design_analysis, principles, ev...

    2000

  76. [76]

    We have successfully designed

    NN + CIM Co-Design (Gibbon)gibbon_codesign(CIM, MNSIM 2.0) Objective:Co-design the network and the ReRAM-CIM accelerator (design.json) minimizing EDP while meeting an accuracy floor and area cap. Workload:CIFAR-10 [25] network + ReRAM-CIM accelerator. Metric:EDP↓vs Gibbon (published).Constraint:CIFAR-10 [25] accuracy floor 0.738, area cap; code≤2000 lines...

    2000

  77. [77]

    has_member

    Validate the storage budget. 4. Submit and evaluate. 5. Iterate based on results.↩→ [...] Budget check for Gshare: Table: 2^14 entries * 2 bits = 4096 bytes. [...]↩→ 2^16 entries * 2 bits = 16384 bytes. (Exactly the budget). [...] The first submission (Gshare with 15-bit GHR) yielded an average MPKI of 6.37.↩→ Per-trace MPKI: 600.perlbench 2.523, 602.gcc ...