Artificial Intelligence · arXiv:2611.16991 v1 · submitted 12 Nov 2026 · 19 references
AnalogLayoutWorld: An Open Benchmark for Agentic Analog Layout Design
The paper says
“Coding agents have demonstrated strong capabilities in software development, yet their ability to tackle long-horizon engineering tasks governed by physical constraints remains less understood.”
more of the abstract
We study this question through analog circuit layout, which demands domain knowledge, geometric reasoning, and iterative interaction with design and verification tools. We introduce \textbfAnalogLayoutWorld, an open-source agent benchmark with 40 expert-reviewed tasks for end-to-end layout generation from fixed transistor-level netlists. It combines an integrated design environment with independent physical and post-layout electrical verification, as well as objective layout quality metrics co-designed with analog layout experts. Across 13 model–harness configurations, 91.1% of attempts produce GDS layouts, but only 23.0% satisfy the benchmark's acceptance criteria. We also evaluate Novara, an internally developed agent system, which achieves the highest reported aggregate score. Our analysis links performance differences to test-time scaling under fixed budgets, suggesting that reasoning efficiency and effective use of verification feedback help sustain design quality as task complexity grows.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
A solid, well-executed benchmark for analog layout agents; the engineering quality metric is the main soft spot, but the paper acknowledges it and the central claim holds up. This line is wrong
the referee pass did not run
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 19 references say.
What would settle it
Recompute the aggregate scores for all 13 configurations under at least three alternative parameter sets: (1) equal weights for matching, routing, and power delivery (α_j = 0.333 each); (2) saturation targets τ_j reduced to 0.60 (more forgiving); (3) engineering quality weight ω_Q doubled to 0.20 with corresponding reduction in electrical weight ω_S to 0.30. Report Spearman rank correlations between the original and perturbed leaderboards. If any correlation falls below 0.90, the ranking is not robust to these plausible variations.
from the skeptic's reading
How the review went
3 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up
- Skeptic partly agrees: says it holds up only with conditions Engineering quality metric Q_t is a composite of layout proxies whose weights and saturation targets are set by expert judgment without sensitivity analysis; the benchmark's ranking of agents could shift under plausible alternative parameterizations.
- Editor wrote the verdict above
Not run on this paper: the referee, the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a serious piece of work. The benchmark is well-designed: 40 expert-reviewed tasks, a fully open-source EDA stack, independent grading in a fresh directory, three trials per task, clear acceptance criteria. The main result—91% produce GDS but only 21% meet acceptance—is cleanly supported and not surprising given the difficulty of the task. The analysis of test-time scaling and verification feedback is thoughtful and goes beyond a simple leaderboard. The case studies (ring VCO, LDO via capacity) are concrete and illustrate real failure modes. The authors are upfront about limitations: no silicon, metrics are proxies, SKY130 only. That honesty earns trust.
The softest spot is the engineering quality score Qt. The weights and saturation targets are set by expert judgment with no sensitivity analysis. The stress-test note is right that the ranking could shift under plausible alternative parameterizations. But the paper acknowledges this (Appendix B: 'may have incomplete coverage or assign low scores to otherwise sound layouts'), and Qt is only 10% of the total score. The compactness term (20%) and electrical term (40%) dominate. So the concern is real but not load-bearing for the main claim about the generation-vs-verification gap. It matters more if someone uses the benchmark to fine-tune models on the quality score specifically.
The other soft spot is that the benchmark is SKY130-only with 40 tasks. That's fine for a first release, but generalizability to other PDKs and larger circuits is unproven. The authors flag this.
Who is this for? Researchers building agents for physical design tasks. It fills a real gap—existing benchmarks focus on netlist design or digital RTL. The paper deserves peer review. The methodology is sound, the data is reproducible (code promised), and the findings are useful even if the quality metric needs refinement.
Recommendation: accept with minor revision. Ask for a sensitivity analysis on the Qt parameters, or at least a discussion of how rankings change under alternative weights. That would address the main concern without requiring new experiments.
The strongest objection
The central claim that AnalogLayoutWorld provides a useful benchmark for ranking AI agents on analog layout quality rests heavily on the engineering quality score Q_t (Eq. 4). Q_t aggregates three categories (matching, critical routing, power delivery) with fixed weights α_j = (0.40, 0.45, 0.15) and saturation targets τ_j = (0.80, 0.80, 0.90). These parameters were set by the authors and their expert consultants, but the paper reports no sensitivity analysis. The authors themselves acknowledge (Appendix B) that 'these metrics may have incomplete coverage or assign low scores to otherwise sound layouts.' Because Q_t contributes 10% of the total score (ω_Q = 0.10), and because the compactness term (ω_A = 0.20) and electrical term (ω_S = 0.40) also depend on metric choices (ρ*_t, q_max_t, the electrical manifest), the overall ranking is sensitive to these parameter values. For example, if the power-delivery weight were increased to reflect real-world reliability concerns, the gap between GPT-6 Astra and DeepSeek V4.1 Flash on the LDO task (Table 2) would widen, but other tasks might see rank reversals. Without a perturbation analysis, the robustness of the reported leaderboard to reasonable alternative parameter choices is unknown. This is a load-bearing concern because the benchmark's utility for guiding model development depends on the stability of its rankings under metric variations that experts could reasonably disagree on.
The reader's weakest_assumption correctly identifies that the evaluation metrics may not correlate with real-world layout quality, and that sensitivity to parameter choices is unexplored. This is indeed the most load-bearing concern. However, the reader's verdict is ACCEPT with HIGH confidence, which I find too strong given this unresolved sensitivity. The paper is a solid contribution, but the central claim about the benchmark's usefulness for ranking agents is conditional on the metrics being meaningful and stable. The authors acknowledge the limitation, but they do not provide evidence that the rankings are robust to parameter variations. A CONDITIONAL verdict is appropriate: accept the paper contingent on the authors either (a) performing and reporting the sensitivity analysis, or (b) releasing the benchmark so the community can independently assess robustness. The concrete test I propose directly addresses this gap. I agree with the reader that the paper is otherwise well-designed and the limitations are honestly discussed, hence 'partial' agreement.
Pith's reading of the paper (a machine's summary, not the authors')
The paper introduces AnalogLayoutWorld, an open-source benchmark for evaluating AI agents on analog circuit layout design. It provides 40 expert-reviewed tasks with a complete open-source EDA toolchain and independent verification of physical, electrical, and layout quality. Evaluating 13 model-harness configurations, the authors find that while 91.1% of attempts produce a GDS layout file, only 21.3% satisfy all acceptance criteria. They identify test-time scaling under fixed budgets as a key factor: agents that efficiently allocate resources and effectively use verification feedback maintain design quality as circuit complexity grows. The paper argues that AnalogLayoutWorld enables reproducible, objective comparison of agent capabilities in a long-horizon engineering domain governed by physical constraints.
What carries the argument. The scoring function S_t(L) that aggregates submission correctness, physical verification (DRC, LVS), compactness (effective device-area ratio ρ_t), engineering quality Q_t (matching, critical routing, power delivery), and electrical verification F_t (post-layout simulation pass fraction). This function, with engineer-specified weights and parameters, defines the benchmark's acceptance criteria and enables objective comparison across agents.
Core claim. The paper establishes that AnalogLayoutWorld provides a reproducible, objective benchmark for evaluating AI agents on analog circuit layout design from fixed netlists. The central empirical finding is a large gap between artifact generation (91.1% produce GDS) and verified completion (21.3% meet all acceptance criteria). The authors discover that performance differences are linked to test-time scaling under fixed budgets: agents that efficiently allocate resources and effectively use verification feedback maintain design quality as circuit complexity grows, while those that saturate their budget on smaller tasks degrade more sharply on larger circuits.
The premise it rests on. The benchmark's evaluation metrics, particularly the engineering quality score Q_t and the geometric/morphological constraints, are meaningful proxies for practical layout quality, but the authors acknowledge these metrics may have incomplete coverage or assign low scores to otherwise sound layouts.
If the paper is right (Pith's inferences, not the authors')
AnalogLayoutWorld provides a standardized testbed for comparing AI agents on analog layout design, enabling reproducible research and progress tracking.
The substantial gap between GDS generation and verified completion suggests that current agents lack robust geometric reasoning and iterative refinement capabilities for physical design tasks.
The correlation between test-time scaling efficiency and performance indicates that agent architectures should prioritize reasoning efficiency and effective use of verification feedback.
The benchmark's open-source toolchain and independent verification protocol can be extended to other physical design domains (e.g., digital layout, PCB design).
The detailed scoring decomposition (physical, electrical, compactness, engineering quality) allows targeted diagnosis of agent weaknesses.
The benchmark's metrics, particularly engineering quality Q_t and compactness ρ_t, are proxies for practical layout quality; their correlation with real-world manufacturing outcomes remains unvalidated without silicon measurements.
The three-hour fixed budget may disadvantage agents that trade speed for thoroughness; alternative budget models (e.g., variable time per device count) could reveal different agent rankings.
The focus on fixed-netlist tasks excludes topology selection and sizing, which are critical parts of analog design; extending the benchmark to include these could significantly change agent difficulty and strategy.
The SKY130 PDK limits generality; results may not transfer to other process nodes with different design rules and parasitic characteristics.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Coding agents have demonstrated strong capabilities in software development, yet their ability to tackle long-horizon engineering tasks governed by physical constraints remains less understood. We study this question through analog circuit layout, which demands domain knowledge, geometric reasoning, and iterative interaction with design and verification tools. We introduce \textbfAnalogLayoutWorld, an open-source agent benchmark with 40 expert-reviewed tasks for end-to-end layout generation from fixed transistor-level netlists. It combines an integrated design environment with independent physical and post-layout electrical verification, as well as objective layout quality metrics co-designed with analog layout experts. Across 13 model–harness configurations, 91.1% of attempts produce GDS layouts, but only 23.0% satisfy the benchmark's acceptance criteria. We also evaluate Novara, an internally developed agent system, which achieves the highest reported aggregate score. Our analysis links performance differences to test-time scaling under fixed budgets, suggesting that reasoning efficiency and effective use of verification feedback help sustain design quality as task complexity grows.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.99
“Our analysis links performance differences to test-time scaling under fixed budgets, suggesting that reasoning efficiency and effective use of verification feedback help sustain design quality as task complexity grows.”
How the main claim is stated: hedged possibility 0.96
What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.74 · human participants 0.19 · controlled comparison 0.21 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.81 · mentions code or data availability 0.83
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Coding agents have demonstrated strong capabilities in software development, yet their ability to tackle long-horizon engineering tasks governed by physical constraints remains less understood. | gap 0.74 | 0.07 | 0.03 | asserted 0.98 |
| We study this question through analog circuit layout, which demands domain knowledge, geometric reasoning, and iterative interaction with design and verification tools. | background 0.41 | 0.37 | 0.03 | asserted 0.91 |
| We introduce \textbf{AnalogLayoutWorld}, an open-source agent benchmark with 40 expert-reviewed tasks for end-to-end layout generation from fixed transistor-level netlists. | contribution 0.99 | 0.94 | 0.06 | asserted 1.00 |
| It combines an integrated design environment with independent physical and post-layout electrical verification, as well as objective layout quality metrics co-designed with analog layout experts. | method 0.73 | 0.86 | 0.04 | asserted 1.00 |
| Across 13 model--harness configurations, 91.1\% of attempts produce GDS layouts, but only 23.0\% satisfy the benchmark's acceptance criteria. | result 1.00 | 0.95 | 0.20 | asserted 1.00 |
| We also evaluate Novara, an internally developed agent system, which achieves the highest reported aggregate score. | result 0.85 | 0.86 | 0.06 | asserted 1.00 |
| Our analysis links performance differences to test-time scaling under fixed budgets, suggesting that reasoning efficiency and effective use of verification feedback help sustain design quality as task complexity grows. | interpretation 1.00 | 0.94 | 0.72 | hedged possibility 0.96 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.93
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 5,600 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (10)
Figures are the authors', reproduced from the paper.
References (19)
- [1] Analog Design Bench Team. Long-horizon analog design bench: Benchmarking agents on hours- long analog and mixed-signal circuit design tasks. arXiv:2609.33356v1, 2026.https://arxiv. org/abs/2609.33356v1. Anomaly. OpenCode.https://github.com/anomalyco/opencode, n.d. Accessed Nov... on PithPith's reviewarXiv
- [2] The area-score contributions are consequently 0.2000 and 0.0108, accounting for99.4%of the total-score difference (0.9617 versus 0.7712). With nearly unchanged effective device content, this pair makes the score’s preference for compact physical organization directly inspectab... on Pith
- [4] URLhttps://arxiv. org/abs/2602.15763. 11 Surya Gunukula, Jerry Yicheng Wang, and Kevin Zhang. Introducing Build-A-Chip-Bench: From workload specification to GDS. ChipAgents Research Blog, 2026.https://chipagents.ai/ research/build-a-chip-bench, accessed November 3, on PithPith's reviewarXiv
- [5] Ali Hammoud, Chetanya Goyal, Sakib Pathen, Arlene Dai, Anhang Li, Gregory Kielian, and Mehdi Saligane. Human language to analog layout using GLayout layout automation framework. In Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, on Pith
- [7] Intern-S1 Team. Intern-S1: A Scientific Multimodal Foundation Model, 2025.https://arxiv. org/abs/2508.15763. Goeran Jerke and Andrew B. Kahng. Mission Profile Aware IC Design—A Case Study. InPro- ceedings of the Design, Automation & Test in Europe Conference & Exhibition, pages 1–6, on PitharXiv
- [9] Conference paper. Marty Johnson, Roger Cline, Scott Ward, and Joe Schichl. Latch-Up. Technical Report SCAA124, Texas Instruments, April 2015.https://www.ti.com/lit/wp/scaa124/scaa124.pdf. KiCad Documentation Contributors. KiCad schematic editor reference manual, version 9.0. O... on Pith
- [10] Markus Leibl, Filipe Azevedo, Stefan Uhlich, Andrea Bonetti, Arun Venkitaraman, Chia-Yu Hsieh, Phuoc Pham, Lorenzo Servadei, Helmut Graeb, and Ricardo Martins. LOADBench: A large open analog dataset and benchmark for machine learning in IC design. InProceedings of the 2026 ACM... on Pith
- [11] doi: 10.1145/ 3831599.3840321.https://arxiv.org/abs/2610.21591v1. Linyang Li. Agent-Orchestration in Autonomous Chip Design. arXiv:2608.14035, 2026.https: //arxiv.org/abs/2608.14035. Peiji Li, Linyang Li, Handa Sun, Wenjin Mai, Yongkang Chen, Xiaozhe Li, Yue Shen, Yichuan Ma, ... on PithPith's reviewarXiv
- [14] Ryan Marten. Terminal-Bench 4.0. Official benchmark release, 2026.https://www.tbench.ai/ news/terminal-bench-4-0, accessed November 3, on Pith
- [15] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026.https://arxiv.org/abs/2601.11868. Moonshot AI. Kimi K2.6: Advancing open-source coding. O... on PithPith's reviewarXiv
- [16] Vishal Saxena and Soumyajit Mandal. OPERA: OPamp design by an expert-taught reasoning agent, from specifications to layout. arXiv preprint arXiv:2610.39395, 2026.https://arxiv.org/ abs/2610.39395. Simra AI. TapeoutBench v0.1: Coding agents on thirteen steps of a chip tapeout. ... on PithPith's reviewarXiv
- [17] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for op... on PithPith's reviewarXiv
- [2014] doi: 10.7873/DATE.2014.077.https://vlsicad.ucsd.edu/Publications/Conferences/ 312/c312.pdf. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInterna- ti... on Pithdoi:10.7873/date.2014.077.https://vlsicad.ucsd.edu/publications/conferences/
- [2019] doi: 10.1109/ICCAD45719.2019.8942060. Invited paper. Zecheng Xu, Guanhua Chen, Yumao Wu, and Keren Zhu. LAMP: LLM-driven analog placement through adaptive tool composition. arXiv preprint arXiv:2610.39882, 2026.https://arxiv. org/abs/2610.39882. Kentaro Yoshioka. The era of vi... on PithPith's reviewarXiv
- [2021] doi: 10.1109/MDAT.2020.3042177. Preprint: https://arxiv.org/abs/2008.10682. Docker, Inc. Docker Documentation: What Is Docker? Official documentation, n.d. Accessed November 3, on PitharXiv
- [2023] doi: 10.1109/ICCAD57390.2023.10323812. https://arxiv.org/abs/2309.07544. 12 Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun ... on PithPith's reviewarXiv
- [2024] doi: 10.1145/3670474.3685971.https://openreview.net/pdf?id=4ZzkZkequO. Harbor Contributors. Harbor Documentation.https://docs.harborframework.com/, n.d. Ac- cessed October 20, on Pithdoi:10.1145/3670474.3685971.https://openreview.net/pdf
- [2025] doi: 10.1109/TCAD.2025.3529805. Bingyang Liu, Ziming Wei, Xiaohan Gao, and David Z. Pan. Simulation-aware in-context policy improvement for LLM-aided analog layout refinement. arXiv:2608.13767, 2026.https:// arxiv.org/abs/2608.13767. Mingjie Liu, Nathaniel Pinckney, Brucek Kha... on PithPith's reviewarXiv
- [2026] Minxing Chu, Anhang Li, Martins Atilla, Ruichen Qi, and Mehdi Saligane. The code behind the layout: AI agents for analog layout generators. arXiv preprint arXiv:2610.38426, 2026.https: //arxiv.org/abs/2610.38426. DeepSeek-AI. DeepSeek Harness: Everything is a Plugin.https://gi... on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment