REVIEW 4 major objections 4 minor 29 references
AnalogTester: A Large Language Model-Based Framework for Automatic Testbench Generation in Analog Circuit Design
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An LLM-based framework generates working analog testbench code from research paper descriptions, with all pipeline stages reporting success above 80 percent on a 63-experiment benchmark.
desk verdict A useful engineering paper on LLM-generated analog testbenches whose headline accuracy is real only under perfect upstream inputs and against author-built golden models; worth refereeing despite that. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a three-stage LLM agent pipeline paired with two curated knowledge bases and a verification loop. The first knowledge base is a repository of 36 experimental templates, each specifying PVT conditions, testbench architecture, simulation type, and computation method for one performance metric. The second is a hand-curated database of 72 TED functions—TED being a Python-based analog design environment—that supplies the LLM with the function names, arguments, and usage patterns needed to write valid testbench code. The verification loop runs each generated testbench against a golden Verilog-A model of the circuit, compares the computed metric to the model's expected value, and sends execution errors back to the LLM for repair, repeating until the result matches. The same loop, extended by jointly generating new Verilog-A models and TED code, is what the paper uses to scale the framework to new circuit types.
What would settle it
Run the generated testbenches for all eight benchmark papers against transistor-level simulations of the actual circuits and compare the computed metrics; if a bandgap-reference or low-dropout testbench that passed the golden-model check reports wrong values at the transistor level, the claimed success rates measure self-consistency rather than correct measurement.
Extended reading notes
Core claim
The paper's central claim is that analog testbench generation, previously done manually by experienced designers, can be decomposed into three LLM-driven tasks and made reliable through curated knowledge and iterative repair. AnalogTester first extracts simulation conditions, targets, and cited figures and tables from a paper; the extraction agent then checks its own output against a spec list and refines it. A second agent matches each extracted experiment to one of 36 templates covering the standard performance metrics for op-amps, bandgap references, and LDOs, adapting the template with the paper's specific conditions. A third agent writes TED testbench code using a hand-built database of 72 TED functions, executes the code against a golden Verilog-A model of the device under test, and feeds syntax and functional errors back to the LLM until the results match the golden model. Across a benchmark of eight papers and 63 experiments, the authors report success rates of 84 percent for information extraction, 86 percent for scheme generation, and 89 percent for testbench code generation, with an average of roughly two minutes per experiment; for one example op-amp, the generated testbenches reproduced all 11 metrics obtained with manually constructed testbenches in a commercial EDA tool, and the framework can generate new templates and Verilog-A models for circuits outside the initial three types.
Load-bearing premise
The results stand on the assumption that the authors' golden Verilog-A models faithfully represent the real circuits from the benchmark papers, because those models are the only reference against which every generated testbench is judged.
Editorial extensions
If this is right
- A paper's testbench suite can be generated automatically in under an hour, so researchers can replicate published op-amp, bandgap-reference, and LDO designs without manual verification setup.
- The generated testbenches can feed a closed-loop design-automation flow, allowing schematic generation, simulation, and optimization to run without human-written testbenches.
- The TED function database and the generated code corpus become a training set for LLMs specialized in analog design, reducing the data scarcity that limits current models.
- The paper's expansion loop generates a new experimental template and a matching golden Verilog-A model, then verifies the pair through the mutual TED-and-Verilog-A check.
Reading between the lines
- A natural next experiment would run every generated testbench against transistor-level simulations of the eight benchmark circuits; the paper only cross-checks the op-amp against a commercial EDA tool, so the functional success rates for bandgap references and LDOs are not yet validated against real transistor behavior.
- Because the evaluation delivers manually verified upstream inputs to each stage, the per-stage success rates are upper bounds on what an unattended end-to-end run would achieve; an integrated experiment without human aid would give the user-facing accuracy.
- The expansion loop still passes through human validation before a new template enters the repository, so the claimed scalability depends on expert labor until that validation step is itself automated or replaced by a stronger reference model.
- If the golden Verilog-A models were constructed using the same metric definitions and simulation assumptions as the template database, the functional checks mainly demonstrate self-consistency; correlating generated results against measured silicon would test whether the testbenches measure the real circuits.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. AnalogTester is a multi-agent LLM framework that takes a research paper as input and produces TED testbench code for analog circuits. The pipeline has three stages: information extraction from text, tables, and figures; experimental scheme generation from a 36-template database with retrieval-augmented generation; and TED code generation from a 72-function database with iterative syntax and functional repair. The evaluation covers 8 papers (3 op-amps, 3 BGRs, 2 LDOs) and 63 experiments, reporting 84%, 86%, and 89% success rates for extraction, scheme generation, and code generation, plus ablations showing that RAG, the TED function database, and iterative repair improve performance. The authors further claim that a full paper can be processed in under an hour and that the framework can generate training data for analog-EDA LLMs.
Significance. If the headline numbers were measured against trustworthy ground truth, the paper would be a useful contribution: automatic analog testbench generation is a real bottleneck, and the task decomposition plus knowledge repositories are sensible engineering choices. The paper's strengths include a concrete 63-experiment benchmark across three circuit families, a modular architecture that is easy to extend, and an honest reporting of several evaluation constraints. The main weakness is that the reference used for functional verification consists of author-built Verilog-A models covering the same metrics as the templates, with only one OTA cross-checked against Cadence Virtuoso, and the per-task accuracies are conditional on manually corrected upstream inputs. These issues prevent the quantitative central claim from being accepted as stated, but they are addressable in revision.
major comments (4)
- [Section III-C] The verification loop uses author-built 'golden Verilog-A' models that are constructed to cover the same performance metrics as the experimental template database. Task 3's 89% accuracy in Table III therefore measures whether the generated TED code reproduces the authors' behavioral models, not whether it correctly measures the real circuits described in the benchmark papers. Only one OTA is cross-checked against Cadence Virtuoso (Table II), and no transistor-level validation is reported for any BGR or LDO. The central claim requires either independent validation of the golden models (e.g., transistor-level simulation for all or a representative sample of benchmark circuits) or a re-scoped claim that the framework achieves self-consistency with respect to author-defined reference models.
- [Section V-A and Table III] The statement 'We manually ensure that the inputs upstream task for each task is correct, decoupling the success rates between tasks' means the three >80% rates are component-level conditional accuracies, not end-to-end pipeline accuracies. The abstract's claim of 'automated Testbench generation capabilities' and the conclusion's 'successfully generates testbenches' are therefore stronger than the evidence supports. Please report an end-to-end success rate over the 63 experiments without manual upstream correction, and report human correction and confirmation time separately from the sub-hour-per-paper claim. Section IV's 'After manual confirmation' step should also be included in this accounting.
- [Table II and Section IV] Table II reports numerically identical TED and Cadence Virtuoso results for all ten metrics. Please clarify whether the golden Verilog-A model for this OTA was calibrated to the Virtuoso results or derived independently, and state whether the same calibration procedure applies to the BGR and LDO golden models. If the golden models are fitted to the same quantities they are later used to verify, the Table II cross-check does not break the self-consistency concern.
- [Section V-B, Figures 7 and 8] The ablation results demonstrate that retrieval augmentation and iterative repair improve functional success within the same golden-model loop. However, because the loop's reference is the author-built Verilog-A models, the ablations do not establish correctness with respect to real circuit behavior; they show internal consistency of the generated code with the reference models. The text should state this limitation explicitly when interpreting the ablation results, and the phrase 'functional verification' should be qualified accordingly.
minor comments (4)
- [Section I] The contribution list contains 'operated amplifiers' where 'operational amplifiers' is intended; please correct the typo.
- [Section V-A and V-B] The main evaluation does not state the maximum iteration count used for Task 3; Section V-B mentions a maximum of 9 iterations only for the control group. Please specify the setting used for the Table III experiments to enable reproduction.
- [General] The paper would benefit from a data/code availability statement or an appendix with the exact prompts, the TED function database schema, and the simulator backend, since LLM-based systems are highly sensitive to prompt and toolchain details.
- [References] Reference [13] is incomplete: it lacks a publication venue and year; please complete the entry.
Circularity Check
Table III's 80%+ accuracy is scored against author-built golden Verilog-A models that cover the same performance metrics as the template database, and Section V-A manually corrects upstream task inputs; the figures certify self-consistency and conditional component accuracy, not independent end-to-end correctness.
-
self definitional
[Section III-C, Testbench Code Generation]
"At the same time, we have also modelled the Verilog-A models of op-amp, BGR and LDO for the performance metrics included in the experimental scheme template database mentioned earlier. These golden Verilog-A are used to verify the generated TED code."
The 'golden' reference is not an independent ground truth: the authors built these Verilog-A models specifically for the same performance metrics that appear in the experimental template database used to generate the schemes and code. The generated TED testbench is therefore judged correct when its simulation results match a model that encodes the same template assumptions (idealized op-amp/BGR/LDO behavior) that guided generation. This makes Task3's 89% a self-consistency score between the generator and the authors' own models. The only independent calibration is one OTA in Table II; no BGR or LDO in the 8-paper benchmark is checked against transistor-level simulation or measured silicon.
-
fitted input called prediction
[Section V-A, Basic Evaluation Method]
"We manually ensure that the inputs upstream task for each task is correct, decoupling the success rates between tasks."
The full-pipeline claim ('the full process of constructing a paper's testbench' under an hour) is evaluated after manually fixing the output of every upstream task. Thus the reported Task1/Task2/Task3 accuracies are conditional on perfect prior outputs, not the probability that the automatic pipeline produces a correct testbench from a raw paper. The 'prediction' of end-to-end automation is not derived by the system; the inputs to each stage are hand-supplied, so the success rates in Table III cannot be combined as an end-to-end success rate.
full rationale
The core generation chain (paper -> extracted specification -> experimental scheme -> TED code) is not itself circular: the TED code is genuinely synthesized from extracted paper text and template-based schemes, and one OTA (Table II) is cross-checked against Cadence Virtuoso, which is independent evidence that the loop can work for at least one circuit. However, the Task3 verification oracle is the authors' own golden Verilog-A models, constructed to cover exactly the same performance metrics as the experimental template database that drives testbench generation. Passing 'functional verification' therefore means the generated testbench reproduces numbers from models that encode the same idealizations as the generator's own inputs; for the 8-paper benchmark this is self-consistency, not validation against real transistor-level or measured behavior. The Section V-A statement that upstream task inputs are manually ensured correct further decouples the three tasks, so Table III reports conditional component accuracies rather than end-to-end automatic performance. The self-citations to prior author work (refs [13], [14]) are background examples of LLM analog design and are not load-bearing for the verification premise, so they do not raise the score further. Overall, the central benchmark claim is partially circular because the definition of 'correct' Task3 output is fixed by the authors' own models, while the one independent OTA check prevents the entire result from reducing to pure tautology.
Assumptions & free parameters
free parameters (4)
- maximum iteration count =
9
- LLM sampling temperature =
0
- experimental template database size =
36 templates
- TED function database size =
72 functions
assumptions (4)
- domain assumption The golden Verilog-A models correctly represent the benchmark circuits' real behavior.
- domain assumption GPT-4o with RAG can reliably extract structured simulation info from papers and generate syntactically correct TED code.
- domain assumption TED is a valid, complete environment for analog testbench construction.
- domain assumption Manually correcting upstream inputs does not invalidate per-task success rates.
Cite this review
Pith. "Pith review of AnalogTester: A Large Language Model-Based Framework for Automatic Testbench Generation in Analog Circuit Design." pith.science (2026). https://pith.science/paper/GVMA7QAC
@misc{pith2026250709965,
author = {Pith},
title = {Pith review of: AnalogTester: A Large Language Model-Based Framework for Automatic Testbench Generation in Analog Circuit Design},
year = {2026},
howpublished = {\url{https://pith.science/paper/GVMA7QAC}},
note = {Machine review of arXiv:2507.09965}
}
read the original abstract
Recent advancements have demonstrated the significant potential of large language models (LLMs) in analog circuit design. Nevertheless, testbench construction for analog circuits remains manual, creating a critical bottleneck in achieving fully automated design processes. Particularly when replicating circuit designs from academic papers, manual Testbench construction demands time-intensive implementation and frequent adjustments, which fails to address the dynamic diversity and flexibility requirements for automation. AnalogTester tackles automated analog design challenges through an LLM-powered pipeline: a) domain-knowledge integration, b) paper information extraction, c) simulation scheme synthesis, and d) testbench code generation with Tsinghua Electronic Design (TED). AnalogTester has demonstrated automated Testbench generation capabilities for three fundamental analog circuit types: operational amplifiers (op-amps), bandgap references (BGRs), and low-dropout regulators (LDOs), while maintaining a scalable framework for adaptation to broader circuit topologies. Furthermore, AnalogTester can generate circuit knowledge data and TED code corpus, establishing fundamental training datasets for LLM specialization in analog circuit design automation.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Stimulus generation for constrained random simulation,
N. Kitchen and A. Kuehlmann, “Stimulus generation for constrained random simulation,” in 2007 IEEE/ACM International Conference on Computer-Aided Design , pp. 258–265, Nov. 2007
work page 2007
-
[2]
Au- toBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design,
R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, “Au- toBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design,” in Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD , (Salt Lake City UT USA), pp. 1–10, ACM, Sept. 2024
work page 2024
-
[3]
UVLLM: An Automated Universal RTL Verification Framework using LLMs,
Y . Hu, J. Ye, K. Xu, J. Sun, S. Zhang, X. Jiao, D. Pan, J. Zhou, N. Wang, W. Shan, X. Fang, X. Wang, N. Guan, and Z. Jiang, “UVLLM: An Automated Universal RTL Verification Framework using LLMs,” Nov. 2024
work page 2024
-
[4]
“Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation | IEEE Conference Publication | IEEE Xplore.” https://ieeexplore.ieee.org/abstract/document/10323812
-
[5]
ChipGPT: How far are we from natural language hardware design,
K. Chang, Y . Wang, H. Ren, M. Wang, S. Liang, Y . Han, H. Li, and X. Li, “ChipGPT: How far are we from natural language hardware design,” June 2023
work page 2023
-
[6]
Chip-Chat: Chal- lenges and Opportunities in Conversational Hardware Design,
J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-Chat: Chal- lenges and Opportunities in Conversational Hardware Design,” in 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD) , pp. 1–6, Sept. 2023
work page 2023
-
[7]
AutoChip: Automating HDL Generation Using LLM Feedback,
S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “AutoChip: Automating HDL Generation Using LLM Feedback,” June 2024
work page 2024
-
[8]
Using llms to facilitate formal verification of rtl,
M. Orenes-Vera, M. Martonosi, and D. Wentzlaff, “Using llms to facilitate formal verification of rtl,” arXiv preprint arXiv:2309.09437 , 2023
arXiv 2023
Show all 29 references
-
[9]
Llm4dv: Using large language models for hardware test stimuli generation,
Z. Zhang, G. Chadwick, H. McNally, Y . Zhao, and R. Mullins, “Llm4dv: Using large language models for hardware test stimuli generation,” arXiv preprint arXiv:2310.04535, 2023
2023 arXiv
-
[10]
TED: A Python-Based Analog Design Environment for Agile Circuit Development,
Z. Ye, Z. Wang, J. Xin, Y . Wang, Q. Qin, C. Chai, Y . Lu, J. Hao, J. Xiao, and Y . Wang, “TED: A Python-Based Analog Design Environment for Agile Circuit Development,” in 2023 International Symposium of Electronics Design Automation (ISEDA) , pp. 5–10, May 2023
2023
-
[11]
Evaluating Large Language Models Trained on Code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, Pinto, et al. , “Evaluating Large Language Models Trained on Code,” July 2021
2021
-
[12]
Artisan: Automated Operational Amplifier Design via Domain-specific Large Language Model,
Z. Chen, J. Huang, Y . Liu, F. Yang, L. Shang, D. Zhou, and X. Zeng, “Artisan: Automated Operational Amplifier Design via Domain-specific Large Language Model,” in Proceedings of the 61st ACM/IEEE Design Automation Conference, (San Francisco CA USA), pp. 1–6, ACM, June 2024
2024
-
[13]
LADAC: Large Language Model- driven Auto-Designer for Analog Circuits,
C. Liu, Y . Liu, Y . Du, and L. Du, “LADAC: Large Language Model- driven Auto-Designer for Analog Circuits,”
-
[14]
AmpAgent: An LLM-based Multi-Agent System for Multi-stage Amplifier Schematic Design from Literature for Process and Performance Porting,
C. Liu, W. Chen, A. Peng, Y . Du, L. Du, and J. Yang, “AmpAgent: An LLM-based Multi-Agent System for Multi-stage Amplifier Schematic Design from Literature for Process and Performance Porting,” Sept. 2024
2024
-
[15]
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions | ACM Transactions on Information Systems
“A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions | ACM Transactions on Information Systems.” https://dl.acm.org/doi/abs/10.1145/3703155
-
[16]
Analog design with verilog-a,
I. Miller, D. FitzPatrick, and R. Aisola, “Analog design with verilog-a,” in Proceedings of Meeting on V erilog HDL (IVC/VIUF’97) , pp. 64–68, 1997
1997
-
[17]
Automated Generation Procedure for Fully Differ- ential Op-Amp Using TED,
Y . Wang, Q. Wu, Y . Wang, Q. Qin, J. Hao, C. Chai, Y . Lu, J. Huang, L. Li, and Z. Ye, “Automated Generation Procedure for Fully Differ- ential Op-Amp Using TED,” in 2024 2nd International Symposium of Electronics Design Automation (ISEDA) , pp. 100–105, May 2024
2024
-
[18]
LaMAGIC: Language-Model-based Topology Generation for Analog Integrated Circuits,
C.-C. Chang, Y . Shen, S. Fan, J. Li, S. Zhang, N. Cao, Y . Chen, and X. Zhang, “LaMAGIC: Language-Model-based Topology Generation for Analog Integrated Circuits,” Aug. 2024
2024
-
[19]
ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models,
Y . Yin, Y . Wang, B. Xu, and P. Li, “ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models,” June 2024
2024
-
[20]
Masala- chai: A large-scale spice netlist dataset for analog circuits by harnessing ai,
J. Bhandari, V . Bhat, Y . He, S. Garg, H. Rahmani, and R. Karri, “Masala- chai: A large-scale spice netlist dataset for analog circuits by harnessing ai,” 2025
2025
-
[21]
OpenAI Platform
“OpenAI Platform.” https://platform.openai.com
-
[22]
An Ultra- Low-V oltage Ultra-Low-Power CMOS Miller OTA With Rail-to-Rail Input/Output Swing,
L. H. C. Ferreira, T. C. Pimenta, and R. L. Moreno, “An Ultra- Low-V oltage Ultra-Low-Power CMOS Miller OTA With Rail-to-Rail Input/Output Swing,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 54, pp. 843–847, Oct. 2007
2007
-
[23]
0.5-V operational transconductance amplifier for CMOS bandgap reference application,
H. Wang and Q. Ye, “0.5-V operational transconductance amplifier for CMOS bandgap reference application,” in 2006 8th International Con- ference on Solid-State and Integrated Circuit Technology Proceedings , pp. 1705–1707, Oct. 2006
2006
-
[24]
A Miller-compensated amplifier with Gm-boosting,
M. W. Lau, M. Ho, K. H. Mak, S. Bu, K. N. Leung, and W. L. Goh, “A Miller-compensated amplifier with Gm-boosting,” in TENCON 2015 - 2015 IEEE Region 10 Conference , pp. 1–6, Nov. 2015
2015
-
[25]
A CMOS 25.3 ppm ◦/C bandgap voltage reference using self-cascode composite transistor,
D. Colombo, F. Werle, G. Wirth, and S. Bampi, “A CMOS 25.3 ppm ◦/C bandgap voltage reference using self-cascode composite transistor,” in 2012 IEEE 3rd Latin American Symposium on Circuits and Systems (LASCAS), pp. 1–4, Feb. 2012
2012
-
[26]
A low-voltage low-power voltage reference based on subthreshold MOSFETs,
G. Giustolisi, G. Palumbo, M. Criscione, and F. Cutri, “A low-voltage low-power voltage reference based on subthreshold MOSFETs,” IEEE Journal of Solid-State Circuits , vol. 38, pp. 151–154, Jan. 2003
2003
-
[27]
A 300 nW, 15 ppm/\circC, 20 ppm/V CMOS V oltage Reference Circuit Consisting of Subthreshold MOSFETs,
K. Ueno, T. Hirose, T. Asai, and Y . Amemiya, “A 300 nW, 15 ppm/\circC, 20 ppm/V CMOS V oltage Reference Circuit Consisting of Subthreshold MOSFETs,” IEEE Journal of Solid-State Circuits , vol. 44, pp. 2047–2054, July 2009
2009
-
[28]
A 65-nm CMOS Low Dropout Regulator Featuring >60-dB PSRR Over 10-MHz Frequency Range and 100-mA Load Current Range,
J. Jiang, W. Shu, and J. S. Chang, “A 65-nm CMOS Low Dropout Regulator Featuring >60-dB PSRR Over 10-MHz Frequency Range and 100-mA Load Current Range,” IEEE Journal of Solid-State Circuits , vol. 53, pp. 2331–2342, Aug. 2018
2018
-
[29]
High PSR Low Drop-Out Regulator With Feed-Forward Ripple Cancellation Technique,
M. El-Nozahi, A. Amer, J. Torres, K. Entesari, and E. Sanchez- Sinencio, “High PSR Low Drop-Out Regulator With Feed-Forward Ripple Cancellation Technique,” IEEE Journal of Solid-State Circuits , vol. 45, pp. 565–577, Mar. 2010
2010
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.