Abstractions-of-Thought: Intermediate Representations for LLM Reasoning in Hardware Design

Dileep Kalathil; Jeyavijayan Rajendran; Kevin Tieu; Matthew DeLorenzo; Piyush Jha; Prithwish Jana; Vijay Ganesh

REVIEW 5 major objections 5 minor 1 cited by

Abstractions-of-Thought: Intermediate Representations for LLM Reasoning in Hardware Design

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that inserting three task-based abstractions between a natural-language hardware specification and the final Verilog code—circuit classification, a structured JSON intermediate representation, and line-by-line…

desk verdict The multi-model AoT results are the real story and deserve a serious look; the single-model GPT-4o headline claim is a 0.3-point gap that the paper's own standard deviations do not support. read the letter →

arxiv 2505.15873 v2 pith:NX2F5IAX submitted 2025-05-21 cs.PL

Matthew DeLorenzo , Kevin Tieu , Prithwish Jana , Piyush Jha , Dileep Kalathil , Vijay Ganesh , Jeyavijayan Rajendran This is my paper · ORCID

classification cs.PL

keywords Abstractions-of-ThoughtLLMpromptingVeriloggenerationhardwaredesignintermediaterepresentationspromptengineeringEvalchain-of-thought

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Abstractions-of-Thought (AoT), a training-free prompting framework for turning natural-language hardware specifications into Verilog code. AoT inserts three intermediate representations between the prompt and the final code: a coarse classification of the circuit type, a structured JSON description of its logic, and line-by-line pseudocode. The claim is that these task-based abstractions make final Verilog generation easier for large language models by cutting off wrong reasoning paths and separating logical design from HDL syntax. On the VerilogEval benchmark, the paper reports that AoT gives GPT-4o the highest functional accuracy among the prompting strategies tested while using 1.8–5.2x fewer generated tokens than Tree-of-Thought prompting, and that a multi-model variant where a large model writes the abstractions and a small model writes the Verilog lifts the small model's functionality well above its baseline.

What carries the argument

The central mechanism is a staged prompt pipeline: a first classification into combinational versus sequential, a second classification into the most suitable logic representation, a JSON intermediate representation that restates the circuit's states, transitions, outputs, equations, truth table, K-map, or MUX mapping, and finally line-by-line pseudocode that bridges to the Verilog solution. The JSON IR is the load-bearing piece because it separates functional decomposition from code syntax, allowing the model to solve the logic before worrying about syntax, while the pseudocode reduces the final translation to a series of small, well-defined steps. Each stage's output is appended to the original prompt, so the final Verilog generation is anchored by the accumulated abstractions.

What would settle it

Take a set of VerilogEval prompts, run the AoT pipeline, then rerun it with one abstraction replaced by a deliberately corrupted version, such as a wrong combinational/sequential label or a JSON state transition that contradicts the specification, while keeping the final translation model fixed. If functional accuracy does not drop, the abstraction is not carrying the claimed load; alternatively, replace each abstraction with a ground-truth version and check whether accuracy rises.

Watch

Extended reading notes

Core claim

The paper's central claim is that structured, task-specific abstractions improve LLM-generated Verilog without any fine-tuning, external tools, or agent loops. The AoT pipeline first classifies the requested circuit as combinational or sequential and, for combinational designs, further identifies a useful representation such as truth table, Boolean expression, Karnaugh map, MUX mapping, or other. It then produces a JSON intermediate representation appropriate to that classification, and finally a line-by-line pseudocode description of the module before generating the Verilog. On the VerilogEval-Human benchmark, this three-stage pipeline gives GPT-4o a functional accuracy of 60.4 percent at pass@1, slightly above Tree-of-Thought at 60.1 percent and above the 59.0 percent of Chain-of-Thought, while using far fewer tokens than Tree-of-Thought. The paper also shows that when abstractions are generated by GPT-4o-mini and the final Verilog by the smaller DS-Coder-V2-Lite-Instruct or Llama-3.1-8B-Instruct, the small models improve substantially over their own single-model results, and the DS-Coder configuration exceeds both component models' individual functionality. The paper is careful to note that AoT does not help when applied alone to small models or to a dedicated reasoning model such as GPT-o3-mini, indicating that the benefit is concentrated in larger non-reasoning models and in multi-model setups.

Load-bearing premise

The pipeline assumes the model-generated classification, JSON intermediate representation, and pseudocode are faithful guides to the final Verilog; if any of those abstractions is wrong, the final prompt is anchored to a wrong plan, and the paper does not measure how often that happens.

Editorial extensions

If this is right

If AoT works as reported, large non-reasoning models can achieve higher Verilog functional correctness with a training-free prompting scheme than with 1-shot, Chain-of-Thought, or Tree-of-Thought prompting on the VerilogEval-Human benchmark.
The 1.8–5.2x reduction in generated tokens relative to Tree-of-Thought implies that multi-stage reasoning does not have to be verbose; structured intermediate representations can be cheaper than tree search.
The multi-model result implies that abstraction quality, not just model size, drives Verilog correctness: a small model paired with a large model's abstractions can exceed both the small model's and the large model's individual functionality in some configurations.
The ablation study shows that pseudocode is the most effective single abstraction and that the JSON IR alone hurts performance, meaning abstractions help most when they are combined with the original specification.
The paper suggests the same abstraction-based prompting pattern could transfer to other engineering domains that use layered abstractions, such as software engineering and cyber-physical system design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test the paper leaves open is measuring the correctness of each abstraction stage; if classification and JSON errors are rare, the gains likely come from pseudocode, whereas if they are common, the final model is absorbing flawed plans and the framework's headroom is larger than reported.
Because AoT is training-free, the generated abstractions could be collected as instruction data for fine-tuning small Verilog models, turning the pipeline into a data-generation mechanism the paper does not explore.
A more aggressive variant would generate multiple candidate IRs per prompt and select among them with a lightweight syntax check, trading a small amount of token overhead for better abstraction fidelity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: the multi-model AoT setup is the real result; the GPT-4o single-model headline is noise. The paper's actual contribution is a concrete three-stage prompting pipeline — classification, JSON intermediate representation, line-by-line pseudocode — for VerilogEval, evaluated systematically across four base models plus multi-model combinations, with ablations, per-strategy standard deviations, and full prompt templates in the appendix. That is a legitimate extension of Abstraction-of-Thought, Paradigm-based HDL generation, and hierarchical prompting. The specific combination is new and reproducible from the appendix, though no code is released.

Now the soft spots, in proportion. The abstract claims AoT outperforms all baseline techniques on GPT-4o. The support is 60.4% versus 60.1% for ToT, 59.0% for CoT, 57.8% baseline, with standard deviations of about 1.7, 1.7, and 1.0 points. The AoT–ToT gap is roughly 0.18 standard deviations. No paired significance test is reported, despite the fact that the same 156 prompts are evaluated under each strategy across the same five samples. A paired McNemar or bootstrap would settle this easily, and the authors should run one. The paper's own Section 5.1 admits AoT wins on only one of four base models, which makes the abstract's sweeping wording hard to defend. Separately, the SoT collapse on GPT-4o-mini — functionality at 6.9% versus a 48.3% baseline — looks like an implementation artifact and needs an explanation. Minor but real: the paper never measures classification accuracy or IR correctness, so a bad abstraction could silently anchor the final Verilog. The ablation mitigates this partially, but direct reporting would be better.

What is solid: the token reduction versus ToT is clearly supported, and the multi-model results are the paper's strongest evidence. GPT-4o-mini abstractions with Llama-3.1-8B give 35.9% functionality versus 16.2% baseline and 25.4% CoT, well outside one standard deviation. The o3-mini plus DS-Coder configuration shows a similar statistically clear gain. That is a meaningful result for the LLM4EDA crowd.

Recommendation: send it to peer review, but with major revision requested. The authors need to add a paired significance test for the headline comparison, fix the abstract to match what Table 1 actually shows, release the prompts or code, and explain the SoT anomaly. A serious referee would get useful work out of this.

Referee Report

5 major / 5 minor

Summary. The paper proposes Abstractions-of-Thought (AoT), a training-free, inference-only prompting framework for generating Verilog code from natural-language hardware specifications. AoT chains three abstraction stages before the final Verilog generation: (1) a two-level classification of the circuit as combinational/sequential and then as a specific design structure (truth table, Boolean expression, Karnaugh map, MUX, FSM, or other), (2) a JSON-based intermediate representation tailored to the classified structure, and (3) a line-by-line pseudocode description. The method is evaluated on the VerilogEval-Human benchmark across four models (GPT-4o, GPT-4o-mini, DS-Coder-V2-Lite-Instruct, Llama-3.1-8B-Instruct), several multi-model combinations, and five seeds, with pass@1 and pass@5 reported. The headline results are a pass@1 functionality improvement on GPT-4o (60.4% versus 60.1% for ToT and 57.8% for baseline) and a 1.8-5.2x reduction in generated tokens relative to ToT, while the multi-model configuration with GPT-4o-mini-generated abstractions improves the functionality of DS-Coder-V2-Lite-Instruct to 51.5% and of Llama-3.1-8B-Instruct to 35.9%.

Significance. If the empirical claims held as stated, AoT would be a useful, economical addition to the prompting toolkit for hardware design: it is model-agnostic, requires no training or external tools, ships reproducible prompt templates in the appendix, and its token reduction relative to ToT is real and clearly supported by Table 2. The multi-model result is the most convincing contribution, with the 4o-mini-plus-small-model configurations exceeding a strong single-model baseline by large margins (e.g., Llama functionality from 16.2% to 35.9%). The paper also deserves credit for reporting standard deviations, pass@5, ablations, and per-stage template prompts. However, the central single-model claim is statistically fragile and the abstract overstates what Table 1 supports, so the paper needs revision before the claims can be accepted as stated.

major comments (5)

[§5.1, Table 1, §A.5] The headline claim that AoT outperforms all baselines on GPT-4o is not statistically supported. The AoT functionality rate is 60.4% versus 60.1% for ToT, and Table 6 reports standard deviations of 1.7% for both; the 0.3 percentage-point gap is far below run-to-run noise. Because all 156 VerilogEval prompts are evaluated under every strategy across the same five samples, a paired test (McNemar, bootstrap, or per-problem comparison) is straightforward and should be reported. Without it, the correct statement is that AoT is competitive with ToT on this configuration while using fewer tokens. Note also that in Table 4 the pass@5 functionality of AoT on GPT-4o (66.7%) is numerically below SoT (69.2%) and equal to baseline, so 'outperforms all baseline techniques' is not supported at pass@5 either.
[Abstract, §5.1, §6, Table 1] The abstract and conclusion overstate the scope of the result. The abstract claims AoT 'demonstrates improvements in functionality when applied to large non-reasoning models (such as GPT-4o), outperforming all baseline techniques,' but Section 5.1 itself states that AoT outperforms all other strategies on only one of four single-model configurations. In Table 1, AoT is below the baseline on DS-Coder-V2-Lite-Instruct (40.4% vs. 46.9%) and on Llama-3.1-8B-Instruct (13.7% vs. 16.2%), and in Table 4 it is below baseline on GPT-o3-mini (69.2% vs. 74.6%). The abstract and conclusion should be rewritten to attribute the functionality claim specifically to GPT-4o and to the multi-model configurations, and to state the statistical caveat from the first major comment.
[§5.2, Table 1] The claim that the multi-model AoT strategy 'exceed[s] the performance of either model's individual capabilities' is false as written. For the 4o-mini & Llama-3.1-8B-Instruct configuration, Table 1 gives AoT functionality of 35.9%, which is below GPT-4o-mini's standalone functionality of 48.3% (and below GPT-4o-mini's baseline of 48.3%). The statement is only true for the 4o-mini & DS-Coder-V2-Lite-Instruct configuration. This claim appears in the contribution list and in Section 5.2 and must be corrected or qualified.
[§5.3, Tables 3 and 5] The ablation discussion contains claims that are not supported by the tables. Section 5.3 says combining IR with the base prompt 'improves the performance in both setups,' but Table 3 shows Base + IR functionality of 45.9% for the DS-Coder configuration versus 46.9% baseline, i.e., a decline; only the Llama configuration improves (18.7% vs. 16.2%). Similarly, the text says the best results are achieved by the full AoT framework, but Table 5 shows that for GPT-4o the full framework (59.7%) is below both Base + Pseudo (60.4%) and Base + IR (60.4%). These statements need to be revised to reflect the actual pattern, and the discussion should address why the full framework is not uniformly the best.
[§3.1, §3.2, §3.3, §5.3] The paper attributes AoT's success to the quality of its intermediate abstractions, but it never measures the fidelity of the classification labels, JSON IRs, or pseudocode generated at each stage. The ablation results in Tables 3 and 5 show that the IR alone can be harmful (e.g., 21.4% functionality for DS-Coder) and that the full pipeline is not always better than its parts, so the role of each stage is not established. I recommend reporting per-stage accuracy on a labeled subset of VerilogEval (e.g., C1 and C2 classifications checked against ground-truth circuit categories) and, if possible, the correlation between IR validity and final pass@1. This would also help assess the risk, acknowledged in the Limitations section, that a wrong abstraction silently propagates to the final Verilog.

minor comments (5)

[§3.2] The sentence beginning 'FSM.utilized for all sequential circuits to delineate all possible states...' is duplicated verbatim in the paragraph describing the intermediate representations; one copy should be removed.
[Tables 2 and 7] The token counts for SoT on GPT-4o-mini are inconsistent between Table 2 (200 tokens) and Table 7 (139 ± 5.7 tokens). Please clarify which quantities are being reported and ensure the two tables agree.
[§4.2, §A.3] The multi-model CoT implementation uses an implicit 'think step-by-step' prompt, whereas the single-model CoT uses an explicit exemplar. This methodological difference should be acknowledged as a potential confound when comparing multi-model strategies, since the two CoT variants are not identical.
[§5, Table 4] The GPT-o3-mini results in Table 4 are discussed only in Appendix A.4; the main text should either point to that discussion or mention that reasoning models are outside the claimed scope of AoT.
[§4.1] The pass@k formula is garbled as printed: 'pass@k = E[(1−C(n−c, k)), C(n, k)]' should be written with binomial coefficients and the standard unbiased estimator, e.g., pass@k = E[1 - C(n-c, k) / C(n, k)] over the n samples.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AoT is an externally benchmarked prompting framework with no fitted parameters or self-citation chain.

full rationale

The paper's central claims are empirical evaluations of a prompting framework against the external VerilogEval benchmark. The AoT prompts are fixed template designs; the intermediate representations, classifications, and pseudocode are all generated by the LLM from the same natural-language specification, with no gold labels or fitted constants inserted by the authors. There is no equation in which an output is defined in terms of an input, no parameter fitted to a subset of the benchmark and then predicted, and no reliance on prior work by the same authors to justify a load-bearing premise. The only referenced prior framework, Sun et al. (2025), is cited as motivation for JSON-style intermediate representations but is not used as an authority to force the result. The statistical weakness of the GPT-4o functionality comparison (60.4% vs 60.1% for ToT, within one standard deviation) is a legitimate correctness and significance concern, but it is not circularity: the numbers come from external testbench simulation, not from the method's own construction. The paper also honestly reports limitations, including that AoT underperforms for smaller models, which further indicates that the framework is not making self-fulfilling predictions. Overall, no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numerical free parameters or invented physical or conceptual entities are introduced. The paper's evidence is an empirical comparison on a public benchmark, with prompts fixed by the authors and no constants fitted to the data.

assumptions (3)

domain assumption LLM-generated abstractions (classification, JSON IR, pseudocode) preserve the information needed for correct Verilog generation
The entire framework concatenates these abstractions into the final prompt, so if they contain errors, the final code inherits them. The paper never measures intermediate accuracy.
domain assumption VerilogEval-Human v1.0.0 is a representative benchmark for LLM hardware design capability
The paper uses it as the sole evaluation suite and acknowledges in the limitations that it only covers small single-module designs.
domain assumption Icarus Verilog simulation correctly determines compilability and functional correctness
The benchmark relies on Icarus Verilog and its testbenches; any limitation of that simulator affects the reported accuracies.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Abstractions-of-Thought: Intermediate Representations for LLM Reasoning in Hardware Design." pith.science (2026). https://pith.science/paper/NX2F5IAX

@misc{pith2026250515873,
  author       = {Pith},
  title        = {Pith review of: Abstractions-of-Thought: Intermediate Representations for LLM Reasoning in Hardware Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NX2F5IAX}},
  note         = {Machine review of arXiv:2505.15873}
}

read the original abstract

Large language models (LLMs) have achieved impressive proficiency on logic and programming tasks, often rivaling expert-level performance. However, generating functionally correct hardware description language (HDL) code from natural language specifications remains challenging, primarily in data-scarce domains. Therefore, we present Abstractions-of-Thought (AoT) - a training-free, inference-only prompting framework to mitigate misinterpretations and reasoning pitfalls of LLMs through a series of task-based abstractions within the prompting procedure, assisting in the transition from high-level to low-level representations of hardware. Furthermore, AoT consists of the following stages: (1) an LLM-based classification of hardware design patterns, (2) a structured intermediate representation (IR) to separate functional decomposition from code syntax, and (3) a line-by-line pseudocode solution enabling a more direct mapping to the final Verilog implementation. Experimental results on the VerilogEval benchmark depict that AoT demonstrates improvements in functionality when applied to large non-reasoning models (such as GPT-4o, outperforming all baseline techniques (including 1-shot, Chain-of-Thought, and Tree-of-Thought) while significantly reducing the generated tokens by 1.8-5.2x compared to popular Tree-of-Thought prompting.

Figures

Figures reproduced from arXiv: 2505.15873 by the authors.

**Figure 2.** Classification prompt (C2). the logic of a circuit, dependent on the design complexity and objective. Therefore, if the circuit is combinational, the C2 prompt classifies the circuit according to a representative set of typical combinational structures, including: truth tables, boolean expressions, Karnaugh-Maps, and multiplexers (MUX), each of which is further detailed in Section 3.2. This prompt, similar to C1,… view at source ↗

**Figure 3.** Counter module IR abstraction. In implementing this abstraction, we seek to translate each of the five structures into a text-based format easily interpretable by the LLM. Therefore, motivated by the paradigm approach [Sun et al., 2025], we provide various JSON formats that represent each of our structures to ensure an organized and concise representation of the information. This is accomplished through five tem… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Counter module pseudocode abstraction. This component is implemented similarly to the prior abstractions, in which the template prompt is first defined. Then, for a given design, the standard prompt and any of its prior abstraction results (i.e., classification and IR)…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Progressive Approach to Synthesizable RTL Design Generation Using LLMs
cs.AR 2026-07 conditional novelty 6.0 of 10

VeriRefine boosts LLM-generated RTL correctness to 94.0% on RTLLM v2.0 and 98.1% on VerilogEval-Human v2 by refining and auditing a per-signal intermediate representation before code generation.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages · cited by 1 Pith paper

[10]

Identify the relationship between the input and output signals based on the description

work page
[11]

Generate the MUX mapping (a circuit built with multiplexers) that describes the relationship between the input and output signals

work page
[12]

mux_1": 14{ 15

Present the result as a MUX mapping, using the following JSON block format. 8 9Respond with a JSON block only. Please do not respond with any other text in your response. 10The JSON block should have the format: 11``` 12{ 13"mux_1": 14{ 15"type": "mux_type_1", 16"output": "output_var_1", 17"select": ["select_var_11", "select_var_12", ...], 18"input": 19{ ...

work page
[13]

Identify all input and output signal names from the module header

work page
[14]

Enumerate every possible combination of the input signals

work page
[15]

For each combination, compute the correct output values based on the description

work page
[16]

input": [

Present the result as a JSON-formatted truth table, with input and output variables matching the signal names in the same order they appear in the module header. 8 9Respond with a JSON block only. Please do not respond with any other text in your response. 10The JSON block should have the format: 11``` 12{{ 13"input": ["input_var_1", "input_var_2", ...], ...

work page
[63]

Generate the boolean expression that describes the relationship between the input and output signals

work page

Show all 10 references

[64]

input": [

Present the result as a JSON-formatted Karnaugh map, with input and output variables matching the signal names in the same order they appear in the module header. 7 8Respond with a JSON block only. Please do not respond with any other text in your response. 9The JSON block sho...
[74]

input": [

Present the result as a boolean expression, using the following format: 8 9Respond with a JSON block only. Please do not respond with any other text in your response. 10The JSON block should have the format: 11``` 12{{ 13"input": ["input_var_1", "input_var_2", ...], 14"output"...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [10]

Identify the relationship between the input and output signals based on the description

work page

[2] [11]

Generate the MUX mapping (a circuit built with multiplexers) that describes the relationship between the input and output signals

work page

[3] [12]

mux_1": 14{ 15

Present the result as a MUX mapping, using the following JSON block format. 8 9Respond with a JSON block only. Please do not respond with any other text in your response. 10The JSON block should have the format: 11``` 12{ 13"mux_1": 14{ 15"type": "mux_type_1", 16"output": "output_var_1", 17"select": ["select_var_11", "select_var_12", ...], 18"input": 19{ ...

work page

[4] [13]

Identify all input and output signal names from the module header

work page

[5] [14]

Enumerate every possible combination of the input signals

work page

[6] [15]

For each combination, compute the correct output values based on the description

work page

[7] [16]

input": [

Present the result as a JSON-formatted truth table, with input and output variables matching the signal names in the same order they appear in the module header. 8 9Respond with a JSON block only. Please do not respond with any other text in your response. 10The JSON block should have the format: 11``` 12{{ 13"input": ["input_var_1", "input_var_2", ...], ...

work page

[8] [63]

Generate the boolean expression that describes the relationship between the input and output signals

work page