Pith. sign in

REVIEW 4 major objections 6 minor 18 references

A structured contract, not a longer prompt, is what lifts LLM-based HLS design reliability, with hardware-in-the-loop feedback carrying the workflow from kernel code to board-level closure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A contract-aligned multi-agent workflow with hardware-in-the-loop feedback raises HLS-Eval testbench pass@1 from 64.0% to 70.4% and enables a dual-bitstream PQC accelerator with 4x faster runtime.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A solid, honest workflow paper whose central HLS-Eval gain is plausible but statistically under-supported; the PQC case is a real board-level demonstration and the open-source artifact earns it a referee. the 4 major comments →

arxiv 2607.25283 v1 pith:MFLSWPOU submitted 2026-07-28 cs.AI cs.ARcs.CR

ContractHIL-HLS: Contract-Aligned Multi-Agent Workflow with Hardware-in-the-Loop Feedback for HLS Design

Jingbo Zhang , Haoxiang Sun , Wenbo Wang , Wenbo Zhang This is my paper
classification cs.AI cs.ARcs.CR
keywords high-level synthesisLLM agentshardware-in-the-loopstructured contractFPGA designpost-quantum cryptographyrollback policyHLS benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the reliability of LLM-generated high-level-synthesis (HLS) hardware code is improved most by converting the natural-language request into a structured, inspectable contract before code generation, and that feeding measured synthesis and board-level evidence back into the loop extends the workflow from small kernel tasks to a complete board-tested system. On 94 locally executable HLS benchmark tasks, the contract alone raises the estimated single-sample testbench pass rate from 64.0% to 70.2%; adding the persistent HTML handoff and hardware-in-the-loop revision reaches 70.4% pass@1 and 76.6% pass@5. The paper then tests the same workflow on a board-tested post-quantum-cryptography secure-message accelerator, where the hardware-feedback loop selects a two-bitstream organization that cuts average text runtime from 207.3 ms to 52.4 ms with legal timing on both bitstreams. A sympathetic reader would care because this is evidence that structured state, rather than merely longer prompting, is what makes LLM-based hardware design more reliable and traceable.

Core claim

The paper claims that converting a natural-language design request into a structured contract — named fields for interface, constraints, validation, and rollback — before code generation is the largest single lever for reliable LLM-based high-level synthesis, and that feeding synthesis, implementation, and board-measured evidence back into the loop extends the workflow from kernel code to system-level closure. On 94 tasks, the contract alone raises the estimated single-sample testbench pass rate from 64.0% to 70.2%; adding the HTML handoff and hardware-feedback loop reaches 70.4% pass@1 and 76.6% pass@5. In a board-tested post-quantum-cryptography secure-message accelerator, the hardware loo

What carries the argument

The central artifact is the structured contract C = ⟨R, I, F, H, K, O, V, B⟩ — role/platform, public interface, functional intent, hard constraints, compatibility, optimization intent, validation evidence, and rollback policy. The Contract Agent lowers natural language into this contract, the HTML Agent renders it as persistent, inspectable HTML that downstream agents and scripts can parse, and the Hardware-in-the-Loop Agent implements from that artifact, using EDA and board measurements to keep or roll back each candidate. This turns tool output into first-class workflow state and makes design decisions traceable.

Load-bearing premise

The central quantitative claim assumes that pass@1 estimated from five independent samples per task on 94 tasks is a stable measure, and that the single model endpoint used across the three regimes is representative; if those deltas are sampling noise, the contract-alignment claim is not established.

What would settle it

Re-run the 94-task comparison with larger sample counts (e.g., 50 per task) or a different underlying model; if the 64.0% to 70.2% testbench gap shrinks to statistical noise, the claim fails. On the board side, re-measure the two-bitstream PQC flow on a different FPGA; if the runtime reduction does not reproduce, the hardware-in-the-loop decision evidence is anecdotal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If true, contract fields (interface, constraints, validation, rollback) are the actionable lever: they preserve design intent even when the underlying model changes.
  • Hardware-in-the-loop feedback can drive design-level decisions, such as partitioning a monolithic bitstream into two images, that are invisible to kernel-only benchmarks.
  • Rollback rules make tool and board measurements first-class workflow state, so a regression cannot silently become the next context prefix.
  • The workflow does not rescue tasks where the model lacks the algorithmic capability (the machsuite family stays at 0%), so gains are bounded by the base model's competence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extending beyond the paper: the same contract artifact could be applied to RTL or software generation, where interface and rollback rules are equally valuable for preserving intent.
  • The reported pass@1 gains are measured against one unstated model endpoint; testing across several models would reveal whether the contract effect is model-dependent or general.
  • The dual-bitstream decision suggests a broader design pattern: use measured resource and timing pressure to split a system across FPGA images, with the contract recording the split rationale for audit.
  • The HTML contract could serve as a machine-readable audit trail for hardware trust and verification, beyond its role in prompt alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ContractHIL-HLS, a three-agent workflow for LLM-based high-level synthesis: a Contract Agent normalizes natural-language requirements into a structured contract, an HTML Agent renders the contract as a persistent HTML artifact, and a Hardware-in-the-Loop Agent generates and revises HLS code using EDA and board-level feedback. The central quantitative claim is evaluated on 94 locally executable HLS-Eval tasks, comparing a Direct prompt baseline, Contract (contract fields only), and the full ContractHIL-HLS flow. The reported results are a testbench pass@1 improvement from 64.0% (Direct) to 70.2% (Contract) and 70.4% (full flow), with pass@5 at 76.6%. A separate board-level PQC (ML-KEM/ML-DSA) case study reports a two-bitstream organization that reduces average text runtime from 207.3 ms to 52.4 ms while maintaining positive routed timing slack and decrypted-message verification. The authors frame HLS-Eval as evidence for contract alignment and the PQC study as a hardware-in-the-loop system demonstration.

Significance. If the central claim holds, the paper makes a useful contribution: it demonstrates that a structured, persistent contract can improve LLM-based HLS reliability beyond prompt-level repetition, and it provides a rare example of a board-tested workflow that uses hardware measurements to drive design decisions. The evaluation is honest in several respects: negative results (machsuite 0/17) are reported, the stage ablation cleanly separates the contract contribution from the HTML/hardware-loop contribution, and the PQC dual-EDP calculation is deliberately conservative. The open-sourced repository and reproducible harness are strengths. However, the main quantitative claim currently lacks statistical support and the model is not identified, which limits the strength of the conclusion that can be drawn.

major comments (4)
  1. [§IV.A, Table II] The paper's central quantitative claim — that the structured contract raises testbench pass@1 from 64.0% to 70.2% — is reported without any uncertainty quantification. The 94 tasks are sampled five times each; since pass/fail outcomes are strongly clustered by task (e.g., machsuite remains 0/17 and the chstone gain moves a small number of tasks), the effective sample size is closer to 94 than 470. A pooled two-proportion test is borderline (z≈2.0) and would likely lose significance under task-level clustering or multiple-comparison correction. Please report paired bootstrap CIs, a McNemar-style test on per-task outcomes, or per-task pass rates, and state the model endpoint explicitly (also see next comment). Without this, 'contract alignment improves HLS reliability' is not established.
  2. [§IV.A] The model is only described as 'same model endpoint'. Neither the model name/version nor sampling parameters are given. Since the effect size and its generalizability are model-dependent, and since the HLS-Eval paper shows large variance across models, please disclose the model (and version, temperature, top-p, seeds) in the text or repository. This is necessary for the result to be independently assessed.
  3. [§IV.C, Table IV] The category-level table shows that the aggregate gain is concentrated in a small number of tasks: chstone (n=20) moves from 80.0 to 95.0 (roughly three tasks at the per-task level), polybench from 92.9 to 96.4 (one task), pp4fpga from 66.7 to 100.0 (one task), and rosetta from 87.5 to 100.0 (one task); machsuite is unchanged at 0/17. The current text says 'HLS-Eval supports the contract-alignment claim' but the distribution of the effect tempers this. Report per-task counts and/or a paired test; if the effect is driven by a handful of tasks, say so explicitly and discuss implications for the claimed generality.
  4. [§V, Table V] The PQC case study is a single non-replicated system demonstration. The 207.3 ms → 52.4 ms runtime reduction and EDP improvement are based on one flow; no run-to-run variance, synthesis seed variance, or measurement uncertainty is reported. As an existence proof this is acceptable, but the manuscript should clearly label it as a single-case demonstration and avoid wording that implies a stable effect size. Also clarify how many independent board runs and message repetitions the 'six-message average' represents beyond the 100 repetitions per message mentioned in the equation.
minor comments (6)
  1. [§VI] The Introduction says Section VI 'concludes with limitations and future directions', but the Conclusion contains no limitations paragraph. Add a brief limitations subsection or fix the roadmap sentence.
  2. [§IV.A] The definition of pass@1 is ambiguous: 'average single-sample success rate across the five independent samples' could mean the mean of per-task sample pass rates or the pooled proportion of all 470 samples. Please define precisely (e.g., pass@1 = (1/N) Σ_i (1/k) Σ_j 1[task i, sample j passes]) and report the underlying counts.
  3. [Table III] The Token column would benefit from a definition (average tokens per task? total?). Also, the claim that Contract is shorter than Direct because it replaces repeated instructions is plausible but should be tied to a clear unit.
  4. [Table II] The DeepSeek V3 row is described as 'published context', not a controlled comparison. This is appropriately hedged, but consider moving it to a separate table or clearly separating it visually to avoid reader confusion.
  5. [Table V] The 'Dual system' column reports 'both non-negative' for WNS; include the actual two WNS values (0.769 and 0.171 ns) in the table or text for completeness.
  6. [Abstract] The phrase 'retained dual-bitstream organization' is unclear on first reading. Consider 'the selected two-bitstream organization' or 'the adopted dual-bitstream organization'.

Circularity Check

0 steps flagged

No significant circularity: the reported comparisons are empirical evaluations against an external benchmark and a board-level case study, not derivations from the paper's own inputs.

full rationale

The paper's central claims are empirical rather than definitional. Direct, Contract, and ContractHIL-HLS are three prompt/workflow regimes evaluated against the external HLS-Eval benchmark using independent scripts for parse, compile, testbench, and synthesis; the reported pass@1 and pass@5 rates are measured outcomes, not quantities defined by the contract. The contract fields are prompt-design interventions, and the paper explicitly tests whether they change testbench pass rates, so the evaluation does not reduce to the contract by construction. The PQC case study is a board-level measurement with an explicit conservative EDP formulation; it is presented as a single deployment demonstration, not as a derived prediction. There are no load-bearing self-citations: references are to external benchmarks and prior work, and no uniqueness or ansatz is imported from the authors' own prior publications. The paper also discloses its own limitations, including the machsuite 0/17 negative result and the single-case nature of the PQC demonstration, which further indicates that the workflow's gains are not asserted as inevitable consequences of its definitions. Statistical concerns about sampling noise and the unnamed model endpoint are correctness/evidence weaknesses, not circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The workflow introduces a structured contract artifact and a two-bitstream organization, but these are engineering artifacts, not postulated scientific entities. No fitted parameters or invented physical entities are introduced. The load-bearing assumptions are statistical (pass@1 estimator), representational (HLS-Eval subset), and tool-accuracy related.

axioms (5)
  • standard math Five independent samples per task yield an unbiased single-sample pass-rate estimate (pass@1).
    Used to report average single-sample success rate in Table II; no confidence intervals are given.
  • domain assumption The 94-task locally executable HLS-Eval subset is representative of the HLS-Eval benchmark for workflow comparison.
    Section IV.A says all quantitative evidence comes from this subset; the full benchmark may differ.
  • domain assumption The same, unnamed LLM endpoint is stable and identical across Direct/Contract/ContractHIL-HLS runs.
    Section IV.A states 'same model endpoint' but never names the model.
  • domain assumption Routed Vivado timing/power reports and board runtime measurements are accurate for comparing design organizations.
    Table V uses WNS, dynamic W, and EDP from routed reports; no independent measurement validation.
  • domain assumption Receiver-side decrypted-message verification is sufficient evidence of functional correctness for the PQC secure-message flow.
    Section V says the contract 'validates only the decrypted message' to prevent sender-path false positives; no other security checks.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ContractHIL-HLS: Contract-Aligned Multi-Agent Workflow with Hardware-in-the-Loop Feedback for HLS Design." pith.science (2026). https://pith.science/paper/MFLSWPOU

@misc{pith2026260725283,
  author       = {Pith},
  title        = {Pith review of: ContractHIL-HLS: Contract-Aligned Multi-Agent Workflow with Hardware-in-the-Loop Feedback for HLS Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFLSWPOU}},
  note         = {Machine review of arXiv:2607.25283}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper presents ContractHIL-HLS, a contract-aligned multi-agent workflow for practical high-level synthesis (HLS) engineering. The workflow makes three contributions. First, it introduces a structured contract as the semantic-alignment and task-execution artifact that translates natural language requirements into explicit interfaces, constraints, validation checks, and rollback rules. Second, it incorporates hardware information into the feedback loop by feeding HLS, Vivado, PYNQ runtime, power, and failure evidence back into generation, thereby extending LLM-assisted HLS from kernel code toward system- and board-level closure. Third, it decomposes agents by semantic lowering and execution tasks rather than by conversational roles: a Contract Agent lowers natural language into the contract, an HTML Agent renders the contract as persistent structured HTML, and a Hardware-in-the-Loop Agent implements and revises the design with measured evidence. We evaluate ContractHIL-HLS in two parts. On 94 locally executable HLS-Eval tasks, the structured contract provides the largest small design gain, improving the estimated single-sample testbench pass rate from 64.0% to 70.2%; the full flow reaches 70.4% pass@1 and 76.6% pass@5. Because HLS-Eval does not exercise board-level design, we also validate ContractHIL-HLS on a board tested ML-KEM/ML-DSA post-quantum cryptography (PQC) secure-message accelerator, where the retained dual-bitstream organization reduces six-message average text runtime from 207.3 ms to 52.4 ms with positive routed WNS on both images while preserving decrypted-message verification. We open-source our work at BJUT-CS316-LAB/ContractHIL-HLS (https://github.com/BJUT-CS316-LAB/ContractHIL-HLS).

Figures

Figures reproduced from arXiv: 2607.25283 by Haoxiang Sun, Jingbo Zhang, Wenbo Wang, Wenbo Zhang.

Figure 1
Figure 1. Figure 1: ContractHIL-HLS workflow overview with retained contract state, EDA feedback, PYNQ (Python Productivity for Zynq) measurements, QoR (quality [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Conceptual positioning of representative LLM-assisted hardware [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Natural-language-to-HTML flow in the contract-aligned workflow. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: HLS-Eval testbench-pass results. The final bar is the HLS-Eval paper’s [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: PQC contract loop under the initial constraint: implement ML-KEM [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 2 linked inside Pith

  1. [1]

    Chip-Chat: Challenges and Opportunities in Conversational Hardware Design,

    J. Blocklove, S. Garg, R. Karriet al., “Chip-Chat: Challenges and Opportunities in Conversational Hardware Design,” inProc. ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD), 2023, pp. 1–6

    2023

  2. [2]

    RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model,

    Y . Lu, S. Liu, Q. Zhanget al., “RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model,” inProc. 29th Asia and South Pacific Design Automation Conf. (ASP-DAC), 2024, pp. 722–727

    2024

  3. [3]

    HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Tasks,

    S. Abi-Karam and C. Hao, “HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Tasks,” inProc. IEEE Int. Conf. LLM-Aided Design (ICLAD), 2025, pp. 219–226

    2025

  4. [4]

    A Survey on Large Language Model Based Autonomous Agents,

    L. Wang, C. Ma, X. Fenget al., “A Survey on Large Language Model Based Autonomous Agents,”Frontiers of Computer Science, vol. 18, art. 186345, 2024

    2024

  5. [5]

    A Survey on LLM-Based Multi-Agent Systems: Workflow, Infrastructure, and Challenges,

    X. Li, S. Wang, S. Zenget al., “A Survey on LLM-Based Multi-Agent Systems: Workflow, Infrastructure, and Challenges,”Vicinagearth, vol. 1, art. 9, 2024

    2024

  6. [6]

    HLStrans: Dataset for C-to-HLS Hardware Code Synthesis,

    Q. Zou, N. Chen, Y . Chenet al., “HLStrans: Dataset for C-to-HLS Hardware Code Synthesis,” arXiv:2507.04315, 2025

    arXiv 2025

  7. [7]

    SAGE-HLS: Syntax- Aware AST-Guided LLM for High-Level Synthesis Code Generation,

    M. Z. S. Khan, N. Mashnoor, M. Akyashet al., “SAGE-HLS: Syntax- Aware AST-Guided LLM for High-Level Synthesis Code Generation,” inProc. IEEE 43rd Int. Conf. Computer Design (ICCD), 2025, pp. 574– 581

    2025

  8. [8]

    KyberMat: Efficient Accelerator for Matrix-Vector Polynomial Multiplication in CRYSTALS-Kyber Scheme via NTT and Polyphase Decomposition,

    W. Tan, Y . Lao, and K. K. Parhi, “KyberMat: Efficient Accelerator for Matrix-Vector Polynomial Multiplication in CRYSTALS-Kyber Scheme via NTT and Polyphase Decomposition,” inProc. IEEE/ACM Int. Conf. Computer-Aided Design (ICCAD), 2023, pp. 1–9

    2023

  9. [9]

    CRYPHTOR: A Memory- Unified NTT-Based Hardware Accelerator for Post-Quantum CRYS- TALS Algorithms,

    S. Di Matteo, I. Sarno, and S. Saponara, “CRYPHTOR: A Memory- Unified NTT-Based Hardware Accelerator for Post-Quantum CRYS- TALS Algorithms,”IEEE Access, vol. 12, pp. 25501–25511, 2024

    2024

  10. [10]

    VerilogEval: Evaluating Large Language Models for Verilog Code Generation,

    M. Liu, N. Pinckney, B. Khailanyet al., “VerilogEval: Evaluating Large Language Models for Verilog Code Generation,” inProc. IEEE/ACM Int. Conf. Computer-Aided Design (ICCAD), 2023, pp. 1–8

    2023

  11. [11]

    VeriGen: A Large Language Model for Verilog Code Generation,

    S. Thakur, B. Ahmad, H. Pearceet al., “VeriGen: A Large Language Model for Verilog Code Generation,”ACM Trans. Design Automation of Electronic Systems, vol. 29, no. 3, pp. 1–31, 2024

    2024

  12. [12]

    ChipGPT: How Far Are We from Natural Language Hardware Design,

    K. Chang, Y . Wang, H. Renet al., “ChipGPT: How Far Are We from Natural Language Hardware Design,” arXiv:2305.14019, 2023

    arXiv 2023

  13. [13]

    ChipNeMo: Domain-Adapted LLMs for Chip Design,

    M. Liu, T.-D. Ene, R. Kirbyet al., “ChipNeMo: Domain-Adapted LLMs for Chip Design,” arXiv:2311.00176, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    FIPS 203: Module- Lattice-Based Key-Encapsulation Mechanism Standard,

    National Institute of Standards and Technology, “FIPS 203: Module- Lattice-Based Key-Encapsulation Mechanism Standard,” 2024. [Online]. Available: https://csrc.nist.gov/pubs/fips/203/final

    2024

  15. [15]

    FIPS 204: Module- Lattice-Based Digital Signature Standard,

    National Institute of Standards and Technology, “FIPS 204: Module- Lattice-Based Digital Signature Standard,” 2024. [Online]. Available: https://csrc.nist.gov/pubs/fips/204/final

    2024

  16. [16]

    KiD: A Hardware Design Frame- work Targeting Unified NTT Multiplication for CRYSTALS-Kyber and CRYSTALS-Dilithium on FPGA,

    S. Mandal and D. Basu Roy, “KiD: A Hardware Design Frame- work Targeting Unified NTT Multiplication for CRYSTALS-Kyber and CRYSTALS-Dilithium on FPGA,” inProc. 2024 37th Int. Conf. VLSI Design and 2024 23rd Int. Conf. Embedded Systems (VLSID), 2024, pp. 455–460

    2024

  17. [17]

    PQC-HA: A Framework for Prototyping and In-Hardware Evaluation of Post-Quantum Cryptography Hardware Accelerators,

    R. Sattel, C. Spang, C. Heinzet al., “PQC-HA: A Framework for Prototyping and In-Hardware Evaluation of Post-Quantum Cryptography Hardware Accelerators,” arXiv:2308.06621, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,

    J. White, Q. Fu, S. Hayset al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” inProc. 30th Conf. Pattern Lan- guages of Programs (PLoP), 2023

    2023

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.