REVIEW 3 major objections 5 minor 25 references
From Tool Invocation to Source-Mechanism Exploration: Protected White-Box DSE for Open-Source EDA
T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Bounded source-level search inside a protected evaluator cuts post-DPL wirelength by 1.78% on average (vs 0.38% for knob-only BO-DSE) and repairs hard cut-row legality failures.
desk verdict Solid empirical systems paper with a real confound: the warm-start overlap makes the headline gain over BO-DSE hard to attribute; worth refereeing with ablations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the protected evaluator contract coupled with mechanism-level evidence records. The evaluator fixes the measurement path and validity checks, so a source edit can only be credited if it passes build, legality, metric, consistency, liveness, and runtime checks. Around it, ReviewDSE organizes search as a Teacher–Student loop: the Teacher plans source starts and routes using frozen Level 1 calibration evidence plus target-local records; Students make bounded edits in private workspaces; every complete candidate is evaluated, and the Teacher assigns review decisions (parent promotion, mechanism evidence, repair lead, negative evidence, rejection). The reusable knowl
What would settle it
Run ReviewDSE on target designs from chip families absent from Level 1 calibration (e.g., calibrate only on AES/JPEG/SWERV, then target a DSP block or a different RISC-V core not in those families) and compare mean HPWL reduction to BO-DSE; if the advantage collapses to near 0.38%, the reported gain is mostly warm-start memorization rather than general source-mechanism exploration.
Extended reading notes
Core claim
The central discovery is that for open-source EDA, the DSE boundary can be drawn inside the optimizer source rather than at its public interface: bounded source-level mechanisms—legalization policies, DPO kernels, handoff state—are searchable design choices, provided every complete candidate is rebuilt, run through a fixed protected evaluator, and reviewed for validity, liveness, full-flow QoR, and legality. ReviewDSE instantiates this on OpenROAD detailed placement, using calibration to build reusable method evidence and source-start branches, then a Teacher–Student loop to specialize source for each target. Across nine targets, it reports a 1.78% mean post-DPL HPWL reduction under a 2× run
Load-bearing premise
The load-bearing premise is that the Level 1 warm-start evidence, built from calibration designs in the AES, JPEG, and SWERV families, transfers to target designs from those same families without inflating the measured improvement; if that transfer is not representative, the 1.78% versus 0.38% comparison overstates the power of source-mechanism exploration on genuinely new chip architectures.
Editorial extensions
If this is right
- Open-source EDA flows can be specialized per design: the effective search space for DSE includes internal mechanisms, and this yields larger, more consistent QoR gains than knob tuning on already-feasible instances (e.g., 1.78% vs 0.38% mean HPWL).
- Stage-local QoR is not a safe promotion criterion: a legalizer can cut post-legalization HPWL by 14.96% yet worsen final post-DPL HPWL by 20.96% after DPO and mirroring, so full-flow review is necessary.
- Source-mechanism exploration can repair strict legality on hard cut-row stress tests where the fixed, evaluated flows fail or time out; ReviewDSE passed all nine patterns.
- Runtime-aware selection keeps most of the wirelength gain at near-default cost (1.68% at 1.11× runtime), showing the approach can be tuned for overhead-sensitive design flows.
- The method is a per-target specialization procedure, not a one-shot universal optimizer rewrite: every candidate is rebuilt, validated, and promoted only for its own target under the protected evaluator.
Reading between the lines
- Porting the protected-evaluator contract to other OpenROAD stages (routing, clock tree synthesis, logic synthesis) could turn per-target source specialization into a general open-EDA service, since the mechanism-evidence record is not placement-specific.
- The Teacher–Student loop could be made cost-adaptive: using the target-local record to skip early iterations or to freeze a branch once gains plateau might cut the 2.15B-token-per-target budget while keeping most of the 1.78%.
- If calibration were deliberately mismatched (e.g., calibrate only on small blocks, target large SoCs), the framework's behavior would reveal how much of the gain is genuine mechanism discovery versus warm-start memorization; that is a clean, testable extension.
- The legality-recovery results suggest a broader use case: source-mechanism exploration as a repair pass for hard physical-design failures, not just QoR improvement on feasible cases.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ReviewDSE, a protected white-box design-space exploration framework for open-source EDA. Instead of tuning only public knobs, ReviewDSE searches bounded source-level mechanisms inside staged optimizers, using a protected evaluator and a Teacher–Student agent structure. Level 1 builds frozen method evidence and source-start branches from calibration designs; Level 2 uses these as warm-start material for target-specific exploration. The instantiation on OpenROAD detailed placement reports a mean post-DPL HPWL reduction of 1.78% across nine targets under a 2× runtime gate, compared with 0.38% for black-box BO-DSE, a runtime-aware selection retaining 1.68% at 1.11× runtime, stage-composability counterexamples, and legality recovery on hard cut-row patterns. The paper includes a detailed artifact appendix with pinned dependencies and acceptance criteria.
Significance. If the quantitative claims survive controlled comparison, the paper would be a meaningful step beyond knob-based DSE: it would show that bounded source-level mechanism exploration inside an open-source EDA tool can produce consistent QoR improvements and repair legality failures that fixed-source flows cannot. The contribution is timely given recent LLM-based code-evolution work in EDA. Strengths include the protected-evaluator contract, liveness and source/binary/metric consistency checks, strict legality verification, and an unusually complete artifact appendix with reproducibility scripts and acceptance criteria. However, the headline empirical claim is not yet established because the central comparison does not control for warm-start transfer or stochastic variance, and the baseline search budget is not comparable to the reported token cost of ReviewDSE.
major comments (3)
- [§3.3, §4.1, Table 4] The headline comparison conflates search-space expressiveness with warm-start transfer. Level 1 calibration uses constructed Nangate45 cases derived from JPEG (UTIL=90), AES (UTIL=70), and a SWERV wrapper (UTIL=60). Level 2 targets include AES N45, JPEG N45, and SWERV N45 — the same design families and the same technology node — plus ASAP7 versions of the same families. The paper notes that instances are disjoint, but mechanism-level evidence (which source branches reduce HPWL, which are fragile, which are expensive) can transfer across instances of the same family. BO-DSE has no analogous warm-start and begins from the default flow. The reported 1.78% vs. 0.38% gap may therefore reflect the transferred prior rather than source-level search per se. To support the central attribution, the paper should run ReviewDSE without Level 1 (cold start), calibrate on disjoint design families, or gi
- [§4.1, Table 4, Artifact A.3] The pipeline is stochastic, but the reported numbers are single-run outcomes with no variance or confidence intervals. Artifact A.3 explicitly states that repeated searches may produce different edits. The difference between 1.78% and 0.38% could be within run-to-run noise, particularly for small designs such as Ibex ASAP7 (−0.60% vs. −0.10%). The paper should report repeated runs (possibly at reduced budget) or at least per-target variability across seeds, and state whether the 1.78% mean is robust. The artifact's acceptance tolerance of 0.06 percentage points does not address stochasticity; it only checks reproduction of the originally logged run.
- [§4.2, Table 4, §5] The baseline budget is not comparable. BO-DSE receives 400 TPE trials per case, while ReviewDSE consumes an average of 2.15B logged tokens (0.10B active) per target. The paper frames the result as 'beyond public-knob BO-DSE,' but the comparison varies both the search space and the computational budget. If BO-DSE were given comparable compute, or ReviewDSE were constrained to a comparable number of evaluations or token budget, the QoR gap could change substantially. A cost-controlled comparison is needed to separate the benefit of source-level mechanisms from the benefit of a much larger search budget.
minor comments (5)
- [Figure 4 and Artifact A.6] Artifact A.6 says Figure 4 must contain 96 observed points and mark three unavailable SWERV points, but the main text describes 9 designs × 10 iterations = 90 points and never mentions unavailable points. Please reconcile the expected point count and explain any missing points in the caption.
- [Artifact A.3] The sentence 'For Table 5 only, core utilization is set to 70, 90, and 60 for AES, JPEG, and SWERV' is confusing because Table 5 is the stage-local counterexample table, not the Level 1 calibration set. Clarify whether these utilizations apply to the calibration designs or to the dense Table 5 variants.
- [Table 4 caption] The 'Tokens' column reports entries such as 1.62/0.08 with a note that values are in billions, but the caption should define the two components (logged vs. active uncached tokens) directly, since the main text only gives the aggregate in §5.
- [Footnote 3, §4.2] The Ariane133 N45 diagnostic reports averaged improvements of +1.52%/1.07× and −3.26%/0.71× without stating the number of diagnostic runs. Please provide the sample size or mark these as illustrative.
- [§5] The limitations discussion covers evaluator scope and search cost, but does not mention the warm-start confound with the baseline or the lack of statistical uncertainty quantification. These should be acknowledged and addressed in the revision.
Circularity Check
No significant circularity: the 1.78% result is a measured evaluation outcome, not a fitted or self-referential constant.
full rationale
ReviewDSE's central claim is an empirical benchmark outcome on nine target tasks, produced by rebuilding and re-evaluating each candidate on its own target under a fixed protected evaluator (Eq. 3, §3.5, Algorithms 1–2). There is no derivation chain in which an output is fed back as an input: Level 1 warm-start evidence is frozen and read-only during Level 2 (Algorithm 2 ends with 'W remains frozen'), and every reported candidate must pass build, legality, metric, source/binary consistency, liveness, and runtime checks before promotion. The G_HR selection score (§4.1) is a post-hoc selection rule over already-evaluated candidates, not a fitted parameter used to generate the reported HPWL numbers. The closest concern is that Level 1 calibration uses AES/JPEG/SWERV Nangate45 families that also appear among Level 2 targets; the paper discloses this ('They may share benchmark families, but no target placement, target metric, target route, or target-specific candidate is used in Level 1'), and it is a transferability/generalizability concern, not circularity, because the reported deltas are measured on independent target placements rather than read off calibration. No load-bearing self-citation or uniqueness theorem is invoked; citations are to prior DSE and LLM-agent literature for context, and the artifact appendix explicitly states 'No expected value is copied into an observed field.' Therefore no circular step is exhibitable by the paper's own equations or construction.
Assumptions & free parameters
free parameters (4)
- 2x runtime gate =
2.0
- G_HR runtime penalty threshold and penalty shape =
1.10x; square-root ramp to 2x
- Calibration design utilizations =
JPEG=90, AES=70, SWERV=60
- BO-DSE trial budget =
400 trials per case
assumptions (5)
- domain assumption HPWL after DPL is an adequate primary objective for design-quality comparison.
- domain assumption The protected evaluator's fixed command sequence and legality/liveness/consistency checks cannot be bypassed by source edits.
- domain assumption OpenROAD detailed placement is a representative staged optimizer for generalizing the framework.
- domain assumption GPT-5.x agents with xhigh reasoning can produce legal, mechanism-preserving source edits under prompt guidance.
- domain assumption Level 1 calibration instances sharing benchmark families with targets do not leak target-specific improvements into warm-start.
Cite this review
Pith. "Pith review of From Tool Invocation to Source-Mechanism Exploration: Protected White-Box DSE for Open-Source EDA." pith.science (2026). https://pith.science/paper/TRCR2LKH
@misc{pith2026260711294,
author = {Pith},
title = {Pith review of: From Tool Invocation to Source-Mechanism Exploration: Protected White-Box DSE for Open-Source EDA},
year = {2026},
howpublished = {\url{https://pith.science/paper/TRCR2LKH}},
note = {Machine review of arXiv:2607.11294}
}
abstract
Open-source EDA tools allow design-space exploration (DSE) to move beyond public knobs and into bounded source-level mechanisms inside staged optimizers. We present ReviewDSE, a protected white-box DSE framework that explores such mechanisms for a target design. ReviewDSE evaluates complete source candidates under a protected evaluator and records reusable search knowledge as reviewed mechanism-level evidence. It first constructs method evidence and source-start branches from calibration designs, then uses these fixed warm-start products to initialize target-case exploration under Teacher review and full-flow validation. We instantiate ReviewDSE on OpenROAD detailed placement as a representative staged open-source EDA optimizer. Across nine target tasks, ReviewDSE reduces final post-DPL half-perimeter wirelength (HPWL) by 1.78\% on average under a 2$\times$ runtime gate, compared with 0.38\% for public-knob black-box DSE. A runtime-aware ReviewDSE selection retains a 1.68\% reduction at 1.11$\times$ runtime, and full-flow review exposes stage-composability failures while source-mechanism exploration repairs hard cut-row legality failures.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Techniques for cad tool parameter auto- tuning in physical synthesis: a survey,
H. Geng, T. Chen, Q. Sun, and B. Yu, “Techniques for cad tool parameter auto- tuning in physical synthesis: a survey, ” in2022 27th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2022, pp. 635–640
2022
-
[2]
CAD tool design space exploration via bayesian optimiza- tion,
Y. Ma, Z. Yu, and B. Yu, “CAD tool design space exploration via bayesian optimiza- tion, ” in2019 ACM/IEEE 1st Workshop on Machine Learning for CAD (MLCAD), 2019, pp. 1–6
2019
-
[3]
The OpenROAD Project, https://theopenroadproject.org/, 2026
2026
-
[4]
Alphaevolve: A coding agent for scientific and algorithmic discovery,
A. Novikov, N. V˜u, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shi- robokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabianet al., “Alphaevolve: A coding agent for scientific and algorithmic discovery, ”arXiv preprint arXiv:2506.13131, 2025
arXiv 2025
-
[5]
Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization,
H. Assumpção, D. Ferreira, L. Campos, and F. Murai, “Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization, ”arXiv preprint arXiv:2510.14150, 2025
arXiv 2025
-
[6]
Autonomous code evolution meets np- completeness,
C. Yu, R. Liang, C.-T. Ho, and H. Ren, “Autonomous code evolution meets np- completeness, ”arXiv preprint arXiv:2509.07367, 2025
arXiv 2025
-
[7]
EvoPlace: Evolution of Op- timization Algorithms for Global Placement via Large Language Models,
X. Yao, J. Jiang, Y. Zhao, P. Liao, Y. Lin, and B. Yu, “EvoPlace: Evolution of Op- timization Algorithms for Global Placement via Large Language Models, ”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2026
2026
-
[8]
Gr-evolve: Design-adaptive global routing via llm-driven algorithm evo- lution,
T. Jafri, “Gr-evolve: Design-adaptive global routing via llm-driven algorithm evo- lution, ” Master’s thesis, Arizona State University, 2026
2026
Show all 25 references
-
[9]
Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC,
C. Yu and H. Ren, “Autonomous Evolution of EDA Tools: Multi-Agent Self-Evolved ABC, ” inACM/IEEE Design Automation Conference (DAC), 2026, pp. 1–6
2026
-
[10]
Automated QoR improvement in OpenROAD with coding agents,
A. Ghose, J. Jang, A. B. Kahng, and J. Lee, “Automated QoR improvement in OpenROAD with coding agents, ”arXiv preprint arXiv:2601.06268, 2026
2026
-
[11]
AutoDSE: Enabling software programmers to design efficient fpga accelerators,
A. Sohrabizadeh, C. H. Yu, M. Gao, and J. Cong, “AutoDSE: Enabling software programmers to design efficient fpga accelerators, ”ACM Transactions on Design Automation of Electronic Systems, vol. 27, no. 4, pp. 1–27, 2022
2022
-
[12]
BOiLS: Bayesian optimisation for logic synthesis,
A. Grosnit, C. Malherbe, R. Tutunov, X. Wan, J. Wang, and H. Bou Ammar, “BOiLS: Bayesian optimisation for logic synthesis, ” in2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2022, pp. 1193–1196
2022
-
[13]
VLSI placement parameter optimization using deep reinforcement learning,
A. Agnesina, K. Chang, and S. K. Lim, “VLSI placement parameter optimization using deep reinforcement learning, ” inProceedings of the 39th International Con- ference on Computer-Aided Design, 2020, pp. 1–9
2020
-
[14]
AutoDMP: Automated DREAMPlace-based macro placement,
A. Agnesina, P. Rajvanshi, T. Yang, G. Pradipta, A. Jiao, B. Keller, B. Khailany, and H. Ren, “AutoDMP: Automated DREAMPlace-based macro placement, ” in Proceedings of the 2023 International Symposium on Physical Design, 2023, pp. 149–157
2023
-
[15]
RTLLM: An open-source benchmark for design rtl generation with large language model,
Y. Lu, S. Liu, Q. Zhang, and Z. Xie, “RTLLM: An open-source benchmark for design rtl generation with large language model, ” in2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), 2024, pp. 722–727
2024
-
[16]
VerilogEval: Evaluating large language models for verilog code generation,
M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: Evaluating large language models for verilog code generation, ” in2023 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2023, pp. 1–8
2023
-
[17]
VeriGen: A large language model for verilog code generation,
S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri, and S. Garg, “VeriGen: A large language model for verilog code generation, ”ACM Transactions on Design Automation of Electronic Systems, 2024
2024
-
[18]
RTLCoder: Fully open-source and efficient llm-assisted rtl code generation technique,
S. Liu, W. Fang, Y. Lu, J. Wang, Q. Zhang, H. Zhang, and Z. Xie, “RTLCoder: Fully open-source and efficient llm-assisted rtl code generation technique, ” 2023
2023
-
[19]
AutoChip: Au- tomating hdl generation using llm feedback,
S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “AutoChip: Au- tomating hdl generation using llm feedback, ” 2023
2023
-
[20]
LLM4DV: Using large language models for hardware test stimuli generation,
Z. Zhang, G. Chadwick, H. McNally, Y. Zhao, and R. Mullins, “LLM4DV: Using large language models for hardware test stimuli generation, ” 2023
2023
-
[21]
AutoBench: Au- tomatic testbench generation and evaluation using llms for hdl design,
R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, “AutoBench: Au- tomatic testbench generation and evaluation using llms for hdl design, ” in2024 ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD), 2024, pp. 1–10
2024
-
[22]
ChatEDA: A large language model powered autonomous agent for eda,
Z. He, H. Wu, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “ChatEDA: A large language model powered autonomous agent for eda, ” in2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD), 2023, pp. 1–6
2023
-
[23]
ChipNeMo: Domain-adapted llms for chip design,
M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu, B. Bhaskaran, B. Catanzaro, A. Chaudhuri, S. Clay, B. Dally, L. Dang, P. Deshpande, S. Dhodhi, S. Halepete, E. Hill, J. Hu, S. Jain, A. Jindal, B. Khailany, G. Koka...
2023
-
[24]
Customized retrieval augmented generation and benchmarking for eda tool documentation qa,
Y. Pu, Z. He, T. Qiu, H. Wu, and B. Yu, “Customized retrieval augmented generation and benchmarking for eda tool documentation qa, ” in2024 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2024
2024
-
[25]
Ask-EDA: A design assistant empowered by llm, hybrid rag and abbreviation de-hallucination,
L. Shi, M. Kazda, B. Sears, N. Shropshire, and R. Puri, “Ask-EDA: A design assistant empowered by llm, hybrid rag and abbreviation de-hallucination, ” in2024 IEEE LLM Aided Design Workshop (LAD), 2024. Zhiyu Zheng, Yiming Du, Ziyi Wang, and Zhiang Wang A Artifact Appendix Wenj...
2024
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.