REVIEW 5 major objections 6 minor 30 references
Accelerating Detailed Routing Convergence through Offline Reinforcement Learning
T0 review · 5 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read This paper claims that an offline reinforcement-learning agent can pick per-iteration routing cost weights so that an iterative detailed router converges in fewer iterations—5% fewer on average and up to 31% fewer on unseen circuits—transla
desk verdict A real and useful new application of offline RL to detailed routing, with a clean held-out speedup — but the missing ispd19_test5 row has to be explained before the headline numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Per-iteration dynamic selection of the router's four cost weights, driven by a conservative Q-learning (CQL) agent trained offline on perturbation-sampled trajectories; the agent maps a compact state—per-partition and total DRV counts, maximum and neighbor DRVs, wirelength, current weights, and design-level statistics—to the next weight vector, with reward combining DRV improvement, iteration penalty, convergence bonus, and a stuck penalty.
What would settle it
Build two test designs with identical aggregate state (same per-partition and total DRV counts, wirelength, weights, design stats) but different spatial layouts of violations; run the learned policy on both. If convergence iterations or final DRVs diverge materially, the state abstraction is incomplete and the transfer claim weakens.
Extended reading notes
Core claim
The claim, stated in the authors' terms, is that an offline conservative Q-learning policy can replace the static or iteration-indexed cost-weight schedule of a detailed router and reliably drive faster convergence. Trained on perturbation-sampled sequences from 17 designs, the policy decides, at the end of each rip-up-and-reroute iteration, the next values of four cost weights (DRV cost, marker cost, fixed-shape cost, marker decay) from a state of aggregate DRV and wirelength statistics. On ten test circuits excluded from training, the policy used 5% fewer iterations on average and up to 31% fewer, speeding total runtime by 1.56x on average and up to 3.01x; final DRV counts were never worse
Load-bearing premise
The policy's decisions assume that the complete effect of a chosen weight vector on future convergence is captured by aggregate state—per-partition DRV counts, total DRVs, wirelength, current weights, and design statistics—so two designs with the same aggregate profile will respond identically to the same weights; exact violation locations are treated as irrelevant.
Editorial extensions
If this is right
- A router's static cost weights are suboptimal: adopting a per-iteration learned weight policy yields 1.56x average and up to 3.01x runtime speedup on held-out circuits with equal or better DRV counts.
- The approach is router-agnostic in the sense that it wraps the existing rip-up-and-reroute loop; any router that assigns costs to violations can host the same agent.
- The policy generalizes across technology nodes, so a weight model trained on older designs can give speedups when first deployed on a newer technology.
- Because the per-run inference cost is about 2 seconds, the speedup is most reliable on designs that take many iterations; already-fast designs may see slowdowns from overhead.
Reading between the lines
- The state excludes exact DRV locations by design; the authors' success suggests aggregate congestion features may be sufficient on these benchmarks, but a spatial-feature variant is the natural next experiment.
- The 2s inference overhead explains the small design slowdowns; a practical deployment could gate the learned policy to designs whose baseline runtime exceeds a threshold.
- The learned weight paths differ from the best seen in training data, hinting that many weight trajectories lead to equivalent convergence; this could be exploited to co-optimize wirelength or other objectives.
- Reward omitted wirelength, matching the small (<1%) wirelength penalty; including a wirelength term in the reward would test the trade-off between convergence speed and route quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes using offline reinforcement learning (conservative Q-learning, CQL) to dynamically select per-iteration cost weights (DRC cost, marker cost, fixed-shape cost, marker decay) inside the OpenROAD detailed router. The authors generate training data by perturbing baseline weight schedules across 17 designs from the OpenROAD Design Suite and ISPD18, then train a CQL policy to choose weights that minimize the number of rip-up-and-reroute iterations until zero DRV convergence. They evaluate on nine reported ISPD19 benchmarks held out during training, claiming an average 1.56x runtime speedup (up to 3.01x) and equal or better DRV counts in all reported cases, along with signs of cross-technology generalization.
Significance. If the result holds, the paper offers a practical and relatively simple way to accelerate a state-of-the-art open-source detailed router by replacing a static weight schedule with a learned dynamic one, and it does so with a clean held-out evaluation protocol: ISPD19 is excluded from training. The use of an offline RL method (CQL) with perturbation-based data generation is a sensible fit for the expensive router environment. The authors also commit to open-sourcing code, which would materially strengthen reproducibility. However, the strength of the claim depends on the completeness and statistical rigor of Tables II-III, and several load-bearing details (exact reward specification, technology-node identities, and state-sufficiency justification) are not provided in the manuscript.
major comments (5)
- [Table III / Section V.C] Table III reports only nine of the ten ISPD19 test benchmarks: ispd19_test5 is absent with no explanation, while the abstract claims 'maintaining or improving the DRV count in all cases' and Section V.A says ten unseen ISPD19 benchmarks were used. Since per-design runtime differences range from -6.98% to +66.84%, omitting test5 could materially change the average speedup and the 'all cases' DRV guarantee. Please add the missing row, explain any exclusion, and recompute totals and headline claims including test5.
- [Section IV.B] The reward function is described only qualitatively ('DRV Improvement Bonus', 'Iteration Penalty', 'Convergence Bonus', 'Stuck Penalty' with importance labels). No formulas, coefficients, or reward scales are given, despite Section IV.B stating the reward was clipped and normalized with tanh. Because the headline metric (fewer iterations) is the training objective, the exact reward specification is essential for reproducibility and for assessing whether the reward balances convergence speed against DRV quality. Please provide the full quantitative reward definition.
- [Section IV.B] The state representation deliberately excludes fine-grained spatial coordinates of DRVs, retaining only aggregate per-partition DRV counts, totals, wirelength, and current weights. The cross-design transfer claim assumes these aggregates are sufficient to determine which weight vector minimizes future iterations. This is not tested: two designs with identical aggregate state may respond differently to the same weights if local congestion structure matters. Please provide an ablation or feature-importance analysis, or a concrete experiment that varies the state granularity, to justify this assumption.
- [Section V.A / Abstract / Section VI] The claim that learning generalizes across technologies is load-bearing for the abstract and conclusion, but no technology node is identified for the ISPD19 benchmarks or for the training designs. The only stated basis is that 'ISPD19 is in a technology node that the model had not seen before' (Section VI). Please specify the technology nodes of both training and test sets, and state whether any design-rule differences beyond node identity (e.g., routing layers, pitch, minimum-area rules) are relevant to the claimed transfer.
- [Section V.A / Table III] Runtime and iteration results are said to be averaged over 10 runs, but no variance, standard deviation, or confidence intervals are reported. On designs where the observed difference is small (e.g., ispd19_test3: +1.89%, ispd19_test8: +0.66%, ispd19_test7: -0.63%), run-to-run noise could change the sign. Please report per-design statistics (mean plus spread) and, where meaningful, a paired test to support the per-design conclusions.
minor comments (6)
- [Throughout] Typos: 'Dijkstra' is misspelled as 'Djiksta' in Section II.A; 'enchancements' should be 'enhancements' in Section II.A. Also, the IEEE reference for OpenROAD [5] is a GOMATECH paper, which may not be the standard citation; consider using the DAC/ICCAD version.
- [Section V.A] The manuscript says 'across 29 standard and benchmark circuits' but Table II has 10 seen and Table III has 9 reported unseen rows, totaling 19; the count discrepancy needs clarification.
- [Table II] The column 'Min.' is used for the fastest converging sequence in the training data but is not defined in the table caption or text; please define it explicitly and clarify how it was chosen.
- [Section V.C / Figure 5] Figure 5 plots DRV convergence for the OpenROAD Design Suite, but the y-axis is log-scale and labels are partially garbled in the text extraction; ensure the figure is legible and axis labels are explicit.
- [Section IV.A / Equation (1)] The CQL objective is written in a nonstandard form with a 'log-sum-exp' term; please verify the notation matches the cited reference [9] and define all symbols (e.g., \hat{B}^\pi, the Bellman operator) explicitly.
- [Section III] The static design characteristics list includes 'number of routing layers' but it is not shown how these are incorporated into the state representation in Section IV.B; clarify their role.
Circularity Check
No significant circularity: held-out ISPD19 evaluation is an independent out-of-sample test; reward alignment is a stated optimization, not a hidden circular prediction.
full rationale
The paper's central claim is an empirical speedup on ISPD19 benchmarks that were explicitly excluded from training. Section III states 'ISPD19 circuits were excluded from training data to be used as test data,' and Section V.A confirms '10 being unseen test benchmarks from ISPD19.' The RL policy selects routing cost weights to minimize iterations, and the headline metric is iteration count/runtime on those unseen benchmarks. This is a clean out-of-sample evaluation: the training data comes from OpenROAD Design Suite and ISPD18 designs, not from the test set. The reward function includes iteration penalty and convergence bonus, so the optimization target matches the evaluation metric, but that is not circularity — it is the defined goal of the method. No parameter is fitted to the ISPD19 results and then reported as a prediction. Self-citations to OpenROAD and TritonRoute are used to describe the baseline router, not to justify the learned policy's effectiveness. The absence of ispd19_test5 from Table III and the lack of variance bars are completeness/robustness concerns, but they do not make the derivation circular. Therefore, no circular step can be exhibited.
Assumptions & free parameters
free parameters (9)
- actor learning rate =
1.0e-3
- critic learning rate =
4.0e-3
- conservative weight (alpha) =
8.0e-1
- batch size =
128
- initial temperature =
1.43298e-1
- tau (Polyak) =
1.97555e-3
- reward component weights
- perturbation mix ratio =
~60% perturbation / 40% Sobol random
- early stopping thresholds =
critic loss 100x; action diff > 1.0
assumptions (5)
- domain assumption The next-iteration optimal routing weights depend only on the aggregate state vector; exact violation coordinates are unnecessary.
- domain assumption A hand-designed reward (DRV improvement, iteration penalty, convergence bonus, stuck penalty) is a valid proxy for wall-clock convergence time and final solution quality.
- domain assumption Perturbation sampling around baseline weights plus Sobol random sampling on 17 training designs covers the weight/design distribution needed for unseen ISPD19 designs.
- domain assumption OpenROAD's routing behavior is deterministic enough that a sequence of weight vectors determines the convergence trajectory, so policy evaluation on 10-run averages is meaningful.
- standard math CQL's conservative regularization controls out-of-distribution Q-values as claimed in Kumar et al. [9].
Cite this review
Pith. "Pith review of Accelerating Detailed Routing Convergence through Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/I64WRTGV
@misc{pith2026251203594,
author = {Pith},
title = {Pith review of: Accelerating Detailed Routing Convergence through Offline Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/I64WRTGV}},
note = {Machine review of arXiv:2512.03594}
}
read the original abstract
Detailed routing remains one of the most complex and time-consuming steps in modern physical design due to the challenges posed by shrinking feature sizes and stricter design rules. Prior detailed routers achieve state-of-the-art results by leveraging iterative pathfinding algorithms to route each net. However, runtimes are a major issue in detailed routers, as converging to a solution with zero design rule violations (DRVs) can be prohibitively expensive. In this paper, we propose leveraging reinforcement learning (RL) to enable rapid convergence in detailed routing by learning from previous designs. We make the key observation that prior detailed routers statically schedule the cost weights used in their routing algorithms, meaning they do not change in response to the design or technology. By training a conservative Q-learning (CQL) model to dynamically select the routing cost weights which minimize the number of algorithm iterations, we find that our work completes the ISPD19 benchmarks with 1.56x average and up to 3.01x faster runtime than the baseline router while maintaining or improving the DRV count in all cases. We also find that this learning shows signs of generalization across technologies, meaning that learning designs in one technology can translate to improved outcomes in other technologies.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
An algorithm for path connections and its applications,
C. Y . Lee, “An algorithm for path connections and its applications,”IRE Transactions on Electronic Computers, vol. EC-10, no. 3, pp. 346–365, 1961
1961
-
[2]
An interactive maze router with hints,
M. H. Arnold and W. S. Scott, “An interactive maze router with hints,” inProceedings of the 25th ACM/IEEE Design Automation Conference, ser. DAC ’88. Washington, DC, USA: IEEE Computer Society Press, 1988, p. 672–676
1988
-
[3]
A solution to line-routing problems on the continuous plane,
D. W. Hightower, “A solution to line-routing problems on the continuous plane,” inProceedings of the 6th Annual Design Automation Conference, ser. DAC ’69. New York, NY , USA: Association for Computing Machinery, 1969, p. 1–24. [Online]. Available: https://doi.org/10.1145/800260.809014
arXiv 1969
-
[4]
Detailed routing by sparse grid graph and minimum-area-captured path search,
G. Chen, C.-W. Pui, H. Li, J. Chen, B. Jiang, and E. F. Y . Young, “Detailed routing by sparse grid graph and minimum-area-captured path search,” inProceedings of the 24th Asia and South Pacific Design Automation Conference, ser. ASPDAC ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 754–760. [Online]. Available: https://doi.org/10....
arXiv 2019
-
[5]
OpenROAD: Toward a Self-Driving, Open-Source Digital Layout Implementation Tool Chain,
T. Ajayi, D. Blaauw, T.-B. Chan, C.-K. Cheng, V . A. Chhabria, D. K. Choo, M. Coltella, S. Dobre, R. G. Dreslinski, M. Fogac ¸a, S. Hashemi, A. Hosny, A. B. Kahng, M. Kim, J. Li, Z. Liang, U. Mallappa, P. Penzes, G. Pradipta, S. Reda, A. Rovinski, K. Samadi, S. S. Sapatnekar, L. Saul, C. Sechen, V . Srinivas, W. Swartz, D. Sylvester, D. Urquhart, L. Wang,...
2019
-
[6]
INVITED: Toward an Open- Source Digital Flow: First Learnings from the OpenROAD Project,
T. Ajayi, V . A. Chhabria, M. Fogac ¸a, S. Hashemi, A. Hosny, A. B. Kahng, M. Kim, J. Lee, U. Mallappa, M. Neseem, G. Pradipta, S. Reda, M. Saligane, S. S. Sapatnekar, C. Sechen, M. Shalan, W. Swartz, L. Wang, Z. Wang, M. Woo, and B. Xu, “INVITED: Toward an Open- Source Digital Flow: First Learnings from the OpenROAD Project,” in Proc. DAC, 2019
2019
-
[7]
Provably efficient offline reinforcement learning with perturbed data sources,
C. Shi, W. Xiong, C. Shen, and J. Yang, “Provably efficient offline reinforcement learning with perturbed data sources,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 31 353–31 388
2023
-
[8]
Robust deep reinforcement learning with adaptive adversarial perturbations in action space,
Q. Liu, Y . Kuang, and J. Wang, “Robust deep reinforcement learning with adaptive adversarial perturbations in action space,” in2024 Inter- national Joint Conference on Neural Networks (IJCNN), 2024, pp. 1–8
2024
Show all 30 references
-
[9]
Conservative q-learning for offline reinforcement learning,
A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” inAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1179–1191
2020
-
[10]
Bidirectional heuristic search reconsidered,
H. Kaindl and G. Kainz, “Bidirectional heuristic search reconsidered,” J. Artif. Int. Res., vol. 7, no. 1, p. 283–317, Dec. 1997
1997
-
[11]
Evaluation of beol design rule impacts using an optimal ilp-based detailed router,
K. Han, A. B. Kahng, and H. Lee, “Evaluation of beol design rule impacts using an optimal ilp-based detailed router,” inProceedings of the 52nd Annual Design Automation Conference, ser. DAC ’15. New York, NY , USA: Association for Computing Machinery, 2015
2015
-
[12]
Tritonroute: The open-source detailed router,
A. B. Kahng, L. Wang, and B. Xu, “Tritonroute: The open-source detailed router,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 40, no. 3, pp. 547–559, 2020
2020
-
[13]
Gridless pin access in detailed routing,
T. Nieberg, “Gridless pin access in detailed routing,” inProceedings of the 48th Design Automation Conference, ser. DAC ’11. New York, NY , USA: Association for Computing Machinery, 2011, p. 170–175
2011
-
[14]
Self-aligned double patterning lithography aware detailed routing with color preassignment,
Y . Ding, C. Chu, and W.-K. Mak, “Self-aligned double patterning lithography aware detailed routing with color preassignment,” vol. 36, no. 8, p. 1381–1394, Aug. 2017
2017
-
[15]
Overlay-aware detailed routing for self-aligned double patterning lithography using the cut process,
I.-J. Liu, S.-Y . Fang, and Y .-W. Chang, “Overlay-aware detailed routing for self-aligned double patterning lithography using the cut process,” in2014 51st ACM/EDAC/IEEE Design Automation Conference (DAC), 2014, pp. 1–6
2014
-
[16]
Detailed routing algorithms for advanced technology nodes,
M. Ahrens, M. Gester, N. Klewinghaus, D. Muller, S. Peyer, C. Schulte, and G. Tellez, “Detailed routing algorithms for advanced technology nodes,”Trans. Comp.-Aided Des. Integ. Cir. Sys., vol. 34, no. 4, p. 563–576, Apr. 2015
2015
-
[17]
Ispd 2018 initial detailed routing contest and benchmarks,
S. Mantik, G. Posser, W.-K. Chow, Y . Ding, and W.-H. Liu, “Ispd 2018 initial detailed routing contest and benchmarks,” inProceedings of the 2018 International Symposium on Physical Design, ser. ISPD ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 140–14...
2018
-
[18]
Ispd 2019 initial detailed routing contest and benchmark with advanced routing rules,
W.-H. Liu, S. Mantik, W.-K. Chow, Y . Ding, A. Farshidi, and G. Posser, “Ispd 2019 initial detailed routing contest and benchmark with advanced routing rules,” inProceedings of the 2019 International Symposium on Physical Design, ser. ISPD ’19. New York, NY , USA: Association ...
2019
-
[19]
Tritonroute: An initial detailed router for advanced vlsi technologies,
A. B. Kahng, L. Wang, and B. Xu, “Tritonroute: An initial detailed router for advanced vlsi technologies,” in2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 2018, pp. 1–8
2018
-
[20]
Explainable drc hotspot prediction with random forest and shap tree explainer,
W. Zeng, A. Davoodi, and R. O. Topaloglu, “Explainable drc hotspot prediction with random forest and shap tree explainer,” in2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2020, pp. 1151–1156
2020
-
[21]
Pin accessibility and routing congestion aware drc hotspot prediction for designs in advanced technology nodes with consolidated practical applicability and sustainability,
H. Park, K. Baek, S. Kim, K. Choi, and T. Kim, “Pin accessibility and routing congestion aware drc hotspot prediction for designs in advanced technology nodes with consolidated practical applicability and sustainability,”IEEE Transactions on Computer-Aided Design of Integrated...
2024
-
[22]
Drc hotspot prediction at sub-10nm process nodes using customized convolutional network,
R. Liang, H. Xiang, D. Pandey, L. Reddy, S. Ramji, G.-J. Nam, and J. Hu, “Drc hotspot prediction at sub-10nm process nodes using customized convolutional network,” inProceedings of the 2020 In- ternational Symposium on Physical Design. New York, NY , USA: Association for Compu...
2020
-
[23]
Reinforcement learning guided detailed routing for custom circuits,
H. Chen, K.-C. Hsu, W. J. Turner, P.-H. Wei, K. Zhu, D. Z. Pan, and H. Ren, “Reinforcement learning guided detailed routing for custom circuits,” inProceedings of the 2023 International Symposium on Physical Design, ser. ISPD ’23. New York, NY , USA: Association for Computing ...
2023
-
[24]
Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences,
S. Mishra and T. K. Rusch, “Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences,”SIAM Journal on Numerical Analysis, vol. 59, no. 3, pp. 1811–1834, 2021
2021
-
[25]
Sampling based on sobol′ sequences for monte carlo techniques applied to building simulations,
S. Burhenne, D. Jacob, and G. P. Henze, “Sampling based on sobol′ sequences for monte carlo techniques applied to building simulations,” inProceedings of Building Simulation 2011: 12th Conference of IBPSA, ser. Building Simulation, vol. 12. Sydney, Australia: IBPSA, November 2...
2011
-
[26]
Bridging Academic Open-Source EDA to Real-World Usability,
A. Rovinski, T. Ajayi, M. Kim, G. Wang, and M. Saligane, “Bridging Academic Open-Source EDA to Real-World Usability,” inProc. ICCAD, November 2020, pp. 1–7
2020
-
[27]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems,
S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” 2020
2020
-
[28]
Reinforcement learning: An introduction,
R. Sutton and A. Barto, “Reinforcement learning: An introduction,” IEEE Transactions on Neural Networks, vol. 9, no. 5, pp. 1054–1054, 1998
1998
-
[29]
d3rlpy: An offline deep reinforcement learning library,
T. Seno and M. Imai, “d3rlpy: An offline deep reinforcement learning library,” 2022. [Online]. Available: https://arxiv.org/abs/2111.03788
2022 arXiv
-
[30]
Hyperparameter selection for offline reinforcement learning,
T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas, “Hyperparameter selection for offline reinforcement learning,” 2020. [Online]. Available: https: //arxiv.org/abs/2007.09055
2020 arXiv
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.