REVIEW 5 major objections 5 minor 15 references
OmniRouting: A Semantic-Coupled Multimodal Benchmark for Constraint-Aware Spatial Reasoning in PCB Routing
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that current large multimodal models cannot reliably route printed circuit boards: the best zero-shot model connects and clears design-rule checks on 12.60% of nets, versus 93.58% for human engineers.
desk verdict A genuinely useful large-scale PCB routing benchmark with a clear (if overstated) negative result; the human-reference inconsistency needs a fix, but the core finding stands. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the OmniRouting benchmark itself: 1,681 industrial-grade schematic-coupled PCB designs with engineer-verified routable placements, complete reference routes (494,349 wire segments, 84,603 vias, 4,314 copper polygons), and a four-task protocol covering geometric routing, design-rule-aware routing, electrical functionality, and tool-augmented agentic routing. Evaluation separates absolute validity, measured by Net Routability Ratio, Pad-to-Pad Routability Ratio, open and short counts, and clearance violations, from reference-relative agreement, measured by Multi-Scale Graph Similarity at coarse and fine resolutions. The agentic setup equips models with layout visualization, overlap checking, and routability verification tools in iterative rounds, and the benchmark data are parsed from EAGLE XML files by a PCB EDA engine the authors built.
What would settle it
Re-run a stratified random sample of the 1,681 reference layouts through an independent commercial DRC tool that also checks trace width, via rules, and boundary distance; if the human-engineered reference routes show material numbers of violations that OmniRouting's clearance-only DRC misses, then the headline LMM-versus-human gap is partly an artifact of the evaluator.
Extended reading notes
Core claim
The paper's central claim is that current large multimodal models fundamentally lack the topological planning and constraint-aware navigation capabilities required for PCB routing. On 1,681 real-world, schematic-coupled boards with engineer-verified routable placements, the strongest zero-shot model produces DRC-clean connections for only 12.60% of nets and 15.41% of required pad-to-pad connections, while the human engineer reference scores 93.58% and 95.91%. Tool-augmented agentic refinement over multiple rounds brings the best model to 27.98% NRR and 30.34% PRR, still far below both the human reference and a classical open-source algorithmic router that reaches 56.20% NRR and 71.37% PRR without any learned model.
Load-bearing premise
The conclusion depends on the assumption that the human-engineered reference routings are complete, correct ground truths and that the benchmark's clearance-focused DRC captures every constraint that matters; the reference routes themselves score only 93.58% NRR and 95.91% PRR, and the paper notes no independent DRC re-verification after XML parsing.
Editorial extensions
If this is right
- If the benchmark results hold, no current large multimodal model can be trusted to produce a complete, conflict-free, electrically correct PCB route on a realistic board without substantial external tooling.
- Agentic tools improve local legality and corridor choice but do not close the gap: the best tool-augmented result still trails the human reference by more than 65 percentage points of NRR.
- Classical algorithmic routers remain far ahead of LMMs, so the benchmark's near-term value is as a testbed for hybrid systems rather than end-to-end multimodal routing.
- The open-sourced evaluation protocol gives PCB routing a reproducible yardstick, where prior studies typically relied on fewer than 20 boards.
- Connectivity metrics must be read jointly with output loadability, because some models achieve low open and short rates mainly by routing far fewer nets.
Reading between the lines
- Going beyond the paper, the results suggest that scaling model size alone will not fix the deficit, since even the largest tested models show high physical-short rates and weak multi-net planning; progress may need explicit constraint solvers or classical routers wrapped by LMMs.
- The human reference ceiling below 100% implies the benchmark's upper bound may understate achievable routability; a DRC-verified automated router or a consensus of multiple engineers could set a stronger reference.
- Because the dataset is schematic-coupled and open, a natural extension is fine-tuning or reinforcement learning on the reference routes, which the paper does not test and which could change the current numbers if models learn to imitate human topology.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces OmniRouting, a large-scale benchmark of 1,681 real-world PCB designs with schematic coupling, engineer-created reference routes, netlists, stackups, and design-rule annotations. It proposes four evaluation tasks (geometric routing reasoning, design-rule-aware routing reasoning, electrical functionality reasoning, and tool-augmented agentic routing) and evaluates a range of commercial and open-source LMMs, alongside classical routers and a human-engineer reference. The central empirical finding is that current LMMs perform far below the human reference (best zero-shot NRR/PRR 12.60%/15.41% versus 93.58%/95.91%), with agentic refinement improving results only modestly. The paper also reports analyses of ablations, tool use, routing steps, and reference similarity via a multi-scale graph similarity metric.
Significance. If the reference annotations and the DRC evaluator are trustworthy, OmniRouting would be a substantial and much-needed contribution: it is the first large-scale, openly promised benchmark for LMM-based PCB routing, with a scale (1,681 industrial-grade designs, 77K components, 168K pads) that far exceeds existing public datasets. The paper also provides a systematic evaluation protocol, a classical-router baseline, and a plan to release data, code, and tool interfaces—all of which are strengths. However, the significance of the main claim ('current LMMs fundamentally lack the topological planning and constraint-aware navigation capabilities required for PCB routing') depends on two load-bearing premises that the manuscript itself does not fully support: the human-reference routes are complete and DRC-clean, and the evaluator faithfully measures the advertised design-rule constraints. Since the reported human-reference row contains opens and clearance violations, and the DRC is explicitly clearance-only, the size of the LMM-human gap is not yet established to the level the paper claims.
major comments (5)
- [Table 1 / Human Engineer Reference] Table 1 reports the Human Engineer Reference row with NRR=93.58%, PRR=95.91%, Open=4.38%, PShort=0.38%, NShort=0.04%, and Clr.=0.80, yet the Introduction states that the reference implementations 'pass industrial design-rule checks (DRC) and satisfy fabrication constraints' and the Annotation Curation section states that 'Every PCB design is verified to be fully routable.' These statements are internally inconsistent: if the reference routes are complete ground truths that pass DRC, the evaluator should report near-zero Open and Clr. for that row. The discrepancy suggests either that the source EAGLE layouts contain real opens/clearance violations, or that the evaluator's netlist/copper-pour handling (e.g., pours connected to pads, thermal reliefs, net-ties) misses valid connections. Both possibilities bias the LMM-versus-human comparison because the same evaluator artifacts penalize LMM outputs. The authors must reconcile this contradiction, e.g., by re-verifying the reference routes with an independent DRC and by reporting the fraction of reference nets that are mis-parsed due to copper pours.
- [Evaluation Metrics (Design Rule)] The Evaluation Metrics section states: 'The published DRC uses the same clearance checks, so Total equals Clr., while Dist. is zero in the current evaluation.' This reduces the advertised design-rule-aware routing reasoning (which according to the Task Formulation includes clearance, trace-width, via, obstacle-avoidance, and board-boundary constraints) to a single clearance check. Consequently, the reported 'design-rule compliance' results do not measure trace-width, via, or boundary constraints, and the central claim about constraint-aware navigation is only supported with respect to clearance. The paper should either implement the full DRC checker or explicitly scope the claims to clearance-only evaluation, and report the other constraint types separately if available.
- [Evaluation Metrics (Connectivity)] The definitions of Open, PShort, and NShort state: 'Missing outputs remain in the error-rate denominators but do not increase the error numerators.' This produces degenerate results: a model that emits no routing at all receives Open=0%, PShort=0%, and NShort=0%, as exemplified by Qwen3-235B-A22B Base in Table 2 (all error rates zero with NRR=0.00%). Such a model is penalized only in NRR/PRR, making error rates non-monotonic in routing completeness and overstating connectivity performance of empty outputs. Missing nets should be counted as opens (or the error rates should be defined conditionally on non-empty outputs) so that a model that routes nothing is not scored as having zero opens.
- [Main Results and Findings] The sentence 'agentic refinement reaches only 27.98%/30.34%' is not supported by a single configuration in Table 2: 27.98% NRR corresponds to GPT-5.5 All, while 30.34% PRR corresponds to GPT-5.5 Pre (Preroute). No row in Table 2 simultaneously achieves both values, so the text overstates the best agentic performance. Similarly, the claim that 'GPT-5.5 All Tools reduces Total from 48.06 to 9.35' is not verifiable from the tables, which report Clr. values (5.19 for Base and 4.08 for All) rather than a Total metric; the Total column is absent. The authors should report either a single consistent configuration or explicitly label the best values from different settings.
- [Introduction (Strength of Claim)] The Introduction's conclusion that 'current LMMs fundamentally lack the topological planning and constraint-aware navigation capabilities required for PCB routing' is presented as a finding of the benchmark. Given the unresolved issues above—the human reference row's own error rates, the clearance-only DRC, and the Open-metric artifact—the reported gap (12.60% vs 93.58% NRR) may be real, but the current evaluation does not yet exclude the possibility that part of the gap is an artifact of the evaluator. The paper should either soften the claim to state that LMMs currently perform far below expert reference on clearance-based NRR/PRR, or strengthen it by showing that the corrected DRC and reference validation still produce a similar gap.
minor comments (5)
- [Abstract / Introduction] The abstract contains the word 'rout-able' (hyphenated at a line break) and the phrase 'routable component placements' is used inconsistently; please fix the hyphenation and ensure consistent terminology.
- [Experimental Setups and Tables] The model naming is inconsistent: the Introduction and Experimental Setups refer to 'Qwen3.6-35B-A3B' (also 'Qwen3.6-35B-A3B' in the references), while Tables 1 and 2 use 'Qwen3 235B-A22B' and 'Qwen3.5-9B' (also 'Qwen3.5 9B'). Please unify the names and verify that the cited references match the tested models.
- [Table 2] The rows for Qwen3-235B-A22B in Table 2 list Vis/Score/Sem/Pre/All as '-' (unavailable) while Base exists; please clarify whether these results are missing due to API limits, model failures, or other reasons, and consider whether the zero-valued Base row (NRR=0.00, PRR=0.00, Open=0.00) is an empty output rather than a routing attempt.
- [Evaluation Metrics / Table 2] The text states that 'Dist. is zero in the current evaluation,' but Table 2 appears to contain non-zero entries in the Dist. column for several rows (e.g., Gemini and Ministral configurations). The table formatting is also corrupted in places (e.g., '3.1210.2410.47'), making it difficult to verify the values. Please clean the table and reconcile the Dist. column with the text.
- [General] The manuscript includes a stray LaTeX artifact in the Evaluation Metrics section (the text shows a literal '"'latex "' fragment), and the phrase 'The published DRC uses the same clearance checks, so Totalequals Clr.' has a missing space. Please proofread the final version.
Circularity Check
No circular derivation: benchmark claims are empirical, metrics are defined independently of model outputs, and the only self-citations are toolchain reuse, not load-bearing inference.
full rationale
OmniRouting is an empirical benchmark paper, not a derivation chain. The central claim—that current LMMs lack PCB-routing capabilities—is supported by externally defined evaluations: NRR and PRR are computed from netlists and DRC-clean connectivity, independent of any model's output, and the human-engineered reference is drawn from external open-source EAGLE designs rather than generated from the LMMs being tested. The only self-citations are OmniSch and OmniLayout, which are used as data-extraction tooling in the Annotation Curation pipeline; they do not supply a uniqueness theorem, an ansatz, or a fitted parameter that later reappears as a prediction. The evaluation metric section explicitly separates absolute validity from reference-relative MGS, and even concedes that a valid alternative route can achieve lower MGS, so the reference-similarity metric is not disguised as electrical correctness. The reported inconsistency that the human reference itself shows NRR 93.58%, Open 4.38%, and Clr. 0.80 despite claims that references pass industrial DRC is a validity/reproducibility concern about the DRC checker and ground-truth completeness, not a circularity: it does not make the LMM-vs-human comparison true by construction. Thus there is no self-definitional loop, no fitted-input-called-prediction pattern, and no load-bearing self-citation chain.
Assumptions & free parameters
free parameters (2)
- DRC clearance threshold =
not specified in main text
- MGS coarse and fine grid resolutions =
not specified in main text
assumptions (5)
- domain assumption EAGLE XML source files faithfully encode the complete physical routing, including traces, vias, and copper polygons.
- domain assumption Open-source hardware boards from SparkFun, Arduino, Adafruit, GitHub, Seeed, and ProtoCentral are representative industrial-grade PCB designs.
- ad hoc to paper Design-rule compliance can be assessed with clearance checks alone.
- domain assumption The human reference routing is a valid expert solution and every board is fully routable.
- standard math Graph conversion for MGS preserves routing topology at coarse and fine resolutions.
Cite this review
Pith. "Pith review of OmniRouting: A Semantic-Coupled Multimodal Benchmark for Constraint-Aware Spatial Reasoning in PCB Routing." pith.science (2026). https://pith.science/paper/O6CDXYBQ
@misc{pith2026260804434,
author = {Pith},
title = {Pith review of: OmniRouting: A Semantic-Coupled Multimodal Benchmark for Constraint-Aware Spatial Reasoning in PCB Routing},
year = {2026},
howpublished = {\url{https://pith.science/paper/O6CDXYBQ}},
note = {Machine review of arXiv:2608.04434}
}
read the original abstract
Recent large language models (LLMs) have demonstrated remarkable progress in constraint-aware navigation, maze reasoning, and graph reasoning. However, their ability to reason about complex routing problems under strict geometric, topological, and electrical constraints remains largely unexplored, despite routing being one of the most challenging and critical stages of electronic design automation (EDA). To bridge this gap, we introduce OmniRouting, the first large-scale benchmark designed to evaluate LLMs on printed-circuit-board (PCB) routing reasoning under real-world industrial design-rule, manufacturability, and connectivity constraints. OmniRouting contains 1,681 industrial-grade schematic-coupled PCB designs, including board geometries, routable component placements by human engineers, footprints, pad locations, netlists, stackup information, and routing constraints. The benchmark comprises four tasks: (1) geometric routing reasoning, generating physically valid copper traces, vias, and layer assignments to connect circuit nets within constrained board regions; (2) design-rule-aware routing reasoning, producing routable layouts that satisfy clearance, trace-width, via, obstacle-avoidance, and board-boundary constraints; (3) electrical functionality reasoning, preserving schematic-specified connectivity while reasoning over net names and functional roles to produce electrically correct routing; and (4) tool-augmented agentic routing, leveraging external tools for tasks (1)-(3). Our results reveal substantial limitations of current LMMs in PCB routing, including weak path-planning capabilities, poor adherence to design-rule constraints, and inconsistent preservation of electrical functionality. We will open-source all benchmark data, evaluation code, and tool interfaces to facilitate future research.
Figures
Reference graph
Works this paper leans on
-
[5]
https://www.eremex.com/products/ topor/
TopoR: Topological Router for Printed Circuit Boards Design. https://www.eremex.com/products/ topor/. Accessed: 2026-07-06. Fang,W.;Guo,L.;Lin,J.;Xiong,S.;He,H.;Xu,J.;andChen, J.2024. Obstacle-AwareLength-MatchingRoutingforAny- DirectionTracesinPrintedCircuitBoard. InProceedingsof the 61st ACM/IEEE Design Automation Conference, 1–6. Fatemi, B.; Halcrow, J...
work page 2026
-
[7]
https://github.com/freerouting/freerouting
FreeRouting: Advanced PCB Autorouter. https://github.com/freerouting/freerouting. Ac- cessed: 2026-07-06. GitHub,Inc.2026.GitHub:SoftwareDevelopmentPlatform. https://github.com. Google DeepMind. 2025a. Gemini 2.5 Flash-Lite. https: //deepmind.google/technologies/gemini. GoogleDeepMind.2025b. Gemini3Pro. https://deepmind. google/technologies/gemini. Accessed:
work page 2026
-
[9]
OmniSch: A Multimodal PCB Schematic Benchmark For Structured Diagram Visual Reasoning
A deep reinforcement learning approach for global routing.Journal of Mechanical Design, 142(6): 061701. Lu, T.; Lin, K.; Tian, Y.; Wang, M.; Wang, Y.; Wang, M.; Khatri, S.; Kartik, A.; Wang, Y.; Rane, A. S.; et al. 2026a.OmniSch:AMultimodalPCBSchematicBenchmark For Structured Diagram Visual Reasoning.arXiv preprint arXiv:2604.00270. Lu, T.; Lin, K.; Wang,...
-
[10]
Mantik,S.;Posser,G.;Chow,W.-K.;Ding,Y.;andLiu,W.-H
Graphinstruct: Empowering large language models with graph understanding and reasoning capability.arXiv preprint arXiv:2403.04483. Mantik,S.;Posser,G.;Chow,W.-K.;Ding,Y.;andLiu,W.-H
-
[13]
https://www.quilter.ai/product
The First Physics-Driven AI for Complete PCB Layout. https://www.quilter.ai/product. Accessed: 2026-07-06. Qwen Team. 2026a. Qwen3-35B-A3B. https://huggingface. co/Qwen/Qwen3.5-35B-A3B. QwenTeam.2026b. Qwen3.5-9B. https://qwenlm.github.io. Seeed Technology Co., Ltd
work page 2026
-
[14]
InInternational Conference on Learning Representations, volume 2025, 48118–48145
Grapharena: Evaluating and exploring large language mod- els on graph computation. InInternational Conference on Learning Representations, volume 2025, 48118–48145. The OpenROAD Project
work page 2025
-
[15]
https://github.com/The-OpenROAD-Project- Attic/PcbRouter
PcbRouter: Printed Circuit BoardRouter. https://github.com/The-OpenROAD-Project- Attic/PcbRouter. GitHubrepository,accessedJuly29,2026. Tseng, T.-M.; Li, B.; Ho, T.-Y.; and Schlichtmann, U
work page 2026
-
[1998]
In Proceedings of the 1998 international symposium on Physi- cal design, 80–85
The ISPD98 circuit benchmark suite. In Proceedings of the 1998 international symposium on Physi- cal design, 80–85. Anthropic.2026. ClaudeSonnet4.6. https://www.anthropic. com. Arduino.2026. Arduino:Open-SourceElectronicsPlatform. https://www.arduino.cc. Cadence Design Systems
work page 1998
Show all 15 references
-
[2008]
InProceedings of the 2008 inter- national symposium on Physical design, 156–159
The ISPD global routing benchmark suite. InProceedings of the 2008 inter- national symposium on Physical design, 156–159. OpenAI
2008
-
[2018]
InProceedingsofthe2018InternationalSymposium on Physical Design, 140–143
ISPD 2018 initial detailed routing contest and bench- marks. InProceedingsofthe2018InternationalSymposium on Physical Design, 140–143. Meta AI
2018
-
[2020]
Li, H.; Zhang, J.; Xu, N.; and Liu, M
Circuit routing using monte carlo tree search and deep neural networks.arXiv preprint arXiv:2006.13607. Li, H.; Zhang, J.; Xu, N.; and Liu, M
2006 arXiv
-
[2023]
Alpert, C
Can large lan- guage models be good path planners? a benchmark and in- vestigation on spatial-temporal reasoning.arXiv preprint arXiv:2310.03249. Alpert, C. J
-
[2024]
InIn- ternational conference on learning representations, volume 2024, 43909–43934
Talk like a graph: Encoding graphs for large language models. InIn- ternational conference on learning representations, volume 2024, 43909–43934. FreeRouting Project
2024
-
[2025]
Einarsson,H.2026
AlphaMaze: Enhancing Large Language Models’ Spatial Intelligence via GRPO.arXiv preprint arXiv:2502.14669. Einarsson,H.2026. MazeEval:ABenchmarkforTestingSe- quentialDecision-MakinginLanguageModels. InProceed- ings of the Fifteenth Language Resources and Evaluation Conference ...
2026 arXiv
-
[2026]
https://www.cadence.com/en_US/home/tools/ pcb-design-and-analysis/allegro-x-design-platform.html
Allegro X AI: AI-driven PCB Design. https://www.cadence.com/en_US/home/tools/ pcb-design-and-analysis/allegro-x-design-platform.html. Accessed: 2026-07-06. Chen,B.;Xu,Z.;Kirmani,S.;Ichter,B.;Sadigh,D.;Guibas, L.;andXia,F.2024a.Spatialvlm:Endowingvision-language modelswithspati...
2026
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.