PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design

Hao Chen; Xiaofei Yu; Yang Luo; Yeyu Tong; Yuchao Wu; Yuzhe Ma

REVIEW 3 major objections 5 minor 15 references

PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces PICBench, a benchmark that asks large language models to write photonic integrated circuit netlists from natural-language descriptions and then checks those netlists by simulation.

desk verdict PICBench is a useful first benchmark for LLM-generated photonic netlists, but its functional pass/fail criterion is underspecified and the restriction gains are measured on the problems that produced the restrictions. read the letter →

arxiv 2502.03159 v1 pith:GY2UHBPX submitted 2025-02-05 cs.LG cs.AR

Yuchao Wu , Xiaofei Yu , Hao Chen , Yang Luo , Yeyu Tong , Yuzhe Ma This is my paper · ORCID

classification cs.LGcs.AR

keywords photonicintegratedcircuitsPICdesignautomationLLMcodegenerationbenchmarkingnetlistS-parametersimulationerrorfeedbackpromptengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PICBench, introduced in this paper, is a benchmark for photonic integrated circuit (PIC) design by large language models: given a natural-language description of a circuit, the model must output a JSON netlist that names the components and wires them together. The authors built 24 expert-written problems spanning optical computing, interconnects, switches, and fundamental devices, each with a golden netlist and its simulated frequency response. They then measured how often five commercial LLMs produce a netlist that simulates at all (syntax) and one whose simulated response matches the golden response (functionality). Their experiments show that feeding simulator errors back to the model and adding hand-curated format restrictions sharply raises both scores, with best configurations reaching near-total syntax success and Pass@5 functionality above 60 percent.

What carries the argument

The engine of the paper is an LLM–simulator feedback loop held together by a structured system prompt. The prompt demands a specific JSON netlist schema and gives an API list of allowed built-in components; a separate restrictions block encodes common failure modes (undefined models, duplicate port connections, dangling ports, extra JSON content) as explicit rules. An open-source frequency-domain circuit simulator takes the generated netlist and either reports a syntax error or returns a frequency response. Errors are classified into a fixed category set and fed back to the model with a request to rewrite the entire netlist; if simulation succeeds, the response is compared with the golden response. This loop is what converts a raw code-generation problem into an iterable, automated design process.

What would settle it

Take any of the 24 problems and run a hand-written valid netlist that implements the same function with a different architecture, such as a 4x4 switching network built from a different but equivalent layout; if the benchmark scores this as a functional failure, the comparison against a single golden response is not actually testing function.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-based PIC design generation can be benchmarked automatically and improved substantially by prompt-level interventions. Each problem in PICBench pairs a natural-language specification with an expert-written golden netlist; evaluation pipes a model's generated netlist into an open-source frequency-domain simulator. Syntax is judged by whether the simulator can build the circuit and produce a frequency response; functionality is judged by comparing that response with the saved golden response. In experiments, restrictions plus three rounds of error feedback lifted every tested model's syntax Pass@1 above 50 percent, with one configuration reaching 95 percent; functionality Pass@1 reached 43.33 percent at best, and Pass@5 functionality reached 62.50 percent. The authors interpret these gains as evidence that feedback mechanisms and in-context learning are productive directions for automating PIC design.

Load-bearing premise

The load-bearing premise is that comparing an LLM-generated circuit's simulated frequency response with the single expert-written golden response, without an explicit tolerance or a set of equally valid responses, correctly determines whether the circuit implements the requested function.

Editorial extensions

If this is right

With restrictions and three error-feedback iterations, all five tested LLMs reach 100% syntax Pass@5 on PICBench, meaning the remaining gap is almost entirely functional correctness.
A single feedback round can be more valuable than five independent samples: several models' Pass@1 syntax with one feedback iteration exceeds their Pass@5 syntax without feedback.
Hand-written restrictions transfer common simulator error messages into the prompt and improve syntax even with no feedback, raising one model's Pass@1 syntax from 9.17% to 64.17%.
Functionality remains the harder test: best Pass@1 with all interventions is 43.33%, so current LLMs can often produce simulatable circuits that do not match the expert design's behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the evaluation were extended to accept a family of equivalent frequency responses rather than one golden curve, the reported functionality scores would likely rise, because many different component placements can realise the same transfer function.
The restriction list is curated from observed errors; automating its construction from simulator messages would make the loop fully self-improving and could transfer to other structured-code domains such as analog or RF circuit netlists.
PICBench's fast simulation and machine-checkable errors make photonic netlist generation a useful probe for studying how LLMs handle domain-specific formal languages, separate from general coding benchmarks.
The Pass@5 functionality ceiling of 62.5% suggests that sampling more candidates alone will not solve the task; future gains likely require feedback loops that check functional, not merely syntactic, correctness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

PICBench is worth taking seriously. It is the first benchmark I know of that adapts LLM code-generation evaluation to photonic netlists, and it ships the two things that matter most: a concrete 24-problem suite with golden solutions, and an automated pipeline that checks both syntax and functionality on top of SAX. That fills a real gap. Prior LLM chip-design benchmarks stop at Verilog and SPICE, and the existing PIC work is either unautomated or limited to seven devices. The domain-specific error taxonomy and the feedback loop are genuinely useful additions, and the paper is honest enough to report low baseline scores rather than cherry-pick results. The tables are consistent, and the finding that feedback plus restrictions sharply improves Pass@1 is plausible.

The soft spots are real but fixable. The load-bearing one is Section III-C. The paper says functionality is checked by simply comparing the simulated frequency response of the generated netlist against the golden response, but it never states the metric, tolerance, wavelength grid, or how complex S-parameters are handled. That matters because equivalent correct designs can produce different raw responses: port relabeling, alternative MZI architectures, or phase shifts differing by 2π are all legitimate and would not match exactly. Without a defined and validated matching criterion, the functionality scores could mislead. This is not a fatal flaw given the benchmark artifact is what future users will actually run, but the paper’s headline comparisons rest on it.

The second soft spot is the restrictions experiment. The restriction list in Table II was assembled by watching models fail on the exact 24 problems in Table I, then tested on those same problems. That is a training-on-the-test-set effect, so the magnitude of the gain (e.g., Gemini from 9% to 64% syntax Pass@1) is not an independent measure of what the restrictions would do on unseen problems. The feedback-loop gains are less contaminated and still show a clear effect.

Minor issues: n=5 samples per problem gives coarse Pass@k estimates with no error bars, and the paper does not say whether the repository contains the exact evaluation code and thresholds. Those are easy to address.

Bottom line: the benchmark itself is a solid contribution and the right kind of artifact to build in this area. The evaluation methodology needs tightening, not replacement. I would send this to peer review expecting major revision, and I would cite it if I worked on LLM-driven chip or photonic design.

Referee Report

3 major / 5 minor

Summary. The paper introduces PICBench, a benchmark and evaluation framework for assessing LLMs on photonic integrated circuit (PIC) design. It comprises 24 natural-language design problems spanning optical computing, optical interconnects, optical switches, and fundamental devices. LLM outputs are JSON netlists for the open-source SAX simulator, and the framework automatically evaluates syntax (whether SAX can simulate the netlist) and functionality (whether the simulated frequency response matches a golden solution). The authors evaluate five commercial LLMs, report Pass@1 and Pass@5 scores, and study two prompt-engineering techniques: an error-feedback loop and a set of restrictions derived from common errors. They report substantial improvements from both techniques.

Significance. If the benchmark's evaluation criteria are valid, PICBench fills a clear gap: prior LLM-for-photonics work tested only a few hand-designed cases, while digital-design benchmarks like VerilogEval and RTLLM have no photonic counterpart. The paper ships open-source code and uses the mainstream SAX simulator, making the framework reusable and comparable. The systematic comparison of five LLMs and the demonstration that feedback and restrictions improve scores are useful for the community. However, the significance hinges on the functionality check being a sound and reproducible measure of design correctness, and on the experimental comparisons being fair; both issues are currently questionable.

major comments (3)

[III-C] The functionality evaluation is underspecified to the point of being non-reproducible. The paper states only that "we simply compare the simulation results between generated code completions and golden reference solutions," without defining the metric, the tolerance, the wavelength grid, or the treatment of complex S-parameters. This matters because distinct correct implementations (e.g., port relabeling, alternative but functionally identical MZI topologies, phase shifts differing by 2π) will produce different frequency responses. If the comparison is exact equality, the benchmark penalizes valid designs; if a tolerance is used, it must be reported and justified. The paper also provides no evidence that the golden response is necessary and sufficient for correctness, nor a human-validated set of alternate correct designs for any problem. Since Tables III and IV and the abstract's claims about LLM capability rest on these functionality numbers, the missing criterion is load-bearing and should be supplied.
[III-D and Table IV] The claim that the restrictions improve LLM performance is circular. The restrictions in Table II are described as "common error types we collected during our trials" on the same 24 problems, and the improvement in Table IV is measured on those same problems. This amounts to tuning on the test set and then reporting the test-set accuracy; the measured gains (e.g., Gemini 1.5 Pro syntax Pass@1 from 9.17% to 64.17%) may largely reflect overfitting to the benchmark's specific idiosyncrasies rather than a general improvement in PIC-design ability. To support the claimed impact of restrictions, the authors should evaluate on a held-out set of new problems or use a temporal split in which restrictions are frozen before evaluating on the benchmark.
[IV-A and IV-B] The comparison between "without feedback" and "with feedback" results is not apples-to-apples because the number of LLM calls per problem differs. In the feedback condition, each sample can undergo up to 1 or 3 correction iterations, so Pass@1 with 3 EFs allows up to four sequential attempts per sample, whereas Pass@1 without EF allows only one attempt. The observed improvements (e.g., Claude 3.5 Sonnet functionality Pass@1 from 1.67% to 24.17%) could be due simply to the additional attempts rather than to the feedback. The paper should either control for the total number of LLM calls (e.g., compare Pass@1 with 3 EFs against Pass@4 without feedback) or report the results as a function of the number of attempts. As written, the feedback experiment conflates iteration count with sample count and the conclusion is not supported.

minor comments (5)

[III-B] The text says "we do not include any purely device-level design problems in our collection," but Table I includes a category called "Fundamental Devices" with MZM and MZI ps. The intended distinction (device-level lacking connections) should be clarified to avoid an apparent contradiction.
[IV-A] The sentence "we queried the selected LLMs both without and with feedback for n iterations where we set n = 1 and n = 3" is ambiguous: it is unclear whether n is the number of feedback iterations per sample or the number of samples per problem, especially because n is already used for the sample count in the Pass@k estimator. Please use distinct notation and specify both values.
[III-A] The error classification loop is described as involving "human inspection" when collecting restrictions, but the feedback loop in Section III-E says the error is "automatically classified." Please clarify whether the classification for feedback is automated or manual, and how.
[Figure 3] In the JSON format example, the second instance key is written "<component_name2" with a missing closing angle bracket; fix the typo.
[Reference [14]] The SAX reference is incomplete: it gives only the GitHub URL and an informal citation. Please provide a version, DOI, or formal publication reference, as the evaluation depends on the exact simulator behavior.

Circularity Check

1 steps flagged · score 4.0 of 10

Restrictions are hand-tuned on the same 24 test problems that then 'demonstrate' their benefit, so the reported restriction-driven improvement is partly in-sample; the benchmark artifact itself is externally grounded.

fitted input called prediction [Section III-D (Error Classification Loop) and Section IV-B2 (Impact of restrictions), with Table II and Table IV.]
"Since the specific aspects causing LLM failures are uncertain, errors are iteratively inspected and summarized during the generation process. For each conversation, if a syntax error is detected, a human expert inspects the error information, identifies common errors, and summarizes them into general restrictions to prevent the recurrence of similar errors. ... TABLE II summarize all the common error types we collected during our trials and the corresponding restrictions that we collect as prompt. ..."

The restrictions are not independent prior knowledge; they are extracted from the exact 24 problems used as the test set. Section III-D says errors are 'inspected and summarized during the generation process,' and Table II lists 'common error types we collected during our trials.' Table IV then measures the benefit of those same restrictions on the same trials and problems. The reported improvement is therefore partly guaranteed by construction: the prompt has been tuned to correct the specific failure modes already observed on these benchmark problems. This is an in-sample fit presented as a demonstrated enhancement of LLM PIC-design ability, although it does not invalidate the benchmark as an external artifact for future evaluation.

full rationale

PICBench's central contribution is an external benchmark artifact: 24 human-crafted problems, golden netlists, SAX-based syntax checks, and a functionality comparison, all grounded in an open-source simulator. That comparison is underspecified, because Section III-C only says the authors 'simply compare the simulation results between generated code completions and golden reference solutions' without defining tolerance, wavelength grid, or treatment of equivalent but non-identical correct circuits. That is a validity threat to the reported functionality scores, but it is not circularity: the golden solutions are expert-written external references, not outputs of the LLMs being tested. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result. The one genuine circularity-adjacent step is the restriction experiment: the restrictions in Table II were collected by human inspection of errors occurring during the same trials, and Table IV reports their benefit on those same problems. That makes the 'impact of restrictions' claim an in-sample demonstration rather than an independent evaluation. Since this affects a secondary contribution rather than the benchmark itself, the overall circularity score is moderate, not high.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims depend mainly on the validity of SAX models, the correctness of golden netlists, and the adequacy of the functional comparison. The experimental setup also relies on an unreported tolerance and on the assumption that LLM scores are not inflated by benchmark leakage. No new physical entities are introduced.

free parameters (3)

Functional comparison tolerance = not reported
The pass/fail decision for functionality depends on a distance between the generated and golden SAX frequency responses; the paper never states the tolerance or matching criterion (Section III-C).
Number of samples per problem (n) = 5
Pass@k is estimated with n=5 samples per problem (Section IV-A); the estimate's variance is unreported.
Maximum error feedback iterations = 1 and 3
Reported results are for 0, 1, and 3 feedback iterations (Table III), chosen by the authors rather than determined by convergence.

assumptions (4)

domain assumption SAX simulator S-parameter models are physically accurate
Section III-A states SAX performs S-parameter-based circuit simulation using built-in models for waveguides, couplers, MMIs, MZIs, MRRs, and phase shifters; the whole evaluation depends on these models being correct.
domain assumption Golden solutions are correct and are suitable unique reference points
Section III-B says human designers hand-craft the correct design for each problem; the benchmark treats these as ground truth, but functional equivalence is checked only against these specific implementations.
domain assumption LLM outputs are not contaminated by benchmark leakage
The paper assumes the 24 PIC problems are rarely seen during pretraining (Section III-D), yet no deduplication or contamination analysis is provided; if models memorized problems, scores would overstate capability.
standard math Pass@k estimator is unbiased
The paper uses the unbiased Pass@k estimator from Chen et al. [15] in Eq. 1; this is a standard statistical identity under the stated sampling scheme.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design." pith.science (2026). https://pith.science/paper/GY2UHBPX

@misc{pith2026250203159,
  author       = {Pith},
  title        = {Pith review of: PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GY2UHBPX}},
  note         = {Machine review of arXiv:2502.03159}
}

read the original abstract

While large language models (LLMs) have shown remarkable potential in automating various tasks in digital chip design, the field of Photonic Integrated Circuits (PICs)-a promising solution to advanced chip designs-remains relatively unexplored in this context. The design of PICs is time-consuming and prone to errors due to the extensive and repetitive nature of code involved in photonic chip design. In this paper, we introduce PICBench, the first benchmarking and evaluation framework specifically designed to automate PIC design generation using LLMs, where the generated output takes the form of a netlist. Our benchmark consists of dozens of meticulously crafted PIC design problems, spanning from fundamental device designs to more complex circuit-level designs. It automatically evaluates both the syntax and functionality of generated PIC designs by comparing simulation outputs with expert-written solutions, leveraging an open-source simulator. We evaluate a range of existing LLMs, while also conducting comparative tests on various prompt engineering techniques to enhance LLM performance in automated PIC design. The results reveal the challenges and potential of LLMs in the PIC design domain, offering insights into the key areas that require further research and development to optimize automation in this field. Our benchmark and evaluation code is available at https://github.com/PICDA/PICBench.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

[1]

Single-chip microprocessor that communicates directly using light,

C. Sun, M. T. Wade, Y . Lee, J. S. Orcutt, L. Alloatti, M. S. Georgas, A. S. Waterman, J. M. Shainline, R. R. Avizienis, S. Lin et al. , “Single-chip microprocessor that communicates directly using light,” Nature, 2015

work page 2015
[2]

Novel frontier of photonics for data processing—photonic accelerator,

K.-i. Kitayama, M. Notomi, M. Naruse, K. Inoue, S. Kawakami, and A. Uchida, “Novel frontier of photonics for data processing—photonic accelerator,” Apl Photonics, 2019

work page 2019
[3]

Empowering high-dimensional optical fiber communications with integrated photonic processors,

K. Lu, Z. Chen, H. Chen, W. Zhou, Z. Zhang, H. K. Tsang, and Y . Tong, “Empowering high-dimensional optical fiber communications with integrated photonic processors,” Nature Communications, 2024

work page 2024
[4]

Large-scale photonic chiplet taichi empowers 160-tops/w artificial general intelligence,

Z. Xu, T. Zhou, M. Ma, C. Deng, Q. Dai, and L. Fang, “Large-scale photonic chiplet taichi empowers 160-tops/w artificial general intelligence,” Science, 2024

work page 2024
[5]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalk- wyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

arXiv 2023
[6]

The claude 3 model family: Opus, sonnet, haiku

Anthropic, “The claude 3 model family: Opus, sonnet, haiku.”

work page
[7]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023
[8]

Rtllm: An open-source benchmark for design rtl generation with large language model,

Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) , 2024

work page 2024

Show all 15 references

[9]

Verilogeval: Evaluating large language models for verilog code generation,

M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) , 2023

2023
[10]

Spicepilot: Navigating spice code generation and simulation with ai guidance,

D. Vungarala, S. Alam, A. Ghosh, and S. Angizi, “Spicepilot: Navigating spice code generation and simulation with ai guidance,” arXiv preprint arXiv:2410.20553, 2024

2024 arXiv
[11]

Chateda: A large language model powered autonomous agent for eda,

H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024

2024
[12]

From english to pcsel: Llm helps design and optimize photonic crystal surface emitting lasers,

R. Li, C. Zhang, S. Mao, H. Huang, M. Zhong, Y . Cui, X. Zhou, F. Yin, S. Theodoridis, and Z. Zhang, “From english to pcsel: Llm helps design and optimize photonic crystal surface emitting lasers,” 2023

2023
[13]

Towards large-language model assisted layout of silicon photonic integrated circuits,

J. Liu, A. Sharma, C. Doumbia, and J. K. Poon, “Towards large-language model assisted layout of silicon photonic integrated circuits,” in European Conference on Integrated Optics , 2024

2024
[14]

Laporte, “Sax,” 2023

F. Laporte, “Sax,” 2023. [Online]. Available: https://github.com/flaport/sax

2023
[15]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

2021 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Single-chip microprocessor that communicates directly using light,

C. Sun, M. T. Wade, Y . Lee, J. S. Orcutt, L. Alloatti, M. S. Georgas, A. S. Waterman, J. M. Shainline, R. R. Avizienis, S. Lin et al. , “Single-chip microprocessor that communicates directly using light,” Nature, 2015

work page 2015

[2] [2]

Novel frontier of photonics for data processing—photonic accelerator,

K.-i. Kitayama, M. Notomi, M. Naruse, K. Inoue, S. Kawakami, and A. Uchida, “Novel frontier of photonics for data processing—photonic accelerator,” Apl Photonics, 2019

work page 2019

[3] [3]

Empowering high-dimensional optical fiber communications with integrated photonic processors,

K. Lu, Z. Chen, H. Chen, W. Zhou, Z. Zhang, H. K. Tsang, and Y . Tong, “Empowering high-dimensional optical fiber communications with integrated photonic processors,” Nature Communications, 2024

work page 2024

[4] [4]

Large-scale photonic chiplet taichi empowers 160-tops/w artificial general intelligence,

Z. Xu, T. Zhou, M. Ma, C. Deng, Q. Dai, and L. Fang, “Large-scale photonic chiplet taichi empowers 160-tops/w artificial general intelligence,” Science, 2024

work page 2024

[5] [5]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalk- wyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

arXiv 2023

[6] [6]

The claude 3 model family: Opus, sonnet, haiku

Anthropic, “The claude 3 model family: Opus, sonnet, haiku.”

work page

[7] [7]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023

[8] [8]

Rtllm: An open-source benchmark for design rtl generation with large language model,

Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) , 2024

work page 2024