REVIEW 5 major objections 5 minor 49 references
PICopilot claims that the limiting factor in LLM-assisted PIC design is retrieval quality, not raw model power, and that its summary-based RAG pipeline plus multi-agent feedback solves all 48 tasks in its own benchmark.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A training-free multi-agent LLM framework with summary-based hybrid retrieval generates correct PIC design tool scripts from natural language, passing all 48 self-built benchmark tasks where the best baseline passes 27.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection Strong system, unverifiable headline: the evaluation's success criterion contradicts its own evaluator design. the 5 major comments →
PICopilot: An LLM-based Agentic Framework for Assisting Photonic Integrated Circuit Design via Script Generation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the paper's own terms, the discovery is that retrieval quality, not model strength, is the binding constraint on LLM script generation in PIC design. The tailored retrieval-augmented generation pipeline—query generation plus summary-based hybrid dense/sparse retrieval over per-tool manual databases—outperforms a generic dense-retriever RAG, and the multi-agent decomposition with feedback adds further gains. The framework reports 48/48 tasks solved, an overall Pass@1 of 93.9%, and overhead that is small in absolute cost and latency compared with direct LLM calls or manual script writing.
What carries the argument
The load-bearing object is the summary-based hybrid retriever combined with per-tool manual databases. Each database stores original manual chunks plus LLM-generated summaries; the retriever scores query–summary matches with both dense cosine similarity and BM25, fuses the ranked lists by weighted reciprocal rank fusion, and returns the original chunks as references for the programmer agent. This carries the argument because component-removal experiments show the largest performance drop when this tailored retrieval is replaced with a general one.
Load-bearing premise
The central claim rests on the authors' own 48-task benchmark, where success is whatever their comparative procedure counts as functionally identical to a hand-written ground-truth script; a biased task set or a lenient success judgment would invalidate the 48/48 result and every comparison derived from it.
What would settle it
Reproduce the benchmark independently: execute each generated script in a fresh installation of the PIC tools and compare actual outputs against the ground truth. The paper's Section 3.6 says running PIC tools is impractical, while Section 4.1.4 says success is determined by executing scripts; reconciling that contradiction, or finding that success was judged by inspection rather than real execution, would settle whether the 48/48 claim holds.
If this is right
- Designers can describe a layout, DRC, or simulation step in plain English and receive executable Python, eliminating most manual API lookups across vendor tools.
- Adding a new PIC tool only requires building a new manual database and retriever, so the system can track a fast-moving ecosystem without retraining.
- Because the method is training-free and costs less than a cent per task on average, small design teams can deploy it without dedicated GPU training infrastructure.
- A static checker plus custom check library catches syntax and common logic errors without running expensive simulations, making iterative refinement cheap in practice.
Where Pith is reading between the lines
- Testable extension not in the paper: the query–summary retrieval paradigm may transfer to other engineering scripting domains with fragmented tool manuals, such as electronic design automation or scientific simulation, where matching English queries to code-heavy documentation is equally hard.
- Testable extension not in the paper: directly measuring retrieval hit rate against human experts' chosen manual pages would isolate retrieval quality from the leniency of the LLM-based evaluator; the paper reports end-to-end success but not per-query retrieval precision.
- Testable extension not in the paper: releasing the 48-task benchmark and an independent execution protocol would allow deterministic reproduction of the 48/48 result; the contradiction between the paper's statement that executing PIC tools is impractical and its claim that success was determined by executing scripts makes the judging protocol the highest-value place to audit.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PICopilot, an LLM-based multi-agent framework that generates Python scripts for photonic integrated circuit (PIC) design from natural-language task descriptions. The framework comprises a task planner, function-specific script generators with a summary-based hybrid RAG pipeline (query generator, multi-database retrievers, dense+BM25 fusion), a script synthesizer, and an LLM-based script evaluator with an adaptive feedback loop. Experiments on a self-constructed 48-task benchmark report that PICopilot solves all 48 tasks, outperforming GPT-5, DeepSeek-V3.2, and Qwen3-Coder under zero-shot and general-RAG settings, with modest overhead. The paper also presents an ablation study on the advanced task set showing contributions of the tailored RAG pipeline and multi-agent architecture.
Significance. If the empirical claims hold, PICopilot would be a valuable contribution to photonic design automation: it is a training-free, domain-adapted system that combines a purpose-built retrieval pipeline with agent orchestration to generate tool-invoking scripts, a task distinct from prior netlist- or layout-oriented PIC-LLM work. The system design is thoughtful, particularly the summary-based retrieval paradigm and multi-database structure, which are clearly motivated by the fragmentation of PIC tool APIs. The overhead analysis is useful for practical deployment. However, the central claim—48/48 tasks solved—rests entirely on the evaluation protocol, and the paper contains a direct contradiction between the stated execution-based success criterion and the described non-execution-based evaluator. Because the benchmark is self-constructed and the judging procedure is unspecified, the headline result is not currently verifiable. The paper would benefit from a clear, reproducible evaluation pipeline, a released benchmark, and sensitivity analyses of its hand-tuned parameters.
major comments (5)
- [§4.1.4 vs §3.6] The success criterion is internally contradictory. §4.1.4 states: "The success of a trial is determined by executing the generated script. If the script's execution result is functionally identical to the test task's ground truth, the trial is considered successful." Yet §3.6 states that executing generated scripts is "impractical because running PIC design tools is extremely time-consuming" and that the Script Evaluator instead uses an LLM with a static code checker and a custom check library. The paper never explains how the 48-task experiments were scored: were scripts actually executed (if so, how were the FDTD-scale tasks run?), or was the §3.6 LLM-based evaluator used as the judge? If the latter, then the Pass@k and #Solved numbers reflect an LLM's static judgment, not functional execution, and the 48/48 claim is not substantiated. This is load-bearing for every quantitative compar
- [§4.1.3 and §4.1.4] The benchmark and judging protocol lack sufficient detail for reproducibility. The 48 tasks are "carefully selected" but Table 3 gives only generic placeholders (e.g., "create a layout of ... by ...") and states that some descriptions are omitted due to space limitations. The criterion "functionally identical to the test task's ground truth" is not operationalized: no judge is specified (human? LLM? exact output comparison?), no tolerance for numerical/semantic equivalence is given, and no examples of failure/success are shown. The ground-truth scripts were "verified by running" them, which makes the benchmark's own reference standard execution-based, but this does not establish that the evaluated scripts were executed. The authors should release the full task descriptions, ground-truth scripts, and the exact judging protocol (including any LLM-evaluator prompts), or the results cannot b
- [§3.4.3 and §4.1.1] Several free parameters are tuned on the same tasks used for evaluation, which risks optimistic bias. The RRF fusion weights are set to (w_dense, w_sparse) = (0.7, 0.3) "based on actual testing" (§3.4.3); top-k is set to k=5 and the maximum feedback iterations to N_f=3 (§4.1.1). No sensitivity analysis or train/test separation is reported for these choices. Since the 48-task benchmark is the only evaluation set, it is unclear whether the reported gains reflect genuine improvements or overfitting to this specific set. The authors should either report a parameter sweep, validate on a held-out set, or at least discuss the stability of results across reasonable parameter ranges.
- [§3.6] The custom check library used by the Script Evaluator is built from "common errors found in generated PIC design scripts"—possibly including errors from PICopilot's own outputs. If the evaluator's checks are aligned with the types of errors that PICopilot's feedback loop is designed to fix, the evaluation becomes partially self-referential: the framework is judged by a rubric derived from its own failure modes. The paper should describe the check library's contents, how it was constructed, and whether it was designed independently of the evaluation protocol. Ideally, the final decision should be a functional comparison (execution) or an independently specified set of acceptance tests.
- [Table 5 and §4.4] The ablation study is conducted only on the 'Advanced' task set (24 tasks). While this is reasonable for isolating difficult cases, the paper does not report ablations on Basic and Medium tasks, leaving open the question of whether the tailored RAG and multi-agent components help uniformly or only on composite tasks. Given that #Solved on Advanced strongly drives the total (24/48), reporting the full-task-set ablation would strengthen the claim that both components are essential across difficulty levels.
minor comments (5)
- [Abstract and §4.3] The abstract states PICopilot improves performance "without incurring substantial extra latency or cost," but Table 4 shows PICopilot's average cost (0.67×10^-2 $) is higher than Qwen3-Coder and DeepSeek-V3.2 baselines (both zero-shot and RAG), and its LLM time (35.88 s avg) exceeds several baselines. The claim should be qualified to indicate the increase is small in absolute terms and still below GPT-5, rather than implying no increase.
- [Table 3] The benchmark table would be much more informative with the actual task descriptions or a link to a hosted dataset. The current ellipses ("...") make it impossible to judge task difficulty or evaluate the appropriateness of the difficulty partitioning.
- [§3.4.2] The database construction workflow mentions a "human expert verification step" but does not specify how many experts, how they verified the summaries, or whether inter-annotator agreement was considered. A brief description of this verification protocol would improve confidence in the database's correctness.
- [§4.1.1] The choice of LLMs (Qwen3-Coder for programming, DeepSeek-V3.2 for other agents) is reasonable, but the paper should justify why DeepSeek-V3.2 is used for planning/evaluation rather than the stronger GPT-5, especially since GPT-5 is used as a baseline. A sentence explaining this choice (e.g., cost, latency, or empirical performance) would prevent confusion about the baseline comparison.
- [Throughout] Minor language issues: "pioneer" (contributions) is informal; "the first tool to pioneer" is redundant. The paper would benefit from a careful proofreading pass.
Circularity Check
The 48/48 headline result is partly self-referential: retrieval hyperparameters are tuned on the same benchmark, and the stated execution-based success criterion is contradicted by an LLM-based evaluator whose check library is built from the system's own generated-script errors.
specific steps
-
fitted input called prediction
[Sec 3.4.3 (RRF weights) and Sec 4.1.1 (k, N_f); results in Table 2]
"w_r is the importance weight and we set(w_dense,w_sparse)=(0.7,0.3) based on actual testing. ... We set the top-k retrieved documents per query to k=5, and the maximum iteration count N_f=3."
The RRF fusion weights are explicitly chosen 'based on actual testing' rather than on a held-out validation set, and the retrieval count k and feedback iteration limit N_f are fixed without any described validation split. All reported Pass@1, Pass@5, and #Solved numbers in Table 2 are computed on the same 48-task benchmark used for this tuning. The headline success rates are therefore in-sample performance of a system whose retrieval hyperparameters were fit to those very tasks, not an independent prediction.
-
self definitional
[Sec 4.1.4 (Metrics) vs Sec 3.6 (Script Evaluator)]
"The success of a trial is determined by executing the generated script. If the script's execution result is functionally identical to the test task's ground truth, the trial is considered successful. ... Although directly executing generated scripts for evaluation is common and effective, it is impractical because running PIC design tools is extremely time-consuming... Therefore, we design this agent to perform evaluation via an LLM equipped with a static code checker and a custom check library."
Section 4.1.4 defines success as execution-based functional identity with the ground truth, but Section 3.6 states that executing PIC tools is impractical and the actual evaluator is an LLM with a static checker and a custom check library. The operational success criterion is therefore not the claimed execution comparison but the judgment of an LLM evaluator that is itself a component of PICopilot. The reported 48/48 and all baseline comparisons rest on this self-contained evaluator, not on an independent functional check of the generated scripts.
-
self definitional
[Sec 3.6 (custom check library)]
"We establish this library by collecting common errors found in generated PIC design scripts and rewriting them as check prompts."
The evaluator's check library is derived from errors observed in generated PIC design scripts, i.e., from the behavior of the very kind of system being evaluated. The feedback loop then iterates until this self-derived evaluator is satisfied, and the same evaluator is used to declare success. This closes the evaluation loop: the rubric is constructed from the system's own outputs, so 'correctness' as measured is not an externally anchored functional comparison but a match to a rubric generated from the system's own error modes.
full rationale
The paper's central empirical claim is that PICopilot solves all 48 tasks and outperforms LLM baselines. This claim is not supported by an independent evaluation: Sec 4.1.4 defines success as executing the generated script and comparing its output to ground truth, but Sec 3.6 says execution is impractical and the actual evaluator is an LLM with a static checker and a custom check library. The check library is built from 'common errors found in generated PIC design scripts,' so the evaluation rubric is derived from the system's own outputs; the feedback loop then iterates until this self-derived evaluator is satisfied. Additionally, the RRF fusion weights are set 'based on actual testing' and k=5, N_f=3 are fixed without a held-out split, all on the same 48-task benchmark whose Pass@1/#Solved are reported. Thus the headline numbers are in-sample and self-judged rather than independent predictions. This is not a case of problematic self-citation; the author-built benchmark is not itself circular. The circularity lies in the evaluation and hyperparameter tuning loop, making the 48/48 result partially reduce to the system's own fitted parameters and self-assessment.
Axiom & Free-Parameter Ledger
free parameters (4)
- RRF fusion weights (w_dense, w_sparse) =
(0.7, 0.3)
- Top-k retrieved documents per query =
5
- Maximum feedback iterations N_f =
3
- RRF smoothing constant c =
60
axioms (5)
- domain assumption Python covers all PIC design tool invocations (layout, DRC, simulation)
- domain assumption LLM-generated, expert-verified manual summaries are faithful retrieval indices
- domain assumption Execution output comparison to ground truth is an objective, unbiased success measure
- ad hoc to paper The 48 hand-selected tasks are representative of real-world PIC scripting
- standard math Standard retrieval math (cosine, BM25, RRF) and the Pass@k estimator are applied correctly
invented entities (3)
-
Summary-based hybrid retrieval paradigm (Query Generator + summary matching + dense/sparse fusion)
independent evidence
-
Custom check library for the Script Evaluator Agent
no independent evidence
-
Adaptive feedback loop across agents
independent evidence
Cite this review
Pith. "Pith review of PICopilot: An LLM-based Agentic Framework for Assisting Photonic Integrated Circuit Design via Script Generation." pith.science (2026). https://pith.science/paper/MMR4ZXAR
@misc{pith2026260801791,
author = {Pith},
title = {Pith review of: PICopilot: An LLM-based Agentic Framework for Assisting Photonic Integrated Circuit Design via Script Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/MMR4ZXAR}},
note = {Machine review of arXiv:2608.01791}
}
read the original abstract
The rapid development of photonic integrated circuits (PICs) is shifting the design flow from traditional graphical user interface (GUI)-based methods to script-based methods for higher flexibility, portability, and maintainability. However, script-based design introduces new challenges, requiring designers to possess additional proficiency in tool application programming interfaces (APIs) and programming. It also demands greater effort and time because it is inherently less intuitive and more complex than GUI-based methods. As PICs grow in scale and complexity, the productivity gap between design needs and manual scripting capabilities continues to widen. To address this gap, we introduce PICopilot, the first large language model (LLM)-based agentic framework that assists in PIC design via automated design script generation from natural language instructions. PICopilot leverages a multi-agent architecture with a feedback mechanism and a specifically designed retrieval-augmented generation (RAG) pipeline, achieving a high success rate and reliability. Experimental results on a benchmark of diverse PIC scripting tasks demonstrate that PICopilot successfully completes all 48 tasks and outperforms other LLM-based approaches without incurring substantial extra latency or cost, even solving 21 more tasks than the advanced GPT-5 model with a general RAG pipeline.
Figures
Reference graph
Works this paper leans on
-
[1]
Shupeng Ning, Hanqing Zhu, Chenghao Feng, Jiaqi Gu, Zhixing Jiang, Zhoufeng Ying, Jason Midkiff, Sourabh Jain, May H Hlaing, David Z Pan, et al . 2024. Photonic-electronic integrated circuits for high-performance computing and ai accelerators.Journal of Lightwave Technology(2024)
work page 2024
-
[2]
Shawn Yohanes Siew, Bo Li, Feng Gao, Hai Yang Zheng, Wenle Zhang, Pengfei Guo, Shawn Wu Xie, Apu Song, Bin Dong, Lian Wee Luo, et al. 2021. Review of silicon photonics technology and platform development.Journal of Lightwave Technology39, 13 (2021), 4374–4389
work page 2021
-
[3]
Farshid Ashtiani, Alexander J Geers, and Firooz Aflatouni. 2022. An on-chip photonic deep neural network for image classification.Nature606, 7914 (2022), 501–506
work page 2022
-
[4]
Zhihao Xu, Tiankuang Zhou, Muzhou Ma, ChenChen Deng, Qionghai Dai, and Lu Fang. 2024. Large-scale photonic chiplet Taichi empowers 160-TOPS/W artificial general intelligence.Science384, 6692 (2024), 202–209
work page 2024
-
[5]
Saumil Bandyopadhyay, Alexander Sludds, Stefan Krastanov, Ryan Hamerly, Nicholas Harris, Darius Bunandar, Matthew Streshinsky, Michael Hochberg, and Dirk Englund. 2024. Single-chip photonic deep neural network with forward-only training.Nature Photonics18, 12 (2024), 1335–1343
work page 2024
-
[6]
Sufi R Ahmed, Reza Baghdadi, Mikhail Bernadskiy, Nate Bowman, Ryan Braid, Jim Carr, Chen Chen, Pietro Ciccarella, Matthew Cole, John Cooke, et al. 2025. Universal photonic artificial intelligence acceleration.Nature640, 8058 (2025), 368–374
2025
-
[7]
Gdsfactory. 2023. GDSFactory 9.20.6. https://gdsfactory.github.io/gdsfactory/
work page 2023
-
[8]
Luceda. 2025. Luceda IPKISS. https://www.lucedaphotonics.com/luceda- photonics-design-platform
work page 2025
-
[9]
Siemens. 2025. L-Edit Photonics. https://eda.sw.siemens.com/en-US/ic/ic- custom/photonic/l-edit-photonics/
work page 2025
-
[10]
Matthias Köfferlein. 2020. KLayout
work page 2020
-
[11]
Spark Photonics. 2025. Check Mate DRC. https://www.sparkphotonics.com/ checkmatedrc
work page 2025
-
[12]
ANSYS Inc. 2025. Ansys Lumerical INTERCONNECT. https://www.ansys.com/ products/optics/interconnect
work page 2025
-
[13]
ANSYS Inc. 2025. Ansys Lumerical FDTD. https://www.ansys.com/products/ optics/fdtd
work page 2025
-
[14]
ANSYS Inc. 2025. Ansys Lumerical MODE. https://www.ansys.com/products/ optics/mode
work page 2025
-
[15]
Flexcompute. 2025. FAST, MODERN PHOTONIC SIMULATIONS. https://www. flexcompute.com/tidy3d/
work page 2025
-
[16]
Synopsys Inc. 2025. Synopsys OptSim. https://www.synopsys.com/photonic- solutions/optocompiler/optsim-photonic-ic.html
work page 2025
-
[17]
Yinyi Liu, Bohan Hu, Zhenguo Liu, Peiyu Chen, Linfeng Du, Jiaqi Liu, Xianbin Li, Wei Zhang, and Jiang Xu. 2023. FIONA: Photonic-Electronic CoSimulation Frame- work and Transferable Prototyping for Photonic Accelerator. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 1–9
work page 2023
-
[18]
Xiaohan Jiang, Yinyi Liu, Peiyu Chen, Wei Zhang, and Jiang Xu. 2025. PICELF: An Automatic Electronic Layer Layout Generation Framework for Photonic Integrated Circuits. In2025 Design, Automation & Test in Europe Conference (DATE). IEEE, 1–7
work page 2025
-
[19]
Hao Chen, Yuzhe Ma, and Yeyu Tong. 2025. Bi-Level Optimization Accelerated DRC-Aware Physical Design Automation for Photonic Devices. In2025 Design, Automation & Test in Europe Conference (DATE). IEEE, 1–7
work page 2025
-
[20]
Yuchao Wu, Xiaofei Yu, Xianyi Feng, Yeyu Tong, and Yuzhe Ma. 2025. Constraints- aware Adaptive Routing with Hybrid Waveguides for Photonic Integrated Cir- cuits. In2025 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 1–8
work page 2025
-
[21]
Jingyu Pan, Guanglei Zhou, Chen-Chia Chang, Isaac Jacobson, Jiang Hu, and Yiran Chen. 2025. A survey of research in large language models for electronic design automation.ACM Transactions on Design Automation of Electronic Systems 30, 3 (2025), 1–21
2025
-
[22]
Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. 2023. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176(2023)
Pith/arXiv arXiv 2023
-
[23]
Haoyuan Wu, Zhuolun He, Xinyun Zhang, Xufeng Yao, Su Zheng, Haisheng Zheng, and Bei Yu. 2024. Chateda: A large language model powered autonomous agent for eda.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems43, 10 (2024), 3184–3197
work page 2024
-
[24]
Wenzhao Sun, Yanan Han, Bijian Lan, Qing Peng, and Jing Wan. 2025. Ana- sizecoder: Code generator for analog integrated circuit sizing automation via large language model. In2025 International Symposium of Electronics Design Automation (ISEDA). IEEE, 817–822
work page 2025
-
[25]
Yiting Wang, Wanghao Ye, Yexiao He, Yiran Chen, Gang Qu, and Ang Li. 2025. MCP4EDA: LLM-Powered Model Context Protocol RTL-to-GDSII Automation with Backend Aware Synthesis Optimization.arXiv preprint arXiv:2507.19570 (2025)
Pith/arXiv arXiv 2025
-
[26]
Bingyang Liu, Haoyi Zhang, Xiaohan Gao, Zichen Kong, Xiyuan Tang, Yibo Lin, Runsheng Wang, and Ru Huang. 2025. Layoutcopilot: An llm-powered multi-agent collaborative framework for interactive analog layout design.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems(2025)
work page 2025
-
[27]
Yao Lai, Sungyoung Lee, Guojin Chen, Souradip Poddar, Mengkang Hu, David Z Pan, and Ping Luo. 2025. Analogcoder: Analog circuit design via training-free code generation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 379–387
work page 2025
-
[28]
Yao Lai, Souradip Poddar, Sungyoung Lee, Guojin Chen, Mengkang Hu, Bei Yu, Ping Luo, and David Z Pan. 2025. Analogcoder-pro: Unifying analog circuit generation and optimization via multi-modal llms.arXiv preprint arXiv:2508.02518 (2025)
Pith/arXiv arXiv 2025
-
[29]
Jason Liu, Ankita Sharma, Cheick Doumbia, and Joyce KS Poon. 2024. Towards large-language model assisted layout of silicon photonic integrated circuits. In European Conference on Integrated Optics. Springer, 441–447
work page 2024
-
[30]
Yuchao Wu, Xiaofei Yu, Hao Chen, Yang Luo, Yeyu Tong, and Yuzhe Ma. 2025. PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design. In2025 Design, Automation & Test in Europe Conference (DATE). IEEE, 1–6
work page 2025
-
[31]
Ankita Sharma, YuQi Fu, Vahid Ansari, Rishabh Iyer, Fiona Kuang, Kashish Mistry, Raisa Islam Aishy, Sara Ahmad, Joaquin Matres, Dirk R Englund, et al
-
[32]
Chen-Chia Chang, Chia-Tung Ho, Yaguang Li, Yiran Chen, and Haoxing Ren
- [33]
-
[34]
InProceedings of the 2025 International Symposium on Physical Design
Drc-coder: Automated drc checker code generation using llm autonomous agent. InProceedings of the 2025 International Symposium on Physical Design. 143–151
work page 2025
-
[35]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems33 (2020), 9459–9474
2020
-
[36]
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al . 2024. A survey on in-context learning. InProceedings of the 2024 conference on empirical methods in natural language processing. 1107–1128
2024
-
[37]
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. 2025. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605 (2025)
Pith/arXiv arXiv 2025
-
[38]
John K Ousterhout. 1993. An Introduction to TCL and TK
work page 1993
-
[39]
Python Software Foundation. 2001. ast — Abstract syntax trees. https://docs. python.org/3/library/ast.html
work page 2001
-
[40]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837
2022
-
[41]
LangChain. 2025. LangChain. https://www.langchain.com/langchain
work page 2025
-
[42]
Python Software Foundation. 2026. pyflakes 3.4.0. https://pypi.org/project/ pyflakes/
work page 2026
-
[43]
Aixin Liu DeepSeek-AI, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. 2025. DeepSeek-V3. 2: Pushing the Frontier of Open Large Language Models.arXiv preprint arXiv:2512.02556(2025)
Pith/arXiv arXiv 2025
-
[44]
Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388
Pith/arXiv arXiv 2025
-
[45]
Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Yichun Yin, Hao Zhang, Yong Liu, Yasheng Wang, and Ruiming Tang. 2024. Coir: A comprehensive benchmark for code information retrieval models.URL https://arxiv. org/abs/2407.02883 (2024)
Pith/arXiv arXiv 2024
-
[46]
Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, et al. 2025. Embeddinggemma: Powerful and lightweight text representations. arXiv preprint arXiv:2509.20354(2025)
Pith/arXiv arXiv 2025
-
[47]
Mark Chen. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)
Pith/arXiv arXiv 2021
-
[48]
OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/
2025
-
[2025]
AI Agents for Photonic Integrated Circuit Design Automation.arXiv preprint arXiv:2508.14123(2025)
Pith/arXiv arXiv 2025
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.