Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Towards Optimal Circuit Generation: Multi-Agent Collaboration Meets Collective Intelligence

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A six-agent LLM framework with syntax locking, retrieval, and dual rewards can make a 14B-parameter model generate gate-level circuits whose efficiency matches or exceeds top human experts on 55.6% of benchmark tasks.

desk verdict Solid multi-agent framework results with real improvements over baselines, but the headline human-competitiveness claim rests on unverified SEI comparability and possible benchmark contamination. read the letter →

arxiv 2504.14625 v3 pith:EUG3MAG6 submitted 2025-04-20 cs.AR cs.AI

Haiyan Qin , Jiahao Feng , Xiaotong Feng , Wei W. Xing , Wang Kang This is my paper · ORCID
classification cs.ARcs.AI
keywords gate-levelcircuitgenerationmulti-agentLLMframeworksyntaxlockingretrieval-augmenteddual-rewardoptimizationSolutionEfficiencyIndexTC-BenchBoolean
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models writing hardware descriptions normally produce gate counts 38%–1075% above human designs. This paper argues that the cause is a Boolean Optimization Barrier: LLMs reason statistically and locally, and they leak behavioral abstractions instead of simplifying Boolean logic at the netlist level. The paper proposes CircuitMind, a six-agent framework that forces gate-level output through syntax locking, reuses optimized subcircuits through retrieval-augmented generation, and balances correctness against efficiency with a dual-reward loop. On a new benchmark, TC-Bench, built from the leaderboard of the competitive design game TuringComplete, the authors report that 55.6% of the framework's implementations reach or exceed top-tier human expert efficiency, and that a 14-billion-parameter model outperforms larger commercial models without fine-tuning. If this holds, collaborative architecture rather than model scale would be the lever for closing the AI–human gap in constrained optimization tasks.

What carries the argument

The load-bearing mechanism is CircuitMind's three-tier, six-agent hierarchy (UserProxy, Mediator, Reviewer, Summarizer, CoderAgent, Executor) combined with three constraints: Syntax Locking, which restricts every generated netlist to five basic gates so the model must do Boolean algebra rather than emit if-else behavior; Retrieval-Augmented Generation, which pulls known-good subcircuits from a knowledge database that the Summarizer grows as tasks succeed; and Dual-Reward Optimization, which scores output by both test-pass rate and a physical-efficiency metric and feeds those scores back through the Reviewer. The matching evaluation object is TC-Bench's Solution Efficiency Index (SEI), defined as $1/(\alpha G + \beta D)$ with $\alpha = \beta = 1$, aggregated across 28 tasks by geometric mean and compared against three human tiers derived from the TuringComplete leaderboard. Syntax Locking does the work of forcing genuine netlist-level reasoning; the RAG database does the work of bootstrapping without pre-training on netlists; dual-reward does the work of steering agent revisions toward efficiency rather than correctness alone.

What would settle it

Run the same SEI calculation on the actual top-ranked human netlists from TuringComplete after re-synthesizing them with the exact synthesis pipeline used for LLM outputs (standard optimization passes disabled, $\alpha = \beta = 1$). If the resulting human top-tier SEI range no longer contains 0.115, or fewer than five of the nine CircuitMind configurations land above the top-tier threshold, the central claim is falsified; if the values are unchanged, the claim holds.

Watch

Extended reading notes

Core claim

The central claim is that the efficiency gap between LLM-generated and human-designed circuits is not a scaling problem but a reasoning-structure problem, and that it can be closed by decomposing gate-level design across specialized agents. The paper reports that, with syntax locking (a grammar restricted to AND, OR, NOT, XOR, and NAND), retrieval of previously validated optimized patterns, and dual-reward feedback, the framework lifts hard-task functional correctness from, for example, 0.02 to 0.26 for the 14B model and raises overall Solution Efficiency Index by as much as 89–90 times for some commercial backends on hard tasks. Five of the nine evaluated model-plus-framework combinations fall inside the top human tier of the benchmark, and the best configuration, built on a 14B-parameter model, reaches an overall SEI of 0.115, higher than larger commercial models. The authors take this as evidence that the Boolean Optimization Barrier can be overcome by architectural innovation and structured collaboration rather than by additional training data or model scale.

Load-bearing premise

The human reference tiers in TC-Bench are computed from TuringComplete leaderboard data using the same SEI formula applied to LLM outputs, but the paper does not specify how gate count and delay were obtained for human designs or whether in-game scoring factors such as simulation ticks were included; if the human SEI values are not measured the same way as the synthesis-based SEI for LLM circuits, the claim that 55.6% of implementations match or exceed top-tier human experts collapses.

Editorial extensions

If this is right

  • Hard design tasks, where baseline LLMs solve almost none of the problems, become solvable: DeepSeek-R1 goes from 0.26 to 0.99 Pass@1 and Phi-4 from 0.02 to 0.26 under CircuitMind.
  • Small open-weight models become viable for gate-level design: the 14B-parameter Phi-4 implementation reaches the highest reported SEI (0.115), above two larger commercial baselines, so hardware teams could run the framework without huge inference budgets.
  • A benchmark built from a competitive game's leaderboard gives a human-aligned scale for future AI circuit generators: TC-Bench defines top, mid, and low human tiers by SEI ranges, and new systems can be ranked directly on that scale.
  • The framework's knowledge database grows with each successful task, so repeated use should compound design efficiency without retraining.
  • Because syntax locking removes behavioral abstraction, the resulting netlists are directly comparable to hand-optimized designs and can be evaluated without synthesis-tool optimization passes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the headline claim would re-synthesize the actual human leaderboard netlists under the same synthesis pipeline used for the LLM circuits; if in-game scoring or manual measurement inflated human efficiency, the 55.6% matching rate would shrink, while if human designs were measured more favorably to them, the gap is already conservative.
  • The same recipe—restricting output to a small primitive grammar, retrieving known-good patterns, and rewarding both correctness and physical cost—could transfer to other constrained generation problems such as instruction scheduling, register allocation, or structured synthetic chemistry design, where the search space is rich and human expert solutions are scarce.
  • The case study's reduction of a logical device from 96 gates to 8 gates suggests that baseline LLM failures are mostly search failures rather than missing knowledge; if so, a cheaper verification-guided or gradient-free search loop might capture part of the gain without a full six-agent stack.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CircuitMind, a multi-agent LLM framework for gate-level circuit generation, and TC-Bench, a benchmark built from human solutions on the TuringComplete game. CircuitMind combines syntax locking (restricting generated netlists to five basic gates), retrieval-augmented generation with a dynamic knowledge database, and a dual-reward feedback loop. The authors report that CircuitMind improves both functional correctness (Pass@1) and physical efficiency (SEI) across nine LLMs, and claim that 55.6% of CircuitMind implementations reach or exceed top-tier human expert efficiency, with a 14B Phi-4 model outperforming larger commercial models. They also include ablations on the RAG components and a case study. Code, data, and models are released.

Significance. If the central claims hold, the paper makes a meaningful contribution: it demonstrates that architectural collaboration among LLM agents, rather than model scale, can substantially improve gate-level efficiency in hardware design, and it proposes a human-aligned evaluation scale (TC-Bench) derived from a large competitive player base. The open-source release of code and data is a strength that supports reproducibility. The multi-agent design with syntax locking and RAG is a plausible and potentially reusable recipe. However, the headline human-competitiveness claim depends entirely on the comparability of the human SEI tiers to the Yosys-based SEI used for LLM-generated netlists, and that comparability is not demonstrated in the manuscript. The knowledge-evolution mechanism also risks leaking information across benchmark tasks. Because these issues affect the central claim, the paper requires major revision before its main conclusions can be accepted.

major comments (3)
  1. [III.C.1 and II.B] The human reference tiers are not defined with a measurable pipeline. Section III.C.1 reports SEI ranges for top/mid/low human tiers, but it does not state how gate count G and delay D were measured for human designs, whether the same 28 tasks were used, whether the same Yosys-based synthesis with optimization passes disabled was applied, or whether the TuringComplete in-game scoring (which Section II.B admits 'might incorporate additional factors like simulation ticks') was converted to the SEI of Eq. (2). Without a common measurement pipeline, the SEI values for humans and for LLM-generated netlists may be on different scales, and the claim that 5 of 9 CircuitMind implementations 'match or exceed top-tier human expert performance' (Section V.C) is not supported. Please specify the exact computation of the human SEI values and demonstrate that they are comparable to the Yosys-based SEI used for the models.
  2. [IV.D and V.A] The knowledge database evolves during the benchmark. Section IV.D states that Summarizer extracts optimized subcircuits from successfully completed tasks and adds them to the knowledge base, and Section IV.B.2 states that the database 'gradually accumulates more sophisticated modules' over time. If the 28 TC-Bench tasks are evaluated sequentially in one growing session, then later tasks can retrieve design patterns extracted from earlier benchmark tasks, meaning the tasks are not independent and the framework is effectively exposed to test-set information. To support the per-task SEI and Pass@1 results in Table IV and the ablation in Table V, the authors must either freeze the knowledge base during evaluation or clearly separate tasks into a development set and a held-out evaluation set with no knowledge transfer between them.
  3. [V.A, Table IV, and Table V] No statistical uncertainty is reported for any of the main results. SEI and Pass@1 values in Tables IV and V appear to be point estimates from a single configuration of sampling and prompting, despite the paper using n=20 samples in the Pass@k estimation. Since LLM outputs are stochastic, the differences that underpin the human-competitiveness claim (e.g., CM(Qwen2.5-Coder:14B) at SEI 0.097 vs. the top-tier boundary 0.0951, or CM(GPT-4o mini) at 0.104 vs. CM(Gemini) at 0.102) may be within run-to-run noise. Please report confidence intervals or variances across multiple independent runs and, where appropriate, a significance test for the comparisons that support the abstract's '55.6%' claim.
minor comments (6)
  1. [II.B, Eq. (2)] The text notes that TuringComplete scoring may involve additional factors like simulation ticks, but the human tiers in Section III.C.1 are presented without explaining how that in-game score maps to the SEI of Eq. (2). A short explicit mapping (or a statement that the in-game score is exactly alpha=beta=1 weighted gates+delay) would remove ambiguity.
  2. [Table II] Table II has a formatting error: the rows for MAGE, BetterV, and CodeV are concatenated in the printed text ('NoBetterV', 'NoCodeV'), making the table hard to read. Please fix the table layout.
  3. [V.B.2] The sentence 'with CM(GPT-4o mini) and CM(Gemini-2.0) increasing overall SEI by 271% and 61.9' is incomplete; the percentage for Gemini 2.0 Flash is cut off. Please provide the full number.
  4. [IV.B.3] The 'Dual-Reward Optimization' component is described as a feedback loop, but no explicit reward function, update rule, or pseudocode is given. Since the paper does not train the LLMs, clarify that 'optimization' refers to prompt-level, in-context refinement rather than parameter updates.
  5. [References] Reference [27] is missing the final period after 'Accessed: 2024-12-25', and the URL is split across lines without a line-break marker. Please also verify that the reference list follows the journal style consistently.
  6. [V.C] The paper states that CM(Qwen2.5-Coder:14B) (SEI 0.097) is 'near the boundary' of the top-tier range [0.0951, 0.1252], which is technically inside the range. Either drop 'near the boundary' or explain that it is close to the lower cutoff.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the main claims rest on measured SEI comparisons and external baselines, not on self-referential definitions, though the human-tier measurement pipeline is under-specified.

full rationale

The claimed derivation chain is experimental rather than formal: CircuitMind is optimized with the same SEI metric used to define human tiers, but that is a shared yardstick, not a circular input. The top-tier thresholds [0.0951, 0.1252] are empirical aggregates from leaderboard ranks; LLM SEI values are measured independently from Yosys gate counts and delays of generated netlists (Sec. V.A). No equation is substituted into itself, and no load-bearing self-citation appears; all framework components are evaluated against external baselines (GPT-4o mini, Gemini, DeepSeek, Qwen, Phi-4). Two caveats are correctness risks rather than circularity: Sec. II.B itself notes TuringComplete scoring 'might incorporate additional factors like simulation ticks,' and Sec. III.C does not document how human gate counts and delays were measured, so the human/LLM SEI scales may not be commensurable. Also, Sec. IV.D states the RAG knowledge database evolves as design tasks are completed; if the same 28 benchmark tasks seed that database during evaluation, later tasks are partly in-sample, which would weaken the generalization claim but is not a self-referential derivation. For these reasons, no circularity step meets the quotation-and-reduction bar.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the comparability of game-derived human scores to Yosys-measured model SEI, the representativeness of the 28 tasks, and non-contamination of LLM training data. The SEI weights and tier cutoffs are hand-picked and directly affect the headline comparison.

free parameters (2)
  • SEI weights alpha, beta = alpha=1, beta=1
    Equation (2) weights gate count and delay equally. This choice determines the numerical SEI values and the human tier ranges. Different weights could reshuffle the ranking of models relative to human tiers.
  • Human tier rank cutoffs = Top: 1-10, Mid: 11-300, Low: 301-1000
    Section III.C partitions TuringComplete leaderboard ranks into three tiers. These cutoffs are arbitrary and directly determine which human SEI range the models are compared against.
assumptions (4)
  • domain assumption TuringComplete leaderboard SEI scores are directly comparable to Yosys-synthesized netlist SEI.
    Human tiers in Section III.C are derived from game leaderboard ranks, but the game may use additional scoring factors (Section II.B mentions simulation ticks). No conversion is described.
  • domain assumption The 28 TC-Bench tasks are representative of gate-level circuit design.
    Section III.B selects tasks based on circuit complexity, functional diversity, and optimization potential, but there is no external validation that these tasks generalize to real hardware design.
  • domain assumption LLM training data does not contain TuringComplete solutions (no contamination).
    The paper does not discuss contamination despite the benchmark being derived from a public game. If models saw these netlists in pretraining, the comparison to humans is invalid.
  • domain assumption Yosys with optimization passes disabled faithfully counts the gates produced by the model.
    Section V.A uses this to ensure results reflect model structure, but the mapping from the model's netlist format to Yosys gate count is not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Optimal Circuit Generation: Multi-Agent Collaboration Meets Collective Intelligence." pith.science (2026). https://pith.science/paper/EUG3MAG6

@misc{pith2026250414625,
  author       = {Pith},
  title        = {Pith review of: Towards Optimal Circuit Generation: Multi-Agent Collaboration Meets Collective Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EUG3MAG6}},
  note         = {Machine review of arXiv:2504.14625}
}
read the original abstract

Large language models (LLMs) have transformed code generation, yet their application in hardware design produces gate counts 38\%--1075\% higher than human designs. We present CircuitMind, a multi-agent framework that achieves human-competitive efficiency through three key innovations: syntax locking (constraining generation to basic logic gates), retrieval-augmented generation (enabling knowledge-driven design), and dual-reward optimization (balancing correctness with efficiency). To evaluate our approach, we introduce TC-Bench, the first gate-level benchmark harnessing collective intelligence from the TuringComplete ecosystem -- a competitive circuit design platform with hundreds of thousands of players. Experiments show CircuitMind enables 55.6\% of model implementations to match or exceed top-tier human experts in composite efficiency metrics. Most remarkably, our framework elevates the 14B Phi-4 model to outperform both GPT-4o mini and Gemini 2.0 Flash, achieving efficiency comparable to the top 25\% of human experts without requiring specialized training. These innovations establish a new paradigm for hardware optimization where collaborative AI systems leverage collective human expertise to achieve optimal circuit designs. Our model, data, and code are open-source at https://github.com/BUAA-CLab/CircuitMind.

Figures

Figures reproduced from arXiv: 2504.14625 by the authors.

Figure 1
Figure 1. LLM-based EDA Evolution: From Behavioral to Phys [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. CircuitMind System Architecture IV. CIRCUITMIND: MULTI-AGENT COLLABORATION To overcome the Boolean Optimization Barrier in circuit design, we introduce CircuitMind—a hierarchical multi-agent framework that distributes complex reasoning tasks across specialized agents, enabling capabilities that individual LLMs cannot achieve alone. A. Architectural Overview CircuitMind adopts a three-tier architecture inspired by en… view at source ↗
Figure 4
Figure 4. CoderAgent and Reviewer dynamic prompting process [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Overall Pass@1 and SEI Comparison: Base LLMs vs. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: SEI vs. Model Size Compared to Human Expert Tiers. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 8 canonical work pages

  1. [1]

    Turingcomplete game,

    LevelHead, “Turingcomplete game,” 2021, https://turingcomplete.game/ Accessed: 2025-04-21

    work page 2021

  2. [2]

    Yosys open synthesis suite,

    C. Wolf, “Yosys open synthesis suite,” 2016

    2016

  3. [3]

    Phi- 4 technical report,

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann et al. , “Phi- 4 technical report,” arXiv preprint arXiv:2412.08905 , 2024

    arXiv 2024

  4. [4]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  5. [5]

    Abc: An academic industrial-strength verification tool,

    R. Brayton and A. Mishchenko, “Abc: An academic industrial-strength verification tool,” in Computer Aided Verification: 22nd International Conference, CAV 2010, Edinburgh, UK, July 15-19, 2010. Proceedings

    work page 2010

  6. [6]

    Verigen: A large language model for verilog code generation,

    S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri, and S. Garg, “Verigen: A large language model for verilog code generation,” ACM Transactions on Design Automation of Electronic Systems, vol. 29, no. 3, pp. 1–31, 2024

    2024

  7. [7]

    Rtl- coder: Fully open-source and efficient llm-assisted rtl code generation technique,

    S. Liu, W. Fang, Y . Lu, J. Wang, Q. Zhang, H. Zhang, and Z. Xie, “Rtl- coder: Fully open-source and efficient llm-assisted rtl code generation technique,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2024

    2024

  8. [8]

    Verilogeval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) . IEEE, 2023, pp. 1–8

    2023

Show all 30 references
  1. [9]

    Pyranet: A multi-layered hierarchical dataset for verilog,

    B. Nadimi, G. O. Boutaib, and H. Zheng, “Pyranet: A multi-layered hierarchical dataset for verilog,” arXiv preprint arXiv:2412.06947, 2024

    2024

  2. [10]

    Mage: A multi-agent engine for automated rtl code generation,

    Y . Zhao, H. Zhang, H. Huang, Z. Yu, and J. Zhao, “Mage: A multi-agent engine for automated rtl code generation,” arXiv preprint arXiv:2412.07822, 2024

    2024 arXiv

  3. [11]

    Aivril: Ai-driven rtl gen- eration with verification in-the-loop,

    H. Sami, P.-E. Gaillardon, V . Tenace et al. , “Aivril: Ai-driven rtl gen- eration with verification in-the-loop,” arXiv preprint arXiv:2409.11411 , 2024

    2024 arXiv

  4. [12]

    Betterv: Con- trolled verilog generation with discriminative guidance,

    Z. Pei, H.-L. Zhen, M. Yuan, Y . Huang, and B. Yu, “Betterv: Con- trolled verilog generation with discriminative guidance,” arXiv preprint arXiv:2402.03375, 2024

    2024 arXiv

  5. [13]

    Codev: Empowering llms for verilog generation through multi-level summarization,

    Y . Zhao, D. Huang, C. Li, P. Jin, Z. Nan, T. Ma, L. Qi, Y . Pan, Z. Zhang, R. Zhang et al. , “Codev: Empowering llms for verilog generation through multi-level summarization,” arXiv preprint arXiv:2407.10424 , 2024

    2024 arXiv

  6. [14]

    Rtlsquad: Multi- agent based interpretable rtl design,

    B. Wang, Q. Xiong, Z. Xiang, L. Wang, and R. Chen, “Rtlsquad: Multi- agent based interpretable rtl design,” arXiv preprint arXiv:2501.05470 , 2025

    2025 arXiv

  7. [15]

    Geneda: Unleashing generative reasoning on netlist via multimodal encoder-decoder aligned foundation model,

    W. Fang, J. Wang, Y . Lu, S. Liu, and Z. Xie, “Geneda: Unleashing generative reasoning on netlist via multimodal encoder-decoder aligned foundation model,” arXiv preprint arXiv:2504.09485 , 2025

    2025 arXiv

  8. [16]

    Hivegen– hierarchical llm-based verilog generation for scalable chip design,

    J. Tang, J. Qin, K. Thorat, C. Zhu-Tian, Y . Cao, C. Dinget al., “Hivegen– hierarchical llm-based verilog generation for scalable chip design,”arXiv preprint arXiv:2412.05393, 2024

    2024

  9. [17]

    Promptv: Leveraging llm-powered multi-agent prompting for high-quality verilog generation,

    Z. Mi, R. Zheng, H. Zhong, Y . Sun, and S. Huang, “Promptv: Leveraging llm-powered multi-agent prompting for high-quality verilog generation,” arXiv preprint arXiv:2412.11014 , 2024

    2024 arXiv

  10. [18]

    The problem of simplifying truth functions,

    W. V . Quine, “The problem of simplifying truth functions,” The Amer- ican mathematical monthly , vol. 59, no. 8, pp. 521–531, 1952

    1952

  11. [19]

    Rtllm: An open-source benchmark for design rtl generation with large language model,

    Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) . IEEE, 2024, pp. 722–727

    2024

  12. [20]

    Resbench: Benchmarking llm-generated fpga designs with resource awareness,

    C. Guo and T. Zhao, “Resbench: Benchmarking llm-generated fpga designs with resource awareness,” arXiv preprint arXiv:2503.08823 , 2025

    2025 arXiv

  13. [21]

    Genben: A genarative benchmark for llm-aided design

    G.-W. Wan, S. Wong, M. Xing, N. Guan, N. Xu, Q. Xu, X. Wang et al., “Genben: A genarative benchmark for llm-aided design.”

  14. [22]

    Springer, 2010, pp. 24–40

    2010

  15. [23]

    Rtl-repo: A benchmark for evaluating llms on large-scale rtl design projects,

    A. Allam and M. Shalan, “Rtl-repo: A benchmark for evaluating llms on large-scale rtl design projects,” in 2024 IEEE LLM Aided Design Workshop (LAD). IEEE, 2024, pp. 1–5

    2024

  16. [24]

    Retrieval-augmented generation for knowledge- intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge- intensive nlp tasks,” in Advances in Neural Information Processing Systems, H. Laroc...

    2020

  17. [25]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  18. [26]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  19. [27]

    Gemini 2.0 flash experimental

    DeepMind, “Gemini 2.0 flash experimental.” 2024, https://deepmind. google/technologies/gemini/flash Accessed: 2024-12-25

    2024

  20. [28]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  21. [30]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  22. [9474]

    Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf

    [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.