Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Improving LLM-Powered EDA Assistants with RAFT

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning an LLM on synthetic Q&A pairs generated from unlabeled documents substantially improves its answers in retrieval-augmented electronic-design automation (EDA) assistants.

desk verdict A useful applied RAFT-for-EDA paper whose synthetic-data headline is undercut by a circular test set and a sub-noise gain on human questions. read the letter →

arxiv 2506.06500 v1 pith:5YJC4BUR submitted 2025-06-06 cs.CL

Luyao Shi , Michael Kazda , Charles Schmitter , Hemlata Gupta This is my paper
classification cs.CL
keywords Retrieval-AugmentedFine-Tuningsyntheticquestion-answerdataElectronicDesignAutomationLLMGenerationleakageaccesscontrolBERTScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes replacing the scarce human-labeled question/answer data needed for Retrieval-Augmented Fine-Tuning (RAFT) with synthetic Q&A pairs generated by a large model from unlabeled EDA documents. The authors fine-tune an 8B Llama model on 305 human Q&A posts plus 900 synthetic pairs, reporting that it outscored several larger base models on both a human-written test set and a synthetic test set (BERTScore F1 of 84.47 versus 79.84 on human Q2A, and 83.07 versus 77.80 on synthetic SynthQA). They also find that using real user questions as few-shot examples during synthetic data generation brings further gains specifically on human-style questions. The paper argues this makes domain-specific assistants feasible without large labeled datasets, while acknowledging that the synthetic test answers were not verified by humans.

What carries the argument

The central object is Retrieval-Augmented Fine-Tuning (RAFT), a training setup that fine-tunes an LLM on prompts containing a question, a set of retrieved document chunks (which may or may not include the relevant one), and the ground-truth answer, teaching the model to integrate retrieved context correctly. The argument is carried by the synthetic-data pipeline: DeepSeek-V3 generates one Q&A pair per unlabeled EDA document, with prompts that adapt question difficulty to document complexity, and optionally conditions on real user questions retrieved via BM25 as few-shot style examples. Evaluation uses hybrid retrieval (semantic search plus BM25, fused by Reciprocal Rank Fusion) to build prompts, and BERTScore and BARTScore to measure answer quality.

What would settle it

Have experienced EDA engineers score a blinded sample of the D2 model's answers against the baseline model's answers for factual correctness on the SynthQA test set. If the D2 model does not receive higher expert correctness ratings, the reported synthetic-data RAFT gains are an artifact of distribution matching rather than improved domain competence.

Watch

Extended reading notes

Core claim

The central claim is that RAFT fine-tuning on synthetic Q&A data—where a strong generator (DeepSeek-V3) produces questions and answers from the same document store the assistant will later retrieve from—significantly improves a smaller LLM's RAG performance in the EDA domain. The trained EDA-Llama-8B-D2 model beats not only the untuned Llama-3.1-8B baseline but also much larger models such as Llama-3.1-70B, both on the human-authored Q2A test set and on the synthetic SynthQA test set. The authors show that mixing a small amount of real expert Q&A posts (305 samples) with the synthetic data yields the best results, and that adding retrieval-augmented few-shot examples from real user questions brings further gains on Q2A. They also report exploratory evidence that fine-tuning on this data leads to some memorization: the model can produce answers even when the relevant retrieved context is removed, which raises concerns for access control over sensitive documents.

Load-bearing premise

The synthetic test set SynthQA was generated by DeepSeek-V3 from the same document pool used to create the synthetic training data, and its answers were never checked by humans, so the large SynthQA gains may reflect the model learning to imitate DeepSeek-V3's answer style rather than gaining real EDA knowledge.

Editorial extensions

If this is right

  • Organizations can build domain-specific RAG assistants without collecting large labeled Q/A datasets, using unlabeled documentation as the raw material.
  • A relatively small 8B model fine-tuned with RAFT on synthetic data can match or exceed much larger general-purpose models on domain tests.
  • Including a modest amount of real expert Q&A data alongside synthetic data produces the best balance of precision and recall.
  • Using real user questions as retrieval-augmented few-shot examples during synthetic data generation improves performance on human-style questions.
  • Fine-tuning on curated domain data carries a memorization risk, so sensitive documents should be kept out of RAFT training or separated by adapter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The SynthQA gains may be inflated: since the same generator and document pool produced both the synthetic training data and the SynthQA test answers, the fine-tuned model could be matching DeepSeek-V3's answer style and distribution rather than acquiring real EDA competence; a human-expert evaluation of generated answers would settle this.
  • Because the synthetic-data pipeline only needs unlabeled documents, the same recipe should transfer to other technical domains (e.g., legal, medical, semiconductor manufacturing) where labeled Q&A is scarce but documentation is plentiful.
  • The paper's memorization findings imply that access control at retrieval time is not sufficient to protect sensitive data once a model has been fine-tuned on it; testable mitigations include removing sensitive documents from training, using per-group LoRA adapters, or adding 'I don't know' training samples—though the latter currently harms performance when context is present.
  • The complexity-adaptive question generation (simple vs. reasoning-level questions based on document type) is a promising but untested ingredient; ablating it would show whether it is responsible for the gains or just a prompt detail.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes using synthetic question-answer (Q/A) data, generated by DeepSeek-V3 from unlabeled EDA documents, to fine-tune Llama-3.1-8B with Retrieval-Augmented Fine-Tuning (RAFT) for EDA question answering. The authors also combine this with refined Q2A posts from a Stack Overflow-style internal system, optionally using real user questions as retrieval-augmented few-shot (RAFS) examples in the synthetic-data prompt. Evaluation uses BERTScore and BARTScore on a Q2A test set (human questions with automatically refined answers) and a SynthQA test set (fully synthetic questions/answers). Additional experiments examine memorization and "MissingContext" behavior for access control. The central claim is that RAFT with synthetic data significantly boosts LLM performance for RAG-based EDA tasks.

Significance. If the central claim were established, the paper would make a practically valuable contribution: it would show that scarce expert-labeled Q/A data in specialized domains can be replaced or augmented with synthetic data generated from unlabeled documents, lowering the barrier to building domain-specific RAG assistants. The paper also addresses an important and timely deployment concern—unintended memorization and data leakage during fine-tuning—and proposes a concrete access-control mechanism. The strengths are the use of standard fine-tuning and evaluation techniques, the connection to a real deployed system, and the explicit discussion of memorization risks with recommendations. However, the evaluation design has a serious circularity issue: the SynthQA test set is generated by the same DeepSeek-V3 model, with the same prompt and document pool, as the synthetic training data. The independent human-authored signal (Q2A) shows only a very small improvement from synthetic data, so the headline claim is not yet supported.

major comments (3)
  1. [Section III.A, Table II, Figure 3] The SynthQA test set consists of 100 questions and reference answers generated by DeepSeek-V3 from the same document pool and with the same prompt template (Figure 3) used to create the synthetic training data. The large D2-versus-D1 gains on SynthQA (BERTScore F1 +4.06, BARTScore F1 +5.45) are therefore confounded by the model matching the generator's answer style and distribution. Because the references are unverified by humans and come from the same generative model, these gains do not on their own establish improved EDA competence. The paper's own conclusion (Section IV) acknowledges the lack of human verification but does not repair the circularity.
  2. [Section III.A, Table II, Section IV] On the Q2A test set, the isolated effect of adding synthetic data (D2 versus D1) is only +0.21 BERTScore F1 and +0.33 BARTScore F1. No training seeds, multiple runs, or confidence intervals are reported, so for a 100-sample test these differences are within plausible noise. The abstract's claim that "RAFT with synthetic data significantly boosts LLM performance" is therefore not supported by the human-authored (or partially human-authored) Q2A results. The conclusion's statement that "the results from the Q2A sets demonstrate a clear and promising trend" is not justified by the magnitude of the observed D2-versus-D1 differences on that set.
  3. [Section III.A, Section II.A] The Q2A test set is not fully independent human-authored data: although the questions originate from human posts, the reference answers were refined by DeepSeek-V3 using the prompt in Figure 2 before being used in both training and test. This means the Q2A test also carries the refinement model's stylistic signature, reducing the strength of the claim that the Q2A results are an independent check on the synthetic-data benefit. The manuscript should either use unmodified human answer texts as references or, better, include a human-verified evaluation subset, to separate genuine competence gains from style matching.
minor comments (5)
  1. [Section II.A, Figure 2 vs Figure 3] In Section II.A, the sentence "These Top-K questions are used as few-shot examples in the prompt (highlighted in the dashed box in Figure 2)" appears to refer to the wrong figure: Figure 2 is the Q2A refinement prompt, while the dashed box for retrieved example questions is in Figure 3 (the synthetic Q&A generation prompt). Please correct the reference.
  2. [Section III heading] The heading "E VALAUTION" appears to be a misspelling of "EVALUATION."
  3. [Table I] In Table I, the category "Paramter Reference" should be "Parameter Reference."
  4. [Section III.B] The paper does not report the random seed used for the train/test split of the 405 Q2A posts and for the 90/10 split of synthetic documents, nor does it report performance variance across seeds. Reporting these would help assess the stability of the small Q2A differences in Table II.
  5. [Section II.B] The descriptions of Top-N and retrieval chunk size in Section II.B are given as hyperparameters, but the sensitivity of the final results to these choices (e.g., N=10, chunk=2000 characters) is not explored. A brief note on robustness would strengthen the practical guidance.

Circularity Check

2 steps flagged · score 6.0 of 10

The main synthetic-data benefit is measured on a SynthQA test generated by the same DeepSeek-V3 model, from the same document pool, and with the same prompt as the training data, while the independent Q2A gain is only +0.21 BERT F1; the 'significantly boosts' claim partially reduces to a same-generator train/test overlap.

  1. fitted input called prediction [Section III.A (Datasets), Table II, Section II.A]
    "Following the steps in II.A, we used DeepSeek-V3 to generate one synthetic Q&A pair for each EDA document. We refer to this test set as SynthQA throughout the paper."

    SynthQA is the test set on which the paper's synthetic-data benefit is largest: D2 beats D1 by 4.06 BERT F1 (83.07 vs 79.01) and 5.45 BART F1 (31.47 vs 26.02). But SynthQA is generated by the same DeepSeek-V3 model, from the same document pool, and using the same Figure 3 prompt as the 900 synthetic training samples added in D2, and the SynthQA references are explicitly unverified by humans. A model fine-tuned on DeepSeek-V3-generated answers is therefore rewarded on SynthQA for matching the generator's style and distribution rather than for improved EDA competence. The human-authored Q2A set does not repair the claim because D2 vs D1 there is only +0.21 BERT F1, a gain not distinguishable from noise without seeds or error bars.

  2. fitted input called prediction [Section III.A (Datasets), Section II.A]
    "The remaining 405 Q&A posts were refined using DeepSeek-V3 and then randomly split into a training set of 305 samples and a test set of 100 samples."

    Even the supposedly independent human-authored Q2A test is partly shaped by the same LLM used to produce the synthetic training data: both the Q2A training answers and the Q2A test answers are rewritten by DeepSeek-V3 using the same refinement prompt shown in Figure 2. Consequently, the Q2A evaluation also rewards models that have learned DeepSeek-V3's answer-refinement style, weakening the claim that Q2A provides separate human validation of the synthetic-data benefit. This is less severe than the SynthQA overlap because the underlying questions and semantic content are human-authored, but it still reduces some of the apparent cross-test transfer to a shared-generator effect.

full rationale

The paper is not globally circular: the RAFT training procedure, hybrid retrieval, baselines, and the access-control/memorization experiments are self-contained and do not reduce to their own inputs. The circularity is concentrated in the headline claim that synthetic data 'significantly boosts' performance. The largest numerical support for that claim comes from the SynthQA test set, which is generated by DeepSeek-V3 from the same document pool and with the same prompt as the synthetic training samples in D2; the test references are unverified by humans, and the conclusion itself concedes this. The paper tries to fall back on the Q2A results as an independent trend, but that fallback fails twice: the Q2A test answers were also DeepSeek-V3-refined, so they are not fully independent of the generator, and the D2 vs D1 Q2A increment is only 0.21 BERT F1 (84.47 vs 84.26), which is sub-noise in the absence of reported seeds or error bars. The SynthQA D2-vs-D1 gain of 4.06 BERT F1 is therefore best understood as a same-generator distribution match rather than demonstrated EDA competence, giving the central synthetic-data prediction a partial circularity score of 6.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality and representativeness of synthetic data, the validity of automated text-similarity metrics, and the assumption that retrieval finds relevant context. These are all domain assumptions, not mathematically derived facts. The free parameters are standard hyperparameters and retrieval settings chosen by the authors without sensitivity analysis. No new theoretical entities are introduced.

free parameters (6)
  • Top-N retrieved chunks = 10
    Number of document chunks included in the RAG prompt during training and inference; chosen by the authors, affects context and performance.
  • Retrieval chunk size = 2,000 characters, 200-character overlap
    Chunking configuration selected before training; determines how much retrieved text fits in the prompt.
  • Top-K RAFS examples = 5
    Number of real user Q/A pairs retrieved by BM25 to act as few-shot examples in synthetic data generation; chosen without sensitivity analysis.
  • LoRA rank and alpha = rank=128, alpha=32, dropout=0
    Parameter-efficient fine-tuning configuration; standard in the PEFT/LoRA ecosystem, not derived from data.
  • Training epochs = 5
    All models trained for exactly five epochs; no early stopping or epoch search is reported.
  • Synthetic Q/A count per document = 1
    One synthetic question-answer pair is generated per sampled document; generating more pairs could change results.
assumptions (4)
  • domain assumption DeepSeek-V3 generates high-quality, representative synthetic Q/A pairs from EDA documents.
    Invoked in Section II.A and used for both the SynthQA training set and the SynthQA test set; no human verification of these pairs is performed.
  • domain assumption BERTScore and BARTScore provide valid measures of answer quality for EDA Q/A.
    Used in Section III.C as the only evaluation metrics; no correlation with human judgments is established in this paper.
  • domain assumption Hybrid retrieval with Reciprocal Rank Fusion surfaces the relevant document chunks for both human and synthetic questions.
    Assumed throughout Sections II.B and II.D; retrieval quality itself is not measured, and the missing-context experiment shows the model can still answer, suggesting other documents may contain answers.
  • domain assumption The proprietary EDA document collection contains the information needed to answer the test questions.
    RAG performance depends on the corpus covering the answer sources; no analysis of answer coverage is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LLM-Powered EDA Assistants with RAFT." pith.science (2026). https://pith.science/paper/5YJC4BUR

@misc{pith2026250606500,
  author       = {Pith},
  title        = {Pith review of: Improving LLM-Powered EDA Assistants with RAFT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5YJC4BUR}},
  note         = {Machine review of arXiv:2506.06500}
}
read the original abstract

Electronic design engineers often struggle to efficiently access relevant information for tasks like design verification and technology development. While large language models (LLMs) can enhance productivity as conversational agents, pre-trained open-source LLMs lack domain-specific knowledge for Electronic Design Automation (EDA). In a Retrieval-Augmented Generation (RAG) context, LLMs rely on external context but may still produce inaccurate responses. Retrieval-Augmented Fine-Tuning (RAFT) improves LLM performance, but acquiring labeled question/answer (Q/A) data in EDA is difficult. To address this, we propose using synthetic Q/A datasets to enhance LLMs with RAFT. Our results show that RAFT with synthetic data significantly boosts LLM performance for RAG-based EDA tasks. We also investigate the impact of using real user questions as Retrieval-Augmented Few-Shot (RAFS) examples for synthetic data generation. Additionally, we implement secure access control to ensure sensitive information is only accessible to authorized personnel. Finally, we assess the risk of data leakage and unintended memorization during fine-tuning with synthetic data, providing practical insights.

Figures

Figures reproduced from arXiv: 2506.06500 by the authors.

Figure 1
Figure 1. Overview of the workflow for training data preparation, RAFT training, and inference. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prompt used for refining answers in Q2A posts for better grammar, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Examples of synthetic Q&A pairs. Questions from Design Guides [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Prompt for generating synthetic question-answer pairs from a target [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: Prompt used for RAFT training and RAG-based inference. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

    2020

  3. [3]

    Chipnemo: Domain- adapted llms for chip design,

    M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu et al. , “Chipnemo: Domain- adapted llms for chip design,” arXiv preprint arXiv:2311.00176 , 2023

    arXiv 2023

  4. [4]

    Ask-eda: A design assistant empowered by llm, hybrid rag and abbreviation de-hallucination,

    L. Shi, M. Kazda, B. Sears, N. Shropshire, and R. Puri, “Ask-eda: A design assistant empowered by llm, hybrid rag and abbreviation de-hallucination,” in 2024 IEEE LLM Aided Design Workshop (LAD) . IEEE, 2024, pp. 1–5

    work page 2024

  5. [5]

    Chateda: A large language model powered autonomous agent for eda,

    H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2024

    work page 2024

  6. [6]

    Openroad-assistant: An open-source large language model for physical design tasks,

    U. Sharma, B.-Y . Wu, S. R. D. Kankipati, V . A. Chhabria, and A. Rovinski, “Openroad-assistant: An open-source large language model for physical design tasks,” in Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD , 2024, pp. 1–7

    work page 2024

  7. [7]

    Raft: Adapting language model to domain specific rag,

    T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez, “Raft: Adapting language model to domain specific rag,” in First Conference on Language Modeling , 2024

    work page 2024

  8. [8]

    Eda corpus: A large language model dataset for enhanced interaction with openroad,

    B.-Y . Wu, U. Sharma, S. R. D. Kankipati, A. Yadav, B. K. George, S. R. Guntupalli, A. Rovinski, and V . A. Chhabria, “Eda corpus: A large language model dataset for enhanced interaction with openroad,” IEEE, New York, NY , June 2024

    work page 2024

Show all 18 references
  1. [9]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019

    2019

  2. [10]

    The probabilistic relevance frame- work: Bm25 and beyond,

    S. Robertson, H. Zaragoza et al. , “The probabilistic relevance frame- work: Bm25 and beyond,” F oundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009

    2009

  3. [11]

    Elasticsearch,

    Elastic N.V ., “Elasticsearch,” 2024. [Online]. Available: https://www. elastic.co/elasticsearch

    2024

  4. [12]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  5. [13]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  6. [14]

    Unsloth,

    M. H. Daniel Han and U. team, “Unsloth,” 2023. [Online]. Available: http://github.com/unslothai/unsloth

    2023

  7. [15]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen et al. , “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  8. [16]

    IBM SLATE-125M English Retriever v2 Model Card,

    IBM, “IBM SLATE-125M English Retriever v2 Model Card,” 2024, accessed: 2025-05-22. [Online]. Avail- able: https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ fm-slate-125m-english-rtrvr-v2-model-card.html?context=wx

    2024

  9. [17]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675 , 2019

    1904 arXiv

  10. [18]

    Bartscore: Evaluating generated text as text generation,

    W. Yuan, G. Neubig, and P. Liu, “Bartscore: Evaluating generated text as text generation,” Advances in neural information processing systems , vol. 34, pp. 27 263–27 277, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.