Pith. sign in

REVIEW 5 major objections 5 minor 46 references

Never Come Up Empty: Adaptive HyDE Retrieval for Improving LLM Developer Support

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAG pipelines that embed a hypothetical answer, retrieve full Stack Overflow accepted answers, and relax the similarity threshold when needed produce higher-scoring LLM answers than zero-shot prompting across most open-source models.

desk verdict A large, transparent RAG pipeline comparison with a useful HB1 result, but the RQ3 prompt confound and thin judge validation keep the headline claims from being settled. read the letter →

arxiv 2507.16754 v1 pith:2P3J2HGF submitted 2025-07-22 cs.SE cs.AI

Fangjian Lei , Mariam El Mezouar , Shayan Noei , Ying Zou This is my paper
classification cs.SEcs.AI
keywords Retrieval-AugmentedGenerationHyDEStackOverflowLLM-as-a-Judgeadaptivethresholdingdevelopersupportanswerqualityopen-sourceLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what retrieval-augmented generation (RAG) can do for LLM answers to programming questions, and which pipeline choices actually matter. The paper builds a knowledge base of 3.4 million Java and Python Stack Overflow posts with accepted answers, then compares 63 pipeline variants that vary in query type, retrieval target, content granularity, and similarity threshold. The central claim is that the best configuration—generating a hypothetical answer with HyDE, retrieving full accepted answers, and lowering the similarity threshold only when needed—produces answers that score higher in helpfulness, correctness, and detail than zero-shot prompting for most open-source LLMs, and often higher than Stack Overflow's own accepted answers. A sympathetic reader would take this as evidence that retrieval makes open-source LLMs measurably more useful for everyday developer questions without changing the underlying model.

What carries the argument

The mechanism that carries the argument is the HB1 retrieval pipeline: (1) HyDE query generation, in which a separate LLM writes a hypothetical answer to the question and that pseudo-answer is embedded instead of the raw question; (2) direct retrieval against full accepted answers, rather than against individual answer sentences or similar questions; and (3) an adaptive similarity threshold that begins at 0.9 and relaxes by 0.1 each time nothing clears the bar. The load-bearing idea is that a pseudo-answer is semantically closer to the target answer than the question wording is, so embedding it improves the match, while full-answer granularity preserves the rationale and edge cases that sentence fragments lose. The adaptive threshold turns a hard 'no match' failure into a graceful degradation: every question receives the best context available, and quality falls off only when the threshold has to drop far enough to pull in weakly related content. The embeddings themselves are computed with the all-mpnet-base-v2 sentence-embedding model, and cosine similarity against the corpus decides what is retrieved.

What would settle it

Take the same 385 seen and 5,510 unseen questions and run a blind head-to-head human preference study between HB1 answers and zero-shot answers, with raters who do not know which is which. If human raters prefer HB1 at roughly the same rate as the judge's scores predict, the claim survives; if they do not, the reported quality advantage is an artifact of the evaluator. A cheaper quantitative check is to rerun the RQ2 and RQ3 comparisons with a judge from a different model family and see whether the weighted mean advantage (5.76 vs. 5.04 for unseen questions) and the Wilcoxon results across the four LLMs persist.

Watch

Extended reading notes

Core claim

The paper's central discovery is that pipeline HB1—direct retrieval of full accepted answers using a GPT-4o-generated hypothetical answer as the query—dominates the other six pipeline families on both answer quality and retrieval coverage. Across similarity thresholds from 0.1 to 0.9, HB1 posts the highest mean judge score (6.11 overall, reaching 7.00 at threshold 0.9) while still retrieving context for 80% of seen questions at threshold 0.7. On 5,510 questions posted after the model's release cutoff, adaptive thresholding—starting at 0.9 and stepping down by 0.1 until context is found—reaches 100% coverage and yields a weighted mean judge score of 5.76 versus 5.04 for the accepted answers, a statistically significant difference with small positive effect size. Across four open-source LLMs, HB1 significantly outperforms each model's zero-shot responses for three of them; for the strongest model, Qwen3-8B, the difference is not significant. The paper concludes that a HyDE-based, full-answer RAG pipeline with adaptive similarity thresholds consistently enhances answer quality for a wide range of Java and Python developer queries.

Load-bearing premise

The load-bearing premise is that GPT-4o's LLM-as-a-judge scores for helpfulness, correctness, and detail are faithful measures of answer quality; only 54 human-scored pairs validate them, agreement is moderate after binary mapping, and the same model family writes the pseudo-answers the judge is grading.

Editorial extensions

If this is right

  • Using HB1 with a high initial threshold (around 0.7) and lowering it only on miss delivers the best balance of answer quality and coverage, and full answers beat sentence-level context because they preserve rationale and edge cases.
  • On the 5,510 unseen questions, adaptive thresholding achieves 100 percent retrieval coverage and a statistically significant overall quality gain over Stack Overflow accepted answers, with the largest gains at high thresholds.
  • Across LLaMA-3.1-8B-Instruct, Granite-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3, HB1 significantly outperforms the same model's zero-shot answers; Qwen3-8B shows no significant difference, suggesting stronger models benefit less from retrieval.
  • The gains concentrate on implementation-oriented questions, where HB1 supplies best-practice code, contextual explanations, and edge-case handling that zero-shot responses often omit, while concept-focused questions sometimes do better without retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the 63-variant comparison gives a strong prior that query type and content granularity matter more than the exact similarity threshold, so teams building code-Q&A RAG systems could adopt the same HyDE-plus-full-answer recipe without re-running the full sweep.
  • Editorial extension: the reported judge agreement (moderate after binary mapping) implies the absolute score gaps, such as 6.11 versus 5.68 between HB1 and the best question-based pipeline, should be read as directional rather than precise; a larger manual validation set would tighten the calibration.
  • Editorial extension: a natural next experiment is to route questions through a lightweight classifier that skips retrieval for conceptual questions, since the paper's own sample finds zero-shot wins in about a quarter of cases, mostly concept-focused ones.
  • Editorial extension: because the corpus is limited to Java and Python Stack Overflow posts, the same adaptive HyDE pipeline could be tested on documentation, security advisories, or internal issue trackers to see whether full-answer granularity still beats sentence-level retrieval on longer, more structured sources.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper constructs a retrieval corpus of 3.4 million Java/Python Stack Overflow posts with accepted answers and evaluates seven RAG pipeline families (63 variants) for answering developer questions. It identifies a HyDE-based pipeline that retrieves full accepted answers using a GPT-4o-generated hypothetical answer as the best configuration, introduces an adaptive thresholding strategy that lowers the similarity threshold until context is retrieved, and reports that the resulting pipeline improves answer quality over zero-shot prompting across four open-source LLMs. All answer-quality claims are based on LLM-as-a-judge scores from GPT-4o.

Significance. If the claims were fully supported, the paper would provide a useful, reproducible study of RAG design choices for developer Q&A, including a large corpus, a systematic comparison of 63 variants, an adaptive retrieval mechanism for unseen questions, and an evaluation across four open LLMs. The replication package and the explicit experimental workflow are strengths. However, several load-bearing aspects of the evaluation need to be corrected or better supported before the central claims can be accepted.

major comments (5)
  1. [Section 2.5, Figure 3] The RQ3 comparison does not isolate the effect of retrieval because the RAG final-response prompt (Figure 3) contains strong style and quality directives ("accurate, concise, and highly relevant response", "clear and well-structured paragraph format, avoiding the use of bullet points or lists"), while the zero-shot prompt is never shown. Since the judge scores helpfulness, correctness, and detail, the more directive RAG prompt alone could mechanically raise all three criteria, even without retrieved context. The paper must show the exact zero-shot prompt and use an instruction-matched control (same template with and without retrieved context) to support the claim that retrieval, rather than prompt wording, drives the improvement.
  2. [Section 3, RQ2 Approach and Table 3] The significance tests in RQ2 use Mann-Whitney U to compare generated answers against accepted answers for the same questions. These observations are paired, so the independent-samples assumption of Mann-Whitney U is violated; a paired test such as Wilcoxon signed-rank, with an appropriate paired effect-size measure, should be used. The reported p-values and Cliff's delta values in Table 3 may therefore be invalid, and the statistical support for the RQ2 conclusions needs to be recomputed.
  3. [Section 3, RQ3 Approach] RQ3 states that it applies "our optimized RAG pipeline (HB1) with dynamic thresholding from 0.9 to 0.1", but RQ1 identified HB1 at a fixed threshold of 0.7 as the best configuration (Section 3, RQ1 Results). Dynamic thresholding down to 0.1 includes low-similarity context, which RQ2 itself shows degrades answer quality at thresholds 0.6 and 0.5. RQ3 therefore does not test the configuration selected in RQ1, and the term "optimal" is misleading. The authors should either use the RQ1-selected configuration or justify and explicitly rename the configuration tested in RQ3.
  4. [Section 3, RQ2 Results and Section 4.2] Table 3 shows that at thresholds 0.6 and 0.5, the generated answers are significantly worse than the corresponding accepted answers (Cliff's delta -0.29 and -0.44), covering about 23.5% of the unseen questions cumulatively. Despite this, the text concludes that adaptive thresholding "reliably improves or matches the quality of accepted answers" and the abstract claims robust enhancement. The aggregate improvement (p < 10^-82) is driven by the high-threshold subsets, so the conclusion should be qualified to reflect the degradation on the low-threshold subset.
  5. [Section 2.2 and Section 2.3.1] The validity of the LLM-as-a-judge is a load-bearing assumption for all three research questions, but the human validation is thin: only 54 answer pairs are manually scored, agreement is reported only after collapsing the 1-10 scores into two binary bins, and the reported Cohen's kappa is only moderate. In addition, the judge is GPT-4o, the same model family used to generate the HyDE pseudo-answers, raising the risk that judge preferences align with the style of HyDE-augmented outputs. The paper should provide a more robust validation (for example, per-criterion agreement on the original scale and an analysis of judge bias toward longer or retrieval-augmented outputs).
minor comments (5)
  1. [Table 2] The text in Section 3, RQ1 Results says HB1 has a mean score of 6.05, but Table 2 lists the HB1 mean as 6.11; please clarify which mean is reported.
  2. [Table 3] The "# Questions" column appears to list the number of questions newly retrieved at each threshold, while the "Coverage" column is cumulative; the table header and caption should state this explicitly to avoid confusion.
  3. [Figure 7] There is a typo: "alterbative" should be "alternative".
  4. [Section 4.1] The sentence "We randomly sample 20 questions (5 per model) from the KB-Synthetic Q-Set where HB1" is incomplete; it should finish with a description of what is sampled and compared.
  5. [Section 5] The threats-to-validity section does not mention the RQ3 prompt confound, the paired-data issue in RQ2, or the inconsistency between the RQ1 optimal threshold and the RQ3 configuration; these should be acknowledged and mitigated.

Circularity Check

1 steps flagged · score 4.0 of 10

Adaptive-thresholding 'full coverage' is definitional; the central RAG quality comparisons are empirical but share the benchmark used to select the pipeline.

  1. self definitional [Section 2.4 (Adaptive Thresholding Strategy) and RQ2 Results (Table 3)]
    "In this approach, if the initial retrieval does not yield any relevant content above the set similarity threshold, the system automatically relaxes the threshold in discrete steps (-0.1 at a time) until some content is retrieved or a minimum threshold is reached. ... While adaptive thresholding achieves full coverage by 0.5, it comes at the cost of reduced but reasonable answer quality."

    The algorithm is defined to keep lowering the similarity threshold until some content is retrieved, so 'full coverage' is the procedure's terminating condition rather than an empirical discovery. Reporting 100% coverage once the threshold reaches 0.5 is a restatement of that definition: for any non-empty corpus, coverage is guaranteed as long as the minimum threshold is low enough, and 'relevant context' is redefined as whatever content clears the lowered threshold. This is a self-definitional result. It does not force the paper's separate quality comparisons, which are empirical and could have gone the other way (e.g., Qwen3-8B shows no significant improvement).

full rationale

The only step that reduces to its own inputs by construction is the RQ2 'full coverage' claim: the adaptive-thresholding strategy is defined as lowering the threshold until content is retrieved, so reaching 100% coverage is guaranteed by the algorithm itself. This is a genuine but limited self-definitional result. The paper's central quality claims—HB1 outperforming zero-shot and accepted answers—are empirical comparisons, not definitional equivalences. They are weakened by methodological concerns that are not circularity: the same Synthetic Question Set is used to select HB1 in RQ1 and to evaluate it in RQ3, and GPT-4o both writes the HyDE pseudo-answers and acts as the judge. These threaten independence and generalizability, but the reported negative result for Qwen3-8B shows the RQ3 comparison is not forced. No load-bearing self-citation or imported uniqueness theorem was found. Overall, the derivation chain is mostly independent, with one self-definitional sub-result, so a score of 4 is appropriate.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

This is an empirical systems study rather than a derivation, so the ledger captures hidden inputs: accepted answers as ground truth, GPT-4o as an independent judge, cosine similarity as relevance, and the hand-set threshold floor. The threshold floor at 0.5 is the clearest ad hoc choice because it guarantees full coverage by construction.

free parameters (2)
  • Optimal similarity threshold = 0.7 (selected on Synthetic Question Set)
    Chosen as the best quality-coverage trade-off in RQ1 after inspecting scores on the same 385 questions later used to report RQ3 gains, so the selection is in-sample.
  • Adaptive threshold floor = 0.5
    Adaptive lowering stops at 0.5, the point where cumulative coverage reaches 100% on the Unseen Question Set. Full coverage is guaranteed by the stopping rule rather than by retrieval quality.
assumptions (5)
  • domain assumption Accepted Stack Overflow answers are a meaningful quality baseline and ground-truth source.
    Used in Section 2.1 to filter the knowledge base and in RQ2 as the comparison baseline. Accepted answers may be outdated, wrong, or not the best possible answer.
  • domain assumption GPT-4o scores (helpfulness, correctness, detail) reflect human judgment of answer quality.
    Section 2.2 validates on 54 pairs with moderate Cohen's kappa after binarizing scores. The validation is small, binary, and uses the same model family that generates HyDE pseudo-answers.
  • domain assumption Cosine similarity with all-mpnet-base-v2 is a valid relevance signal for Stack Overflow text.
    Section 2.3.2 uses cosine similarity as the retrieval criterion. No retrieval-quality metrics such as recall@k or nDCG are reported, so threshold-based coverage is assumed to mean relevance.
  • domain assumption Mann-Whitney U test assumptions hold for RQ2 comparisons.
    The test assumes independent samples, but generated answers and accepted answers are paired per question in RQ2, so the reported p-values may be invalid.
  • ad hoc to paper Lowering the similarity threshold until content is found yields useful context.
    Section 2.4 relaxes the threshold in 0.1 steps down to 0.5. The paper assumes whatever is retrieved above 0.5 is relevant enough to help generation, although RQ2 shows quality drops at 0.6 and 0.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Never Come Up Empty: Adaptive HyDE Retrieval for Improving LLM Developer Support." pith.science (2026). https://pith.science/paper/2P3J2HGF

@misc{pith2026250716754,
  author       = {Pith},
  title        = {Pith review of: Never Come Up Empty: Adaptive HyDE Retrieval for Improving LLM Developer Support},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2P3J2HGF}},
  note         = {Machine review of arXiv:2507.16754}
}
read the original abstract

Large Language Models (LLMs) have shown promise in assisting developers with code-related questions; however, LLMs carry the risk of generating unreliable answers. To address this, Retrieval-Augmented Generation (RAG) has been proposed to reduce the unreliability (i.e., hallucinations) of LLMs. However, designing effective pipelines remains challenging due to numerous design choices. In this paper, we construct a retrieval corpus of over 3 million Java and Python related Stack Overflow posts with accepted answers, and explore various RAG pipeline designs to answer developer questions, evaluating their effectiveness in generating accurate and reliable responses. More specifically, we (1) design and evaluate 7 different RAG pipelines and 63 pipeline variants to answer questions that have historically similar matches, and (2) address new questions without any close prior matches by automatically lowering the similarity threshold during retrieval, thereby increasing the chance of finding partially relevant context and improving coverage for unseen cases. We find that implementing a RAG pipeline combining hypothetical-documentation-embedding (HyDE) with the full-answer context performs best in retrieving and answering similarcontent for Stack Overflow questions. Finally, we apply our optimal RAG pipeline to 4 open-source LLMs and compare the results to their zero-shot performance. Our findings show that RAG with our optimal RAG pipeline consistently outperforms zero-shot baselines across models, achieving higher scores for helpfulness, correctness, and detail with LLM-as-a-judge. These findings demonstrate that our optimal RAG pipelines robustly enhance answer quality for a wide range of developer queries including both previously seen and novel questions across different LLMs

Figures

Figures reproduced from arXiv: 2507.16754 by the authors.

Figure 1
Figure 1. Experimental Workflow. RAG KB = Stack Overflow Knowledge Base (3.4 M accepted-answer documents). Synthetic Question Set: 385 questions auto-generated from the KB (seen). Unseen Question Set: 5,510 new Stack Overflow questions posted after the KB snapshot (unseen). relevance of retrieved content. The two RAG implementations are further characterized by three key design dimensions: the first di￾mension, retrieval targ… view at source ↗
Figure 2
Figure 2. An Overview of Question-Based and HyDE-Based RAG pipeline. HyDE specific design elements shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The prompt is used to generate the final response [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Percentage of retrieval coverage with retrievable [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Prompt used by the LLM-as-Judge to score generated [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Comparison of answer score distributions between [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 28 canonical work pages

  1. [1]

    Davit Abrahamyan and Foutse Khomh Fard. 2024. StackRAG Agent: Improving Developer Answers with Retrieval-Augmented Generation. In IEEE International Conference on Software Maintenance and Evolution (ICSME) . https://ieeexplore. ieee.org/document/10795043

    work page arXiv 2024

  2. [2]

    Meta AI. 2024. LLaMA 3.1–8B-Instruct. https://huggingface.co/meta-llama/Meta- Llama-3-8B-Instruct

    work page 2024

  3. [3]

    Mistral AI. 2024. Mistral-7B-Instruct-v0.3. https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.3

    work page 2024

  4. [4]

    Anonymous Authors. 2025. ICSE-C2-Stack-overflow: Retrieval-Augmented Gen- eration for Developer Questions. https://anonymous.4open.science/r/ICSE-C2- Stack-overflow-16E8/README.md. Accessed: July 2025

    work page 2025

  5. [5]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623

    work page 2021

  6. [6]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

    arXiv 2021

  7. [7]

    Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. arXiv preprint arXiv:2106.09231 (2021)

    work page Pith review arXiv 2021

  8. [8]

    Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, et al. 2025. A survey on knowledge-oriented retrieval-augmented generation. arXiv preprint arXiv:2503.10677 (2025)

    arXiv 2025

Show all 46 references
  1. [9]

    Vineesh Cutting and Nehemiah Stephen. 2021. Comparative review of java and python. International Journal of Research and Development in Applied Science and Engineering (IJRDASE) 21, 1 (2021)

    2021

  2. [10]

    Dilrukshi Gamage, Piyush Ghasiya, Vamshi Bonagiri, Mark E Whiting, and Kazutoshi Sasahara. 2022. Are deepfakes concerning? analyzing conversations of deepfakes on reddit and exploring societal implications. In Proceedings of the 2022 CHI conference on human factors in computin...

    2022

  3. [11]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero- shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 1762–1777

    2023

  4. [12]

    Ziyu Gao, Xin Xia, David Lo, John Grundy, and Tian Zhang. 2023. I Know What You Are Searching For: Code Snippet Recommendation from Stack Overflow Posts. ACM Transactions on Software Engineering and Methodology (TOSEM) (2023). https://dl.acm.org/doi/10.1145/3550150

    2023 doi

  5. [13]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith

  6. [14]

    Aidan Gilson, Xuguang Ai, Thilaka Arunachalam, Ziyou Chen, Ki Xiong Cheong, Amisha Dave, Cameron Duic, Mercy Kibe, Annette Kaminaka, Minali Prasad, et al. 2024. Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consume...

    2024 arXiv

  7. [15]

    Wael H Gomaa, Aly A Fahmy, et al. 2013. A survey of text similarity approaches. international journal of Computer Applications 68, 13 (2013), 13–18

    2013

  8. [16]

    Sai Muralidhar Jayanthi, Varsha Embar, and Karthik Raghunathan. 2021. Evalu- ating pretrained transformer models for entity linking in task-oriented dialog. arXiv preprint arXiv:2112.08327 (2021)

    2021 arXiv

  9. [17]

    Thomas Junk. 1999. Confidence level computation for combining searches with small statistics. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 434, 2-3 (1999), 435–443

    1999

  10. [18]

    Kumar and N

    S. Kumar and N. Patel. 2025. CodeCapBench: Benchmarking LLMs for Capability- Oriented Software Engineering Tasks. In Proceedings of ICSE 2025

    2025

  11. [19]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kulkarni, Xiang Cheng, Angela Fan, Vishrav Chaudhary, and et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Pr...

    2020

  12. [20]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kulkarni, Xiang Cheng, Angela Fan, Vishrav Chaudhary, and et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Pr...

    2020

  13. [21]

    Siran Li, Linus Stenzel, Carsten Eickhoff, and Seyed Ali Bahrainian. 2025. Enhanc- ing Retrieval-Augmented Generation: A Study of Best Practices. arXiv preprint arXiv:2501.07391 (2025)

    2025 arXiv

  14. [22]

    Elijah Mansur, Johnson Chen, Muhammad Anas Raza, and Mohammad Wardat

  15. [23]

    Mirza, L

    P. Mirza, L. Weber, and F. Küch. 2025. Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategy. arXiv preprint arXiv:2505.22157 (2025)

    2025

  16. [24]

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. 2024. Granite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:...

    2024 arXiv

  17. [25]

    Hellendoorn

    Mononito Mukherjee and Veselin J. Hellendoorn. 2025. SOSecure: Safer Code Gen- eration with RAG and StackOverflow Discussions.arXiv preprint arXiv:2503.13654 (2025). https://arxiv.org/abs/2503.13654

    2025

  18. [26]

    Sarah Nadi and Christoph Treude. 2020. Essential sentences for navigating stack overflow answers. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 229–239

    2020

  19. [27]

    Md Rabiul Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, and Baishakhi Ray

  20. [28]

    Md Masudur Rahman, Jed Barson, Sydney Paul, Joshua Kayani, Federico Andrés Lois, Sebastián Fernandez Quezada, Christopher Parnin, Kathryn T Stolee, and Baishakhi Ray. 2018. Evaluating how developers use general-purpose web-search for code retrieval. In Proceedings of the 15th ...

    2018

  21. [29]

    Nikitha Rao, Chetan Bansal, Thomas Zimmermann, Ahmed Hassan Awadallah, and Nachiappan Nagappan. 2020. Analyzing web search behavior for software engineering tasks. In 2020 IEEE International Conference on Big Data (Big Data) . IEEE, 768–777

    2020

  22. [30]

    Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embed- dings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813 (2020)

    2020 arXiv

  23. [31]

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023. The programmer’s assistant: Conversational interaction with a large language model for software development. InProceedings of the 28th International Conference on Intelligent User Inte...

    2023

  24. [32]

    James Skripchuk, Neil Bennett, Jeffrey Zhang, Eric Li, and Thomas Price. 2023. Analysis of novices’ web-based help-seeking behavior while programming. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 945–951

    2023

  25. [33]

    Jinyan Su, Jin Peng Zhou, Zhengxin Zhang, Preslav Nakov, and Claire Cardie

  26. [34]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  27. [35]

    Bogdan Vasilescu, Vladimir Filkov, and Alexander Serebrenik. 2013. Stackover- flow and github: Associations between software development and crowdsourced knowledge. In 2013 International conference on social computing . IEEE, 188–195

    2013

  28. [36]

    Zizhao Wang, Akari Asai, Xinyi Yu, Frank F Xu, Yizhou Xie, and Graham Neubig

  29. [37]

    arXiv preprint arXiv:2412.16708 (2024)

    Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks. arXiv preprint arXiv:2412.16708 (2024)

    2024 arXiv

  30. [38]

    Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E Hassan, and Zhenchang Xing. 2017. What do developers search for on the web? Empirical Software Engineering 22 (2017), 3149–3185

    2017

  31. [39]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    2025 arXiv

  32. [40]

    Chen Yang, Baowen Xu, Ferdian Thung, Yuxin Shi, and Tiancheng Zhang. 2022. Answer summarization for technical queries: Benchmark and new approach. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2022). https://dl.acm.org/doi/1...

    2022

  33. [41]

    https://arxiv.org/abs/2406.14497

    Coderag-bench: Can retrieval augment code generation? arXiv preprint arXiv:2406.14497 (2024). https://arxiv.org/abs/2406.14497

    2024 arXiv

  34. [42]

    Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin 1, 6 (1945), 80–83

    1945

  35. [46]

    Tianyi Zhang, Ting Zhang, Yanjun Di, Minghui Chen, and Tao Zhang. 2022. SOSum: A Dataset of Stack Overflow Post Summaries. Proceedings of the 19th Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al. International Conference on Mining Software Repositories (M...

    2022

  36. [2020]

    arXiv preprint arXiv:2009.11462 (2020)

    Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 (2020)

    2020 arXiv

  37. [2021]

    arXiv preprint arXiv:2108.11601 (2021)

    Retrieval Augmented Code Generation and Summarization. arXiv preprint arXiv:2108.11601 (2021). https://arxiv.org/abs/2108.11601

    2021 arXiv

  38. [2024]

    In 2024 IEEE International Conference on Big Data (BigData)

    RAGFix: Enhancing LLM Code Repair Using RAG and Stack Overflow Posts. In 2024 IEEE International Conference on Big Data (BigData) . IEEE, 7491–7496

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.