Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories

T0 review · 5 major / 6 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On educational C code, pure semantic retrieval beats quality-aware modes overall, while quality metadata helps mainly when the query itself is about quality.

desk verdict Honest small-scale prototype comparison of semantic vs quality-aware code retrieval; useful local evidence, but the mode ranking and 15/15 router claim rest on a hand-crafted 15-query set. read the letter →

arxiv 2607.09161 v1 pith:RYAL3D7T submitted 2026-07-10 cs.SE

Marek Horváth , Emília Pietriková This is my paper · ORCID
classification cs.SE
keywords semanticcodesearchretrieval-augmentedgenerationvectorembeddingssourceanalysisqualityevaluationfunction-levelfragmentationquality-awareretrievaleducationalCcorpus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Keyword search fails when people ask source-code repositories about intent or quality rather than exact tokens. This paper builds and tests a prototype that splits C files into function-level fragments, embeds them, attaches LLM-scored quality metadata, and compares four retrieval modes: semantic, quality-filtered, hybrid, and automatic routing. On a fixed 10% sample of an educational C corpus and 15 manually judged queries, semantic search was strongest overall (nDCG@5 0.820, Success@5 0.800, MRR 0.644), while filter and hybrid modes led only on quality-oriented queries. The router chose the expected mode for every query but added about tenfold latency, and a small audit found LLM quality scores usually within one point of human judgment except on trivial functions. A sympathetic reader cares because the result tells when to trust plain semantic search and when explicit quality scores actually change who shows up in the top results.

What carries the argument

Four retrieval modes over function-level fragments in a vector store with LLM-derived eight-dimension quality metadata: pure semantic cosine ranking, quality-filtered re-ranking of a semantic pool, hybrid scoring that blends quality and similarity (α = 0.5), and an LLM router that picks among those modes from the query text.

What would settle it

Re-run the same four modes on the full 563-identifier corpus with 30–50 independently written queries, multiple annotators measuring agreement, and blinded expected-mode labels; if semantic no longer leads aggregate nDCG@5 or the router is far below 15/15, the central ranking claim fails.

Watch

Extended reading notes

Core claim

Within the reported 15-query evaluation on a reproducible 10% educational C sample, semantic retrieval was the strongest overall mode, with nDCG@5 of 0.820, Success@5 of 0.800, and MRR of 0.644. Explicit quality metadata was most useful for explicitly quality-oriented queries, where hybrid and filter modes reached nDCG@5 of 0.959 and 0.947. The automatic router selected the expected mode for all 15 queries, and LLM-derived quality scores matched manual assessment within one point for 9 of 12 audited fragments.

Load-bearing premise

The mode ranking and perfect router score rest on only fifteen author-built queries, single-annotator pooled judgments, and evaluator-defined expected modes on a ten-percent sample.

Editorial extensions

If this is right

  • Pure semantic search is the better default for mixed natural-language queries over this kind of educational repository.
  • Quality metadata should be applied when users explicitly ask about cleanliness, edge cases, readability, or similar—not as a blanket re-ranker.
  • An automatic router can map short, well-typed queries to the right mode, at the cost of an extra model call and roughly 1.9 s average latency versus under 0.2 s for direct modes.
  • Quality-aware modes return substantially different programmer identifiers than semantic search (low Jaccard overlap), so they change who appears in the top ranks, not only their order.
  • The prototype is framed as an exploratory inspection tool with cited excerpts, not as automated grading or assessment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Local course and assignment archives may get more day-to-day value from intent-level semantic search than from quality re-ranking unless the workflow is specifically quality filtering.
  • Encoding non-applicable quality dimensions as a neutral score of 5 systematically overrates trivial or empty functions and would likely improve filter and hybrid reliability if “not applicable” were stored separately.
  • On larger, multi-topic, or industrial multi-language corpora the mode ranking could reverse if quality variance and topical diversity grow, so the educational-C result should be treated as setting-specific until retested.
  • Because the paper evaluates retrieval and not answer faithfulness, it remains open whether citation-grounded generation actually cuts unsupported claims on these fragments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper evaluates a prototype RAG-style retrieval system for educational C-code repositories that indexes function-level fragments with text-and-code embeddings (text-embedding-3-small) and LLM-derived quality metadata (eight dimensions) in ChromaDB, then compares four modes: pure semantic, quality-filtered, hybrid (α=0.5), and LLM-routed automatic selection. On a reproducible 10% sample (56 programmer IDs, 847 files, 3,839 fragments) and 15 author-written English queries with single-annotator 0–3 relevance judgments over pooled returned identifiers, semantic mode is strongest in aggregate (nDCG@5 0.820, Success@5 0.800, MRR 0.644), while filter/hybrid lead only on the five quality-oriented queries; the router matches author-expected modes on all 15 queries at ~1.9 s mean latency; a 12-fragment audit finds LLM quality scores within ±1 of manual scores in 9/12 cases. Claims are explicitly scoped to this corpus and query set, with a substantial threats-to-validity discussion.

Significance. If the comparative ranking holds under stronger evaluation, the work would be a useful, practical contribution to educational and repository-scale code search: it cleanly separates semantic similarity from explicit quality metadata, documents a reproducible function-level indexing pipeline, and shows when quality re-ranking helps versus when it distorts topical relevance. Strengths include careful claim scoping, transparent reporting of an out-of-coverage linked-list failure, concrete failure modes of the LLM quality judge on trivial/empty functions, and an evidence-first design (identifier grouping, citations, optional verify pass). The significance is currently limited by evaluation scale rather than by an unsound system design; the prototype is a credible exploratory tool, not yet a settled ranking of retrieval modes for the broader community.

major comments (5)
  1. §5.1 and Table 4: Mode ranking and the 15/15 router result rest on 15 author-constructed queries pre-labeled by expected mode (5 semantic / 5 filter / 5 hybrid). Router “accuracy” is defined as matching those same labels (§6.3). With no held-out or independently authored queries, the perfect router score and the category-wise pattern are not independent evidence of routing skill. At minimum, expand to ≥30–50 queries (as the paper itself suggests in §7.4), include queries written without mode labels in mind, and report router performance against labels from a second person or against an oracle based on which mode actually maximizes nDCG for that query.
  2. §5.1 relevance protocol: Judgments are single-annotator, pooled only over identifiers returned by the four modes, with no inter-annotator agreement. This is load-bearing for Tables 6–7 and RQ1–RQ2. Unreturned but relevant identifiers are never scored, so modes that miss relevant material can look better than they are. Please add at least a second annotator on the full pooled set (or a random sample of non-returned IDs), report agreement, and state how disagreements were resolved.
  3. §6.2 Tables 6–7 and RQ1: Aggregate nDCG@5 order (semantic 0.820 > filter 0.668 > hybrid 0.658 > auto 0.640) is reported without confidence intervals, bootstrap, or paired significance tests over n=15. Category cells have only five queries each. The central claim that semantic is “strongest overall” and quality modes help only quality queries is therefore under-powered. Either enlarge the query set and test differences, or reframe the contribution as a systems/prototype description with illustrative metrics rather than a comparative ranking.
  4. Table 4 quality/hybrid wording vs. Table 3 dimensions: Queries such as “Who writes the cleanest code?”, “best error handling”, “Robust file reading with proper error handling”, and “Well-documented sorting” closely echo the stored metadata field names (clean_code, error_handling, documentation, etc.) that the router must also select as tool arguments. This creates a mild circularity risk for both category wins of filter/hybrid and perfect routing. Discuss this overlap explicitly and include quality-intent queries whose wording does not mirror the eight dimension labels.
  5. §6.4 / RQ4: The quality-score audit (12 fragments, 9/12 within ±1) is too small to support reliability claims about the metadata that filter and hybrid depend on. The paper already notes systematic over-rating of trivial/empty functions; that failure mode can reorder quality-aware results. Either enlarge the audit with multi-rater agreement and applicability labels, or demote RQ4 to a diagnostic note and avoid implying that the metadata are validated for ranking use.
minor comments (6)
  1. §4.5 Eq. (1): The distance-to-similarity map S = max(0, min(1, 1−d/2))·10 assumes cosine distance in [0,2]; state the Chroma/HNSW distance convention explicitly so readers can reproduce scores.
  2. §4.5 hybrid formula: α=0.5 is fixed with no sensitivity analysis; a short α sweep (or leave-one-dimension-out) would strengthen interpretation of Table 7 hybrid-category results.
  3. Figure 2: Router correctness bars (5/5 per category) restate the 15/15 claim; consider adding latency distributions (e.g., box plots) rather than only means.
  4. §3.2 / Table 1: Clarify whether the 10% sample was stratified by assignment family or only by programmer ID; assignment coverage affects the linked-list out-of-domain case.
  5. References and related work: CodeSearchNet, CodeBERT, GraphCodeBERT, and CodeRAG-Bench are cited appropriately; a brief comparison to classical IR baselines (BM25 over function text) would help readers gauge the absolute value of the semantic baseline.
  6. Typographical consistency: occasional missing spaces after commas in compound phrases (e.g., “text-and-code embeddings,ChromaDB”) and mixed hyphenation of “quality-aware” / “quality aware” should be cleaned in copy-editing.

Circularity Check

2 steps flagged · score 2.0 of 10

Mostly non-circular empirical IR evaluation; mild circularity only in router “accuracy” against author-constructed expected-mode labels on the same 15 hand-written queries.

  1. other [§5.1 Table 4; §6.3 router accuracy]
    "The automatic router classified all 15 test queries into the expected mode, giving 15/15 correct decisions. [...] The router's 15/15 expected-mode match must be interpreted with caution because the query set is small and the expected mode labels were constructed by the evaluator."

    Router “accuracy” is defined as matching expected-mode labels that the same evaluators assigned when they wrote and pre-grouped the 15 queries (5 semantic / 5 filter / 5 hybrid in Table 4). Success is therefore agreement with author intent on author-designed, mode-aligned prompts, not an independent held-out classification task. This is mild evaluation circularity, not a forced derivation of the main nDCG ranking.

  2. other [§4.5 quality dimensions; §5.1 Table 4 quality/hybrid queries; §6.2 Table 7]
    "filter: Who writes the cleanest code?; Which students best handle edge cases?; Find code with the best error handling; Who has the worst input validation?; Which students write the most readable code?. hybrid: Clean implementation of dynamic memory allocation; Robust file reading with proper error handling; [...]"

    Quality and hybrid test queries are worded with the same labels as the eight LLM quality-metadata dimensions (clean_code, edge_case_handling, error_handling, input_validation, readability, memory_management, etc.) that filter/hybrid modes use for re-ranking. Category-level superiority of quality-aware modes on those queries (Table 7) is therefore partly built into query design, though human relevance judgments still provide an independent score and do not force the overall semantic-mode win.

full rationale

This is an empirical retrieval evaluation, not a first-principles derivation. The load-bearing comparative claim—semantic mode strongest overall (nDCG@5 0.820, Success@5 0.800, MRR 0.644), quality-aware modes strongest only on the quality category—rests on human 0–3 relevance judgments of pooled programmer identifiers and standard ranking metrics. Embeddings (text-embedding-3-small), ChromaDB cosine HNSW retrieval, and LLM quality metadata are external model outputs, not quantities defined in terms of the reported metrics. Hybrid scoring uses a fixed α=0.5, not a fit then re-predicted on related data. Self-citations ([6],[7],[8],[15]–[17]) are background on education, clone detection, and attribution and do not underwrite uniqueness or force the mode ranking. The only mild circularity is evaluation design for the router: Table 4 and §5.1 pre-partition the 15 author-written queries into expected modes, and §6.3 reports 15/15 “correct” decisions as matching those same evaluator labels—the paper itself flags this caution. Quality-query wording also tracks the eight rubric dimension names, so category wins for filter/hybrid are partly by experimental construction, but relevance remains independently judged and the aggregate semantic win is not forced. No self-definitional equation, fitted-input-as-prediction, uniqueness import, or ansatz smuggling is present. Score 2 reflects one non-load-bearing evaluation circularity; central retrieval ranking has independent content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central comparative claim rests on standard IR metrics and engineering choices rather than deep theory. Load-bearing free parameters are the hybrid weight, retrieval pool sizes, and the fixed eight quality dimensions; domain assumptions include function-level units, LLM-as-judge scores as usable metadata, and author-defined query/mode categories. No new physical or mathematical entities are postulated.

free parameters (5)
  • hybrid_alpha = 0.5
    Fixed at 0.5 in Eq. (3) without tuning; directly controls quality vs similarity trade-off in hybrid mode results.
  • semantic_pool_sizes = filter ~50–100; hybrid 120 (cap 200)
    Filter uses min(100, max(50, 3n)); hybrid defaults to 120 capped at 200. These hand-chosen pool sizes affect which fragments quality re-ranking can promote.
  • n_results_and_top_candidates = 30 / 5
    Evaluation fixed n_results=30 and top_candidates=5; metrics are computed on this cutoff.
  • quality_dimension_set = 8 dimensions; N/A=5
    Eight hand-specified dimensions and neutral score 5 for N/A; defines all quality metadata used by filter/hybrid modes.
  • embedding_and_judge_models = text-embedding-3-small (+ role-configured LLMs)
    text-embedding-3-small and configured OpenAI judge/router models are fixed choices that determine all vectors and quality labels.
assumptions (5)
  • domain assumption Function-level (and occasional struct/file) fragments are an appropriate indexing unit for semantic and quality-aware retrieval over educational C code.
    Section 4.3; fragmentation defines every embedding, score, and judgment unit.
  • domain assumption LLM-as-a-judge integer scores on eight rubric dimensions are usable ranking metadata even though they are not calibrated ground truth.
    Sections 4.4 and 6.4; filter/hybrid modes depend on these scores.
  • standard math Standard ranking metrics (nDCG@5, Success@k, MRR) on programmer-identifier-level 0–3 relevance judgments measure retrieval effectiveness for the stated RQs.
    Section 5; used for all mode comparisons.
  • ad hoc to paper Author-constructed expected mode labels for the 15 queries are the correct targets for router accuracy.
    Section 6.3; 15/15 router claim depends on these labels.
  • domain assumption A reproducible 10% identifier-level sample with seed 42 is adequate for within-sample mode comparison (not full-corpus generalization).
    Section 3.2 and threats; all reported metrics use this sample.
invented entities (1)
  • Four-mode quality-aware code retrieval prototype (semantic / filter / hybrid / auto router with eight quality metadata fields)
    purpose: System under evaluation that attaches LLM quality profiles to fragments and routes or re-ranks queries.
    Engineering assembly of known components rather than a new scientific object; no independent existence outside the prototype.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories." pith.science (2026). https://pith.science/paper/RYAL3D7T

@misc{pith2026260709161,
  author       = {Pith},
  title        = {Pith review of: Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYAL3D7T}},
  note         = {Machine review of arXiv:2607.09161}
}
read the original abstract

Keyword-based retrieval is limited for source-code repositories when queries are expressed in natural language or concern implementation intent and code quality rather than exact tokens. This study evaluates a prototype retrieval system that combines function-level fragmentation, text-and-code embeddings, ChromaDB vector storage, LLM-derived quality metadata, and four retrieval modes: semantic, quality-filtered, hybrid, and automatic routing. The concrete evaluation uses an educational C-code corpus. The full corpus contains 563 anonymized programmer identifiers and 8,951 C files; a reproducible 10% indexed sample contains 56 programmer identifiers, 847 files, and 3,839 fragments. Across 15 manually judged queries, semantic retrieval achieved nDCG@5 of 0.820, Success@5 of 0.800, and MRR of 0.644. The automatic router selected the expected mode for all 15 queries. In a small manual audit, LLM-derived quality scores were within one point of the manual assessment for 9 of 12 fragments. Within the reported query set, semantic retrieval was the strongest overall mode, while explicit quality metadata was most useful for explicitly quality-oriented queries.

Figures

Figures reproduced from arXiv: 2607.09161 by the authors.

Figure 1
Figure 1. Retrieval workflow used in the evaluation. C source files are parsed into fragments, scored, embedded, and stored with meta￾data in ChromaDB. At query time, a retrieval mode returns fragments that are grouped by programmer identifier; generated answers are optional and are not separately evaluated in this study. Tab. 2: Main methodological stages used in the retrieval evaluation. Stage Operationalization Purpose in … view at source ↗
Figure 2
Figure 2. Latency and router behavior. Direct retrieval modes were below 0.2 s on average. Auto mode includes an additional router call. Router correctness is based on 15 predefined queries, five per category. Tab. 9: Representative manual quality-score audit cases. Dimension abbreviations: eh = edge-case handling, iv = input validation, rd = readability, ef = efficiency, cc = clean code. Fragment Dim. LLM Manual assessment r… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A fine-tuned CodeBERT classifier reaches 70.7% Top-1 on 1,000 Code Jam authors but 0.2% or less on real coursework submissions, showing that competition benchmarks do not transfer to classrooms.

Reference graph

Works this paper leans on

20 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Clone detection techniques and tools in software engineer- ing

    Lenka Bubenkova, Emilia Pietrikova, and Marcel Volosin. Clone detection techniques and tools in software engineer- ing. In2024 IEEE 17th International Scientific Conference on Informatics (Informatics), pages 39–44. IEEE, 2024

    2024

  2. [2]

    Chroma documentation, 2024

    Chroma. Chroma documentation, 2024. URL https://docs. trychroma.com. Accessed 2026-01-12

    2024

  3. [3]

    CodeBERT: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547. Association for Computational Lin- guistics, 2020

    2020

  4. [4]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2024

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2024. URL https://arxiv.org/abs/2312.10997

    arXiv 2024

  5. [5]

    GraphCodeBERT: Pre- training code representations with data flow

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svy- atkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. GraphCodeBERT: Pre- training code representations with data flow. InInterna- tional Conference on Learning Representatio...

    2021

  6. [6]

    Per- sonalized learning analytics through static code analysis in computer science education.Acta Informatica Pragensia, 15 (1):54–71, 2026

    Marek Horváth, Emília Pietriková, and Filip Gurbál’. Per- sonalized learning analytics through static code analysis in computer science education.Acta Informatica Pragensia, 15 (1):54–71, 2026

    2026

  7. [7]

    Bridging behavioral biometrics and source code stylome- try: A survey of programmer attribution.arXiv preprint arXiv:2603.11150, 2026

    Marek Horvath, Emilia Pietrikova, and Diomidis Spinellis. Bridging behavioral biometrics and source code stylome- try: A survey of programmer attribution.arXiv preprint arXiv:2603.11150, 2026

    arXiv 2026

  8. [8]

    Benchmarking ai models for grading cs assignments across multiple domains.Acta Polytechnica Hungarica, 23(5), 2026

    Marek Horváth, Lukáš Tomaščík, Nikola Geciová, Norbert Ádám, and Emília Pietriková. Benchmarking ai models for grading cs assignments across multiple domains.Acta Polytechnica Hungarica, 23(5), 2026

    2026

Show all 20 references
  1. [9]

    A survey on hallucination in large language models: Principles, taxon- omy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxon- omy, challenges, and open questions.ACM Transactions on In...

    2025 arXiv

  2. [10]

    CodeSearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Alla- manis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    1909 arXiv

  3. [11]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küt- tler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neura...

  4. [12]

    Curran Associates, Inc., 2020

    2020

  5. [13]

    Malkov and Dmitry A

    Yury A. Malkov and Dmitry A. Yashunin. Efficient and ro- bust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020

    2020

  6. [14]

    Text and code embeddings by contrastive pre- training.arXiv preprint arXiv:2201.10005, 2022

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski S...

    2022 arXiv

  7. [15]

    Embeddings — OpenAI API documentation, 2024

    OpenAI. Embeddings — OpenAI API documentation, 2024. URL https://platform.openai.com/docs/guides/embeddings. Accessed 2026-02-03

    2024

  8. [16]

    Profile-driven source code exploration

    Emília Pietriková and Sergej Chodarev. Profile-driven source code exploration. In2015 Federated Conference on Com- puter Science and Information Systems (FedCSIS), pages 929–934. IEEE, 2015

    2015

  9. [17]

    Towards automated assessment in game-creative programming courses

    Emilia Pietrikova, Jan Juhar, and Jana Št’astná. Towards automated assessment in game-creative programming courses. In2015 13th International conference on emerg- ing eLearning technologies and applications (ICETA), pages 1–6. IEEE, 2015

    2015

  10. [18]

    Game- creative learning in programming courses over 15 years.Acta Polytechnica Hungarica, 22(1):123–141, 2025

    Emília Pietriková, Norbert Ádám, and Anton Baláž. Game- creative learning in programming courses over 15 years.Acta Polytechnica Hungarica, 22(1):123–141, 2025

    2025

  11. [19]

    Xu, Yiqing Xie, Graham Neubig, and Daniel Fried

    Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. CodeRAG-Bench: Can retrieval augment code generation? InFindings of the Association for Computational Linguistics: NAACL 2025, pages 3199–3214. Association for Computa- ...

    2025

  12. [20]

    ICE-Score: Instructing large language mod- els to evaluate code

    Terry Yue Zhuo. ICE-Score: Instructing large language mod- els to evaluate code. InFindings of the Association for Computational Linguistics: EACL 2024, pages 2232–2242. Association for Computational Linguistics, 2024

    2024

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.