Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation

Hainan Zhang; Lingxiang Wang; Shiwen Zhang; Sijia Wen; Zhiming Zheng; Ziwei Wang

REVIEW 5 major objections 6 minor 21 references

Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A retrieval model that ranks competitive-programming problems by underlying solution logic—rather than surface wording—improves code generation on hard problems.

desk verdict The ranking claim is circular—Table 2 measures memorization of training augmentations, not solution-aware retrieval—so the paper's central evidence fails, though the framing and some execution-based results are worth a second look. read the letter →

arxiv 2509.01129 v1 pith:TJTCXWKD submitted 2025-09-01 cs.IR

Shiwen Zhang , Lingxiang Wang , Hainan Zhang , Ziwei Wang , Sijia Wen , Zhiming Zheng This is my paper

classification cs.IR

keywords solution-awareretrievalcompetitiveprogrammingcodegenerationretrieval-augmentedcontrastivelearningsyntheticdatalogicalequivalencexEval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SolveRank is a retrieval model for competitive programming that ranks problems by whether the reference solution logic matches the query, instead of by surface wording. The paper argues that retrieval-augmented generation helps LLMs solve hard problems precisely when the retrieved example shares the solution's algorithm, and that semantic retrieval can actively mislead because similar-sounding problems often require different algorithms. To train such a retriever, the paper generates synthetic variants of anchor problems using DeepSeek-R1, filters them with GPT-4o for logical equivalence, and trains a DPR-style encoder with contrastive loss. On the xCodeEval benchmark, SolveRank reports a retrieval P@1 of 0.682 and MRR of 0.755, and raises Pass@1 on the hardest problems from 13.89% to 16.67% with GPT-4o and from 5.56% to 11.11% with GPT-3.5. The paper also shows the benefit is concentrated on hard problems: for easy tasks, retrieval tends to hurt.

What carries the argument

The load-bearing mechanism is contrastive training over 'logically equivalent' synthetic pairs. DeepSeek-R1 generates five differently phrased versions of each anchor problem; GPT-4o then filters them by judging whether the algorithmic model, optimization objective, and typical solution approach match the anchor while ignoring the story. These accepted variants are positives; BM25 top hits and random problems are negatives; and a DPR bi-encoder is trained with InfoNCE loss to pull solution-equivalent pairs together. At inference, the top-K problem-code pairs are prepended to the query as few-shot examples for a zero-shot code generator.

What would settle it

Take a random sample of pairs GPT-4o judged 'logically equivalent' and test whether the anchor's reference code actually solves the generated variant's test cases, or have expert competitive programmers judge the pairs. If agreement is near chance, the reported P@1/MRR gains would not survive an execution-based re-evaluation; alternatively, re-running Table 2 with only execution-verified positives as ground truth would show whether the ranking advantage persists.

Watch

Extended reading notes

Core claim

The core claim is that solution-level logical equivalence is the right relevance signal for retrieval-augmented competitive code generation, and that a retriever trained on synthetic logically-equivalent paraphrases can capture it. Concretely, the paper's ranking experiments on xCodeEval-python show that SolveRank retrieves solution-relevant problems with P@1=0.682 and MRR=0.755, far above BM25 (MRR 0.186), CodeBERT (0.147), DPR (0.057), and ReACC (0.057). In downstream generation, the benefit appears only for difficult problems (D>2000): with GPT-4o, Pass@1 rises from 13.89% with no retrieval to 16.67% with SolveRank, while semantic retrievers give at most 13.89%; with GPT-3.5 the rise is 5

Load-bearing premise

The whole pipeline stands on GPT-4o's yes/no judgment that a generated variant is logically equivalent to its anchor; if that judgment is noisy, the training positives and the ranking evaluation both inherit that noise without an execution-based check.

Editorial extensions

If this is right

Hard competitive-programming problems benefit from retrieved examples only when the examples are solution-equivalent; surface-similar examples can be worse than none.
Retrieval for code generation can be framed as a solution-level ranking task, with synthetic data replacing scarce human labels.
On the hardest problems, SolveRank's gain is consistent across two generators (GPT-3.5 and GPT-4o), though larger for the stronger generator.
On easy problems, even good retrieval reduces pass rates, so solution-aware retrieval should be gated by difficulty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If GPT-4o's equivalence verifications are unreliable, the ranking numbers and training signal have an unknown ceiling; an execution-based audit of a sample of accepted pairs would settle how much of the reported gain is real.
The APPS results hint that solution-aware retrieval mainly helps when problem statements carry narrative distractors; on stripped-down statements, BM25 ties it.
A natural extension is to use the retriever's embeddings to map problems by algorithm family, or to train the generator to ignore retrieved context when the retriever's confidence is low.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the core ranking result in Table 2 is a closed loop, so the paper's headline claim is not supported by the evidence. That said, the paper is useful for the problem it frames and the recipe it puts together.

The actual novelty is the solution-level ranking task for competitive programming and the synthetic-data training pipeline: DeepSeek-R1 to paraphrase anchors, GPT-4o to filter for logical equivalence, and contrastive DPR to build a retriever. That combination is new, and the writing is clear. The code-generation experiments use execution-based pass@1 on xCodeEval, which is the right measure for the downstream claim, and the case study in Figure 4 nicely illustrates how surface-term matching can mislead.

The soft spots are serious. The ranking evaluation (Table 2) uses 'synthetic solution-relevant problems from the xCodeEval-Python training dataset' as queries and ground truth. Those are the same anchor–variant pairs SolveRank was trained on, and the label comes from the same GPT-4o verifier that filtered the training data. No held-out split by anchor ID is reported. So the 0.682 P@1 and 0.755 MRR likely measure memorization of augmentation pairs, not solution-aware retrieval on new problems. To make the claim, they'd need execution-based equivalence checks (does the anchor's reference code actually pass the generated variant?) and a held-out evaluation set.

The downstream results are more external, but they are thin. The test set is 342 filtered problems with no stated selection criteria beyond ExecEval support; hard-problem cells are tiny fractions (e.g., D>2000 with GPT-4o: 16.67% vs 13.89% for no retrieval—a difference of a handful of problems). There are no error bars or significance tests. The APPS result in Table 5 ties with BM25, which actually weakens the claim of a solution-aware advantage. And Section 2.3 says retrieved candidates are GPT-4o-verified at inference; the paper doesn't say whether baselines get the same post-filter, so the comparison may be unfair.

Bottom line: the idea is reasonable and the paper may be worth a salvage, but the central evidence is circular and the external evidence is too noisy. I wouldn't send this to a referee as it stands. If the authors fix the evaluation—execution-based ground truth, held-out anchors, significance tests, and a uniform baseline protocol—the paper would become a solid candidate. For now, it's a 'maybe' for a reading group, but not a citation.

Referee Report

5 major / 6 minor

Summary. The paper proposes SolveRank, a bi-encoder retrieval model for competitive programming that aims to retrieve problems sharing the same algorithmic solution rather than surface-level wording. Positive training pairs are synthetic paraphrases generated by DeepSeek-R1 and filtered by GPT-4o; negatives come from BM25/random sampling; training uses the InfoNCE loss (Eq. 1). At inference, top-K retrieved problem–code pairs are prepended to the prompt of a downstream code generator. The paper reports large ranking gains on xCodeEval (SolveRank P@1 0.682, MRR 0.755 vs. BM25 MRR 0.186) and improved Pass@1 on hard problems (e.g., GPT-4o: 16.67% vs. 13.89% without retrieval).

Significance. The idea of solution-aware retrieval is timely and potentially useful for competition-level code generation, and the synthetic-data construction is a reasonable first step. The authors also provide code and release two supporting datasets, which is a practical contribution. However, the experimental evaluation as presented does not establish the central claims: the ranking benchmark is built from the same training anchors and the same GPT-4o verifier used for training, and the downstream gains are based on very small samples without significance testing. If these evaluation issues were fixed, the paper could make a contribution; as written, the evidence is not convincing.

major comments (5)

[§3.2, Table 2] The ranking evaluation is circular with respect to training. The caption states that the targets are 'synthetic solution-relevant problems from the xCodeEval-Python training dataset'—the same anchors used to construct positives and train SolveRank with the InfoNCE loss in Eq. (1). No held-out split by anchor ID is reported. A model trained to maximize similarity to each anchor's five generated variants can score highly on exactly those pairs by memorizing the augmentation, not by generalizing solution-aware retrieval. Please report results on anchors disjoint from training, or on an external human-annotated test set; otherwise the P@1/MRR numbers measure memorization.
[§2.2, Appendix A.2] GPT-4o serves both as the filter for training positives and as the ground-truth label for the ranking evaluation in Table 2. There is no execution check that the anchor's accepted reference code actually solves the generated variant, no human validation, and no inter-annotator agreement statistics. If GPT-4o's 'Yes/No' judgments are permissive or biased, the positive labels are systematically wrong, and the ranking metrics reflect agreement with GPT-4o rather than true solution equivalence. Independent validation—e.g., running the anchor's solution on the generated problem's tests, or human-annotated samples—is required.
[§3.2, Table 1] The hard-problem claim is not statistically supported. The paper does not report the number of test problems per difficulty band after filtering to 342. With small buckets, the difference between 16.67% and 13.89% for GPT-4o is likely a handful of problems; no confidence intervals, error bars, or significance tests are provided. The 'about 20% pass@1 increment' is a relative percentage and should be accompanied by absolute counts and uncertainty estimates.
[Appendix D, Table 5] On APPS, SolveRank ties with BM25 at 26.1% Pass@1, which directly weakens the central claim that solution-aware retrieval outperforms semantic retrieval. The explanation that APPS problems have shorter narratives and weaker background is plausible, but it is a post-hoc boundary condition. It should be presented in the main paper, and the abstract/introduction claims should be tempered accordingly.
[§3.1] The filtering of the xCodeEval NL-Code Retrieval test set to '342 suitable problems' is not described; no inclusion/exclusion criteria are given. This selection could bias the difficulty distribution and the comparison across methods. Please specify the filtering criteria and report the per-difficulty counts of the filtered test set.

minor comments (6)

[Figure 3] Typo in the figure: 'Prroblem' should be 'Problem'.
[§2.2] 'See in Appendix 6' should refer to the correct appendix label (likely Appendix E).
[Abstract / §3.1] The abstract says 'Experiments on the xCodeEval dataset' but the experiments are on the Python subset; please specify.
[Limitations] The Limitations section says experiments are 'solely on the xCodeEval benchmark', but Appendix D evaluates APPS. This inconsistency should be reconciled.
[Figure 2] The overall 'D > 1400' group is not defined in the figure itself; clarify in the caption.
[§2.3] Training hyperparameters (τ, number of epochs, batch size) are reported, but there is no sensitivity analysis for τ or the number of synthetic positives/negatives. A brief ablation or a statement that these were fixed would improve reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

Table 2's ranking evaluation is a closed loop: it scores SolveRank on the same synthetic anchor–variant pairs and GPT-4o labels used to train it, so the headline P@1/MRR gains are forced by construction; code-generation results are external but narrower.

fitted input called prediction [§3.2, Table 2 (ranking evaluation)]
"We evaluate the ranking performance by comparing SolveRank with SOTA ranking baselines, using the retrievability of synthetic solution-relevant problems from the xCodeEval-Python training dataset as the evaluation criterion."

SolveRank is trained with InfoNCE (Eq. 1) on exactly these synthetic variants as positives: §2.3 says 'The positive samples are drawn from the synthetic dataset Pq'. Table 2 then scores retrieval of 'synthetic solution-relevant problems from the xCodeEval-Python training dataset'. This is the same set of anchor–variant pairs, with no reported held-out split. Therefore P@1=0.682/MRR=0.755 measure how well the encoder reproduces its own training targets; the contrastive loss was optimized to bring exactly these q,q+ pairs together and push the 25 negatives apart. The large gap over BM25 is forced by construction, not evidence of learned solution-awareness.
self definitional [§2.2 and Appendix A.2 (GPT-4o verifier); Table 2 labels]
"To ensure true logical equivalence, we apply GPT-4o as an automatic verifier. For each generated variant q+ i ∈ Pq, GPT-4o is prompted to assess whether Logic(q+) ≡ Logic(q), focusing strictly on algorithm class and solution decomposition while ignoring superficial narrative or vocabulary differences."

The ranking 'ground truth' (P@1/MRR in Table 2) is defined by the same GPT-4o 'Yes/No' judgment that was used to filter the training positives. There is no execution check that the anchor's reference code solves the generated variant (Appendix A.2 asks only 'answer Yes or No'). Thus the retrieval metric measures agreement with GPT-4o's verifier, not an independent notion of solution equivalence; the model is trained to imitate that verifier's labels and then evaluated against labels from the same verifier, so high scores are partly a closed loop.

full rationale

SolveRank's retrieval head is trained by InfoNCE on synthetic anchor–variant pairs generated by DeepSeek-R1 and filtered by GPT-4o. Table 2 then reports P@1/MRR using exactly those 'synthetic solution-relevant problems from the xCodeEval-Python training dataset' as the evaluation criterion, with no held-out split described. This makes the headline ranking gain largely a memorization check: the model is optimized to map each anchor to its five generated variants and to separate them from the same BM25/random negatives, so reproducing those pairs on the training corpus is forced by the training objective, not evidence of a general solution-aware ranking ability. The label 'logical equivalence' is likewise supplied by the same GPT-4o verifier used to construct and filter the positives, so the metric measures agreement with that verifier rather than an execution-verified ground truth. The code-generation experiments (Table 1, ExecEval) are independent, execution-based, and do show a hard-problem Pass@1 improvement, so the paper is not wholly circular; however, the central ranking claim is weakened to the point of being partly circular. Score 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the correctness of LLM-generated synthetic positives and the representativeness of an opaque filtered test set. The most consequential choice is the definition of ground truth via GPT-4o, which makes the ranking evaluation self-referential. Hyperparameters and filtering choices are supplementary but under-specified.

free parameters (5)

InfoNCE temperature tau = Not reported
Hyperparameter in the contrastive loss (Equation 1); chosen by hand, not fitted, but affects the embedding geometry.
Number of synthetic positives per anchor = 5
Section 2.2 fixes five generated variants per anchor; arbitrary choice.
Number and composition of negatives = 25 (5 BM25 + 20 random)
Section 2.3 fixes the negative set size and mixture; arbitrary choice.
Training hyperparameters = 10 epochs, batch size 4, lr 3e-5
Section 3.1 reports these values without sensitivity analysis.
Top-K retrieved examples = Not specified in text
Section 2.4 uses top-K but the exact K used in experiments is not stated.

assumptions (5)

ad hoc to paper DeepSeek-R1 generated variants preserve solution logic when GPT-4o answers 'Yes'.
Section 2.2 relies on LLM judgment without execution-based or human verification; this is the load-bearing premise for both training and ranking evaluation.
domain assumption Contrastive training on synthetic positives makes dot-product similarity align with logical equivalence.
Standard ML assumption that the learned embedding reflects the training label structure; here the labels are LLM-generated.
domain assumption Retrieved problem-code pairs improve downstream code generation.
The RAG benefit is taken from prior work; the paper does assume the retrieved code is useful and not misleading.
ad hoc to paper The 342-problem filtered test set is representative of the full xCodeEval test set.
Section 3.1 filters the test set without stating criteria, which could introduce selection bias affecting the hard-problem results.
domain assumption xCodeEval difficulty scores are reliable proxies for problem difficulty.
The paper bins problems by official difficulty score; this is taken from the benchmark as ground truth.

invented entities (1)

Synthetic logically equivalent problems as ground-truth labels
purpose: Used as positive training pairs for SolveRank and as the ground-truth relevance set for the ranking evaluation in Table 2.
The paper defines these problems as logically equivalent solely via GPT-4o verification (Section 2.2), with no execution-based check or human annotation. The ranking results therefore measure agreement with the LLM pipeline, not verified solution equivalence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation." pith.science (2026). https://pith.science/paper/TJTCXWKD

@misc{pith2026250901129,
  author       = {Pith},
  title        = {Pith review of: Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TJTCXWKD}},
  note         = {Machine review of arXiv:2509.01129}
}

read the original abstract

In competitive programming task, problem statements are often embedded within elaborate narrative backgrounds, requiring deep understanding of the underlying solutions to successfully complete the tasks. Current code generation models primarily focus on token-level semantic modeling, highly susceptible to distractions from irrelevant narrative statements. Inspired by RAG, retrieving reference code with similar solutions may help enhance model performance on difficult problems. However, existing retrieval models also emphasize surface-level semantic similarity, neglecting the deeper solution-level logical similarities that are critical in competitive programming. Therefore, designing ranking models capable of accurately identifying and retrieving problems and corresponding codes remains an urgent research problem in competitive code generation. In this paper, we propose SolveRank, a solution-aware ranking model empowered by synthetic data for competitive programming tasks. Specifically, we leverage the DeepSeek-R1 model to generate logically equivalent but differently phrased new problems, verified by GPT-4o for solution consistency. Then, we train SolveRank with these as positive samples and BM25/random-retrieved problems as negatives. During inference, SolveRank retrieves relevant problems and corresponding code from the corpus to assist a downstream code generator. Experiments on the xCodeEval dataset demonstrate that SolveRank outperforms SOTA ranking methods in precision and recall metrics, and boosts code generation performance for difficult problems.

Figures

Figures reproduced from arXiv: 2509.01129 by the authors.

**Figure 2.** Comparison results of RAG with different ranking methods and without RAG on xCodeEvalpython test set. The x-axis is the difficulty score of problems and the y-axis is the pass@1 rate. 2024) dataset demonstrate that SolveRank significantly outperforms existing retrieval models(about 406% MRR increment), and improves code generation performance(about 20% pass@1 increment) for difficult problems, validating the eff… view at source ↗

**Figure 3.** Training pipeline for Solution-Aware Retriever [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: An example of retrieval results for a logic-intensive query. BM25 retrieves a problem with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

[1]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021
[2]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and 1 others. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536--1547

work page 2020
[3]

Dan Hendrycks, Steven Basart, Mantas Kadavath, and 1 others. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938

arXiv 2021
[4]

Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436

arXiv 2019
[5]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769--6781

2020
[6]

Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

work page 2024
[7]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092--1097

2022
[8]

Yinlin Liu, Pengcheng Yin, and Graham Neubig. 2019. Conala: The code/natural language challenge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)

work page 2019

Show all 21 references

[9]

Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. https://arxiv.org/abs/2305.12295 Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning . In Proceedings of the AAAI Conference on Artificial Intelligence

2023 arXiv
[10]

Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

2009
[11]

Guilherme Moraes Rosa, Ruan Chaves Rodrigues, Roberto Lotufo, and Rodrigo Nogueira. 2021. Yes, bm25 is a strong baseline for legal case retrieval. In Proceedings of the COLIEE 2021 Workshop: Competition on Legal Information Extraction/Entailment (COLIEE 2021)

2021
[12]

Yao Wan, Yuxin Wang, Zhenyu Zhang, and Zhi Jin. 2022. Reacc: A retrieval-augmented code completion framework. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)

2022
[13]

Lingxiang Wang, Hainan Zhang, Qinnan Zhang, Ziwei Wang, Hongwei Zheng, Jin Dong, and Zhiming Zheng. 2025. Codebc: A more secure large language model for smart contract code generation in blockchain. arXiv preprint arXiv:2504.21043

2025 arXiv
[14]

Tianle Xia, Liang Ding, Guojia Wan, Yibing Zhan, Bo Du, and Dacheng Tao. 2024. https://arxiv.org/abs/2405.01649 Improving complex reasoning over knowledge graph with logic-aware curriculum tuning . In Proceedings of the 33rd International Joint Conference on Artificial Intelli...

2024 arXiv
[15]

Zhun Yang, Adam Ishay, and Joohyung Lee. 2023. https://arxiv.org/abs/2307.07696 Coupling large language models with logic programming for robust and general reasoning from text . In Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI)

2023 arXiv
[16]

Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L. Gaunt. 2021. Unixcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)

2021
[17]

Feng Zhang, Daya Guo, Duyu Tang, Nan Duan, Xiang Ren, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

2020
[18]

Hanlin Zhang, Jiani Huang, Ziyang Li, Mayur Naik, and Eric Xing. 2023. https://arxiv.org/abs/2305.03742 Improved logical reasoning of language models via differentiable symbolic programming . In Proceedings of the AAAI Conference on Artificial Intelligence

2023 arXiv
[19]

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, and 1 others. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on...

2023
[20]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[21]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021

[2] [2]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and 1 others. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536--1547

work page 2020

[3] [3]

Dan Hendrycks, Steven Basart, Mantas Kadavath, and 1 others. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938

arXiv 2021

[4] [4]

Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436

arXiv 2019

[5] [5]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769--6781

2020

[6] [6]

Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

work page 2024

[7] [7]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092--1097

2022

[8] [8]

Yinlin Liu, Pengcheng Yin, and Graham Neubig. 2019. Conala: The code/natural language challenge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)

work page 2019