Pith. sign in

REVIEW 10 cited by

CodeJudge: Evaluating Code Generation with Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.02184 v1 pith:KNP6RNYO submitted 2024-10-03 cs.LG cs.CLcs.SE

Weixi Tong , Tianyi Zhang This is my paper
classification cs.LGcs.CLcs.SE
keywords codecodejudgellmsevaluationgenerationdatasetsevaluatefour
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub https://github.com/VichyTong/CodeJudge.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging

    cs.SE 2026-07 conditional novelty 6.0 of 10

    SEDCoT combines LLM translation, symbolic-execution test generation, and delta-debugging repair to raise COBOL-to-C correctness by ≥12% over SOTA LLM baselines while preserving human-readable output.

  2. Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code

    cs.SE 2026-05 accept novelty 6.0 of 10

    A review of 114 studies creates taxonomies for code and data quality issues, formalizes 18 propagation mechanisms from training data defects to LLM-generated code defects, and synthesizes detection and mitigation techniques.

  3. Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis

    cs.SE 2025-07 conditional novelty 6.0 of 10

    In a benchmark of LLM-generated C code for graph analysis, Claude Sonnet 4 Extended produced the most correct, fastest, and most memory-efficient implementations, beating human baselines on triangle counting.

  4. FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A new benchmark adds 148 interactive front-end development tasks with automated sandbox tests, reporting a 90.54% agreement rate with human evaluation across four LLMs.

  5. Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

    cs.SE 2026-04 unverdicted novelty 5.0 of 10

    LLM judges for code tasks show high sensitivity to prompt biases that systematically favor certain options, changing accuracy and model rankings even when code is unchanged.

  6. Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection

    cs.CL 2025-08 conditional novelty 5.0 of 10

    ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.

  7. An Empirical Study on Logging Evolution On Stack Overflow: Trends, Topics, and Challenges

    cs.SE 2026-05 unverdicted novelty 4.0 of 10

    Empirical study of Stack Overflow logging posts identifies 11 topics where containerized environments show the highest difficulty via unanswered rates and resolution time.

  8. Learning Bug Context for PyTorch-to-JAX Translation with LLMs

    cs.LG 2025-10 conditional novelty 4.0 of 10

    T2J uses 163 human-annotated bug–fix pairs from 20 PyTorch-to-JAX translations to prompt gpt-4o-mini, improving its JAX output on several LLM-judged metrics while showing mixed results on CodeBLEU.

  9. Towards High Supervised Learning Utility Training Data Generation: Data Pruning and Column Reordering

    cs.LG 2025-07 reject novelty 4.0 of 10

    PRRO combines signal-based data pruning and column reordering to improve the supervised learning utility of synthetic tabular data, but its evaluation is undermined by data manipulation and an ill-defined correlation measure.

  10. Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy

    cs.SE 2025-07 conditional novelty 3.0 of 10

    A five-task case study shows DSPy prompt optimization can improve LLM accuracy on some tasks, notably contradiction detection (46.2% to 64.0%), but results vary and no code or data are released.

Pith tools