Pith. sign in

REVIEW 4 major objections 4 minor 70 references

Engineering AI Judge Systems

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A search-driven constitution-based framework for building AI judges turns judging requirements into reusable principles, and in a commit-message case study improves judgment accuracy by up to 6.2% while cutting development effort through…

desk verdict A useful industrial framework paper with a confounded evaluation: the 6.2% accuracy claim rests on an unvalidated automatic-metric vote and a comparison that changes scoring format alongside content. read the letter →

arxiv 2411.17793 v1 pith:RXZYBB5G submitted 2024-11-26 cs.SE cs.AI

Jiahuei Lin (Justina) , Dayi Lin , Sky Zhang , Ahmed E. Hassan This is my paper · ORCID
classification cs.SEcs.AI
keywords AIjudgesystemsLLM-as-a-judgefoundationmodelsoftware(FMware)constitution-basedframeworkcommitmessagegenerationevaluationmetricsknowledge-drivenengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that building AI judge systems—automated evaluators for software powered by foundation models—is an engineering problem with its own life cycle, and that current practice is ad hoc and costly. It proposes a search-driven, constitution-based framework that turns abstract judging requirements into reusable, context-specific principles, then uses those principles to construct and evolve the judge. In a case study on commit message generation spanning five programming languages, the judge built with the framework agreed with the paper's ground-truth heuristic more often than a judge built without it, by up to 6.2%, while reusing an average of 58% of general principles. A sympathetic reading takes this as evidence that organizing judging requirements into explicit, reusable criteria improves both the productivity and the quality of AI judge development.

What carries the argument

The central object is the constitution: a set of generic judging principles derived from high-level requirements by an FM, refined through at least four rounds of critique and revision, then specialized into a contextualized constitution for a specific FMware context. Four stages carry the argument: (I) creating the general constitution, (II) specializing it, (III) searching for cognitive architectures (jury FMs, heuristics, metrics, and their interactions) using the contextualized constitution, and (IV) evolving the judge by tracing judgment flaws back to requirement bugs. The case study also relies on MetricVote (Eq. 1), a majority vote among BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT that serves as the ground-truth heuristic for which of two commit messages is better; the paper measures judge accuracy as agreement with this vote.

What would settle it

Take a random sample of the commit-message pairs from the case study, have human experts label which message is better, and compute whether the framework-built judge agrees with humans more often than the baseline judge does. If the framework's advantage over the baseline disappears under human labels, the central claim about judgment quality is not supported. A second check: run the framework judge and the baseline judge with the same scoring granularity (both per-principle or both holistic) to see whether the 6.2% gain persists.

Watch

Extended reading notes

Core claim

The central claim is that transforming judging requirements into a reusable 'constitution' of generic principles, then specializing those principles to a target context, yields AI judge systems that are both cheaper to build and more accurate. In the paper's commit-message case study, the framework's judge scores each principle separately and sums the scores, while a baseline judge assigns one holistic 0-to-n score; across C++, C#, Java, Python, and JavaScript, the framework's judge matches the MetricVote ground truth more often, with gains from 1.0% (Java) to 6.2% (C#). The authors attribute the improvement to fine-grained, principle-based evaluation and to the reuse of roughly 58% of general principles across languages, which reduces the manual work of defining judging criteria.

Load-bearing premise

The central claim rests on treating MetricVote—a majority vote among five automatic text metrics—as the correct answer for which commit message is better; if that proxy does not track what human reviewers judge to be good commit messages, the reported accuracy gain is agreement with metrics, not evidence of better judging.

Editorial extensions

If this is right

  • Judging requirements for similar FMware can be reused across teams and contexts, so AI judge development shifts from writing criteria from scratch to customizing a shared constitution.
  • Decomposing a judgment into per-principle scores (a fine-grained metric per criterion) can improve agreement with reference metrics compared with a single holistic score.
  • Because principles are traceable back to requirements, judgment errors can be debugged as requirement bugs rather than as isolated prompt or model issues.
  • New judging techniques can be added to the search space in Stage III, letting the framework automatically reconstruct a judging architecture without manual redesign.
  • If Stages III and IV work as designed, the framework would support continuous evolution of judges as foundation models and FMware change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy gain may partly come from the scoring granularity change (per-principle vs holistic) rather than from the constitution's content; a controlled comparison that keeps granularity fixed would isolate the contribution of reusable principles.
  • Because MetricVote is itself a majority of automatic metrics, the 6.2% improvement measures agreement with those metrics. A human-judgment validation on a sample of pairs would show whether the framework also improves agreement with people.
  • The 58% reuse figure comes from one commit-message domain; reuse rates could differ sharply for other FMware types, so the productivity claim is strongest for similar text-generation tasks.
  • The framework's search and evolution stages (III and IV) are not evaluated in the case study, so the 'search-driven' part of the framework is currently a proposal rather than a demonstrated result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports on industrial experience developing AI judge systems for Foundation Model-powered software (FMware). It catalogs nine challenges across defining judging requirements, developing judges, and evolving judges, and proposes a search-driven constitution-based framework with four stages: creation of a general constitution, specialization to contextualized constitutions, search for cognitive architectures, and evolution of the judge. The empirical evaluation is a case study on judging commit message generation, comparing an AI judge system built with the framework (per-principle scores summed) against one built without it (a single holistic score), using agreement with an automatic-metric majority vote as the accuracy measure. The paper claims up to 6.2% accuracy improvement and about 58% average reuse of general principles across five programming languages.

Significance. The challenge catalog and the idea of reusable, constitutional judging principles are potentially useful for practitioners building LLM-as-a-judge systems. The paper makes an explicit falsifiable claim that its framework improves judge accuracy and reduces development effort. However, the significance hinges entirely on the validity of the evaluation: the ground truth is an unvalidated majority vote of automatic metrics, the comparison is confounded with scoring granularity, and the reported accuracy values are below the standard chance level for binary comparisons. If these issues are addressed, the framework could be a meaningful contribution to the emerging practice of AI judge engineering; as presented, the empirical evidence does not establish the claimed benefits.

major comments (4)
  1. [Section IV-B, Eq. (1)] The ground-truth heuristic MetricVote is a majority vote over BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT, but the paper provides no evidence that this vote corresponds to human judgments of commit-message quality. This is especially problematic because Section II-C explicitly argues that automatic metrics 'primarily focus on surface-level similarities' and can miss contextual relevance, coherence, and creativity, and because BLEURT is trained for machine-translation evaluation rather than for commit messages. Without a human-validated ground truth, Eq. (3) measures only agreement with a particular automatic-metric vote, not judging accuracy.
  2. [Section IV-C, Table III] All reported accuracy values (37.0% to 46.1%) are below 50%, which is the expected agreement level for a two-alternative forced choice when ties are rare. The paper compares against a 'randomly assigning a class in a three-categorie classification (i.e., 33%)' baseline, but this is only appropriate if MetricVote and JudgeVote frequently return the tie value 0; the paper does not report the frequency of ties. If ties are rare, both judge systems are actually worse than random with respect to the chosen ground truth, which undermines the interpretation of these numbers as 'accuracy'. The authors should report the outcome distribution of Eqs. (1) and (2) and use the correct chance baseline.
  3. [Section IV-B] The comparison between the w/ and w/o systems is confounded. The w/ system scores each principle on a 0-1 scale and sums the scores, while the w/o system assigns a single holistic 0-to-n score. Thus the treatment differs in both the content (constitution principles) and the scoring format (per-principle aggregation vs. holistic). The paper's own result heading attributes the gain to 'fine-grained evaluation methods and scoring,' which suggests the 6.2% improvement cannot be attributed to the constitution framework per se. An ablation that holds the scoring format fixed (e.g., a holistic score using the same principles) is needed to isolate the framework's contribution.
  4. [Section IV-C, 'Productivity of developing AI judge systems'] The reuse percentages in Table II are computed over the authors' own LLM-generated general and contextualized principles and do not measure development effort. No baseline for creating principles from scratch, no time or cost data, and no controlled comparison are provided; the single internal user quote is anecdotal. The abstract's claim of 'significant reduction in development effort' is therefore not supported by the reported evidence.
minor comments (4)
  1. [Section II-C, first paragraph] 'Claude 3.5 Sonnect' appears to be a typo for 'Sonnet'.
  2. [Section IV-A] The description of the MCMD dataset says 'We leverage the MCMD dataset obtained by Shi et al. [58] that the authors added...' The antecedent of 'the authors' is ambiguous.
  3. [Section IV-B] The replication of Wu et al. [67] (16-shot result) is described only briefly; details such as prompt templates, model version, temperature, and sample selection would be needed for reproducibility.
  4. [Section IV-C] The paper states the accuracy improvement is 'up to 6.2%', but no confidence intervals or significance tests are reported for the differences in Table III.

Circularity Check

1 steps flagged · score 2.0 of 10

Central 6.2% accuracy claim is externally benchmarked and not circular; only the productivity/reuse evidence is self-referential.

  1. self definitional [Section IV-C, 'Productivity of developing AI judge systems' (Table II); cross-reference Section III Stage II]
    "In the AI judge system with our proposed framework, the majority (i.e., an average of 58%) of the general principles that are generated in Stage I are reused in the contextualized constitutions (Stage II) across the 5 programming languages. ... The high percentages of reusable principles also indicate the high quality of generated principles through the iterations of critiques and revisions in our framework, similar to the prior work [11]."

    Stage II is defined as specializing the Stage I general constitution: 'develop a new set of specific principles with assistance from an FM that embeds the general constitution as knowledge.' Thus the 'reused' percentage is the overlap between an input and its own transformation: by construction, the specialized constitution is generated from, and prompted with, the general principles, so retaining them is a measure of internal invariance, not an external measure of productivity or quality. The same overlap is then used to conclude 'high quality', making the productivity/quality claim self-referential. No independent baseline (manual effort, human-ranked quality) is used. This is minor relative to the central accuracy claim, which is benchmarked externally against MetricVote.

full rationale

The headline accuracy claim (up to 6.2%) is not circular in the derivation sense. It is evaluated by Eq. 3 as agreement between JudgeVote (Eq. 2) and MetricVote (Eq. 1), where MetricVote is an external, automatic-metric majority vote over BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT. The framework's judge is not fitted to MetricVote, and no parameter is learned from that heuristic; the comparison is therefore an external benchmark rather than a self-defined prediction. Concerns that MetricVote may not track human judgments of commit-message quality are validity threats, not circularity. Similarly, the use of a holistic 0-to-n baseline alongside a sum-of-principles score creates a comparison confound, but it does not reduce Eq. 3 to Eq. 1 by construction. Self-citations such as [32] and [33] are contextual and not load-bearing for the central result. The only mildly circular element is the productivity evidence: reuse percentages are computed between two stages of the same LLM-driven constitution pipeline, so the claimed productivity benefit is partly self-referential. This is a minor, non-central circularity, yielding an overall score of 2.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on assumptions about ground-truth validity, baseline equivalence, and the transferability of LLM-generated principles. There are no fitted numerical parameters in the formal sense, but several hand-set choices affect the evaluation: the 3-of-5 metric majority, the addition of BLEURT, four critique rounds, 16-shot replication, and sample sizes. No new physical or formal entities are introduced; the framework's constitution and agents are process artifacts rather than invented entities with independent evidence.

free parameters (6)
  • Majority-vote threshold in MetricVote (Eq. 1) = 3 out of 5 metrics
    Chosen as the voting rule for ground truth; no justification or sensitivity analysis is given. BLEURT was added to reduce ties, which changes the threshold's behavior.
  • Number of critique and revision rounds for principles = 4 rounds
    Adopted from Constitutional AI [11]; no experiment tests whether fewer or more rounds produce better principles.
  • Few-shot setting for replicated CMG FMware = 16-shot
    Replicates Wu et al. [67], chosen to avoid context-window limits, not independently tuned.
  • Per-language sample sizes = 377 or 379 data points
    Random representative sample with CL=95% and CI=5%; sizes are not derived from any power analysis for pair accuracy differences.
  • Addition of BLEURT as fifth metric = 5 metrics total
    Added to reduce ties in majority voting; this hand-set choice directly shapes the ground truth.
  • Baseline holistic score scale = 0 to n, where n is the number of principles
    Chosen so the with-framework and without-framework conditions are on a comparable scale; this conflates scoring granularity with the framework.
assumptions (5)
  • domain assumption MetricVote, a majority vote over BLEU, ROUGE-L, CiDEr, METEOR, and BLEURT, correctly orders pairs of commit messages by quality.
    Used as ground truth in Eq. 1 and Section IV-B; no human validation is provided, and the metric set is reference-based rather than assessing the principles in the constitution.
  • domain assumption A single LLM judge, GPT-4o-2024-05-13, prompted with constitution principles produces scores whose aggregation reflects commit-message quality.
    The case study in Section IV-B uses this model and prompt style without calibration against human judgments or variance analysis.
  • ad hoc to paper The without-framework baseline, a holistic 0-to-n score, is a fair representation of AI judge development without the framework.
    The comparison in Section IV-B changes both the presence of the constitution and the scoring granularity, since the framework condition sums per-principle scores.
  • ad hoc to paper Reuse percentage of LLM-generated principles across languages is a valid measure of development effort reduction.
    Section IV-C infers productivity from 53-64% reuse ratios and one internal user quote, not from measured developer time.
  • domain assumption Four rounds of critique and revision, from Constitutional AI [11], are sufficient to make principles high quality.
    Stages I and II apply at least four rounds; no within-paper evidence tests this sufficiency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Engineering AI Judge Systems." pith.science (2026). https://pith.science/paper/RXZYBB5G

@misc{pith2026241117793,
  author       = {Pith},
  title        = {Pith review of: Engineering AI Judge Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RXZYBB5G}},
  note         = {Machine review of arXiv:2411.17793}
}
read the original abstract

AI judge systems are designed to automatically evaluate Foundation Model-powered software (i.e., FMware). Due to the intrinsic dynamic and stochastic nature of FMware, the development of AI judge systems requires a unique engineering life cycle and presents new challenges. In this paper, we discuss the challenges based on our industrial experiences in developing AI judge systems for FMware. These challenges lead to substantial time consumption, cost and inaccurate judgments. We propose a framework that tackles the challenges with the goal of improving the productivity of developing high-quality AI judge systems. Finally, we evaluate our framework with a case study on judging a commit message generation FMware. The accuracy of the judgments made by the AI judge system developed with our framework outperforms those made by the AI judge system that is developed without our framework by up to 6.2%, with a significant reduction in development effort.

Figures

Figures reproduced from arXiv: 2411.17793 by the authors.

Figure 1
Figure 1. Illustration of common cognitive architectures for judging. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the search-driven constitution-based framework. The [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 42 canonical work pages

  1. [1]

    Claude 3 sonnet has become very lazy,

    “Claude 3 sonnet has become very lazy,” accessed: 2024-09-

    work page 2024

  2. [2]

    Do you guy think the cost of gpt-4 is high,

    “Do you guy think the cost of gpt-4 is high,” accessed: 2024-09-24. [Online]. Available: https://community.openai.com/t/do-you-guy-think- the-cost-of-gpt-4-is-high/622210

    work page 2024

  3. [3]

    Gpt-4 is crazy expensive,

    “Gpt-4 is crazy expensive,” accessed: 2024-09-24. [Online]. Available: https://news.ycombinator.com/item?id=35604896

    work page 2024

  4. [4]

    Open llm leaderboard,

    “Open llm leaderboard,” accessed: 2024-09-24. [Online]. Available: https://huggingface.co/spaces/open-llm-leaderboard/open llm leaderboard

    work page 2024

  5. [5]

    Use agent metrics & llm judges to evaluate app performance,

    “Use agent metrics & llm judges to evaluate app performance,” accessed: 2024-09-24. [Online]. Available: https://learn.microsoft.com/ en-us/azure/databricks/generative-ai/agent-evaluation/llm-judge-metrics

    work page 2024

  6. [6]

    2030 software engineering,

    “2030 software engineering,” https://conf.researchr.org/home/2030-se, 2024, accessed: 2024-09-24

    work page 2024

  7. [7]

    The acm international conference on the foundations of software en- gineering (fse) 2024,

    “The acm international conference on the foundations of software en- gineering (fse) 2024,” https://2024.esec-fse.org/, 2024, accessed: 2024- 09-24

    work page 2024

  8. [8]

    Fm+se summit 2024,

    “Fm+se summit 2024,” https://fmse.io/, 2024, accessed: 2024-09-24

    work page 2024

Show all 70 references
  1. [9]

    Opea initiative (open platform for enterprise ai (opea),

    “Opea initiative (open platform for enterprise ai (opea),” https://opea. dev/, 2024, accessed: 2024-09-24

    2024

  2. [10]

    Re-thinking data strategy and integration for artificial intelligence: concepts, opportuni- ties, and challenges,

    A. Aldoseri, K. N. Al-Khalifa, and A. M. Hamouda, “Re-thinking data strategy and integration for artificial intelligence: concepts, opportuni- ties, and challenges,” Applied Sciences, vol. 13, no. 12, p. 7082, 2023

    2023

  3. [11]

    Constitutional ai: harmlessness from ai feedback,

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones et al., “Constitutional ai: harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  4. [12]

    Leak, cheat, repeat: data contamination and evaluation malpractices in closed-source llms,

    S. Balloccu, P. Schmidtov ´a, M. Lango, and O. Du ˇsek, “Leak, cheat, repeat: data contamination and evaluation malpractices in closed-source llms,” arXiv preprint arXiv:2402.03927 , 2024

    2024 arXiv

  5. [13]

    Meteor: an automatic metric for MT evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: an automatic metric for MT evalua- tion with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , 2005

    2005

  6. [14]

    Test driven development: By example addison-wesley,

    K. Beck, “Test driven development: By example addison-wesley,” The Addison-Wesley Signature Series, 2002

    2002

  7. [15]

    On the dangers of stochastic parrots: can language models be too big?

    E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: can language models be too big?” in Proceedings of the ACM conference on fairness, accountability, and transparency, 2021

    2021

  8. [16]

    Sparks of artificial general intelligence: early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Ka- mar et al. , “Sparks of artificial general intelligence: early experiments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023

    2023 arXiv

  9. [17]

    Chateval: towards better llm-based evaluators through multi-agent debate,

    C.-M. Chan, W. Chen, Y . Su, J. Yu, W. Xue, S. Zhang et al., “Chateval: towards better llm-based evaluators through multi-agent debate,” arXiv preprint arXiv:2308.07201, 2023

    2023 arXiv

  10. [18]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang et al. , “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , 2024

    2024

  11. [19]

    Unleashing the potential of prompt engineering in large language models: a comprehensive review,

    B. Chen, Z. Zhang, N. Langren ´e, and S. Zhu, “Unleashing the potential of prompt engineering in large language models: a comprehensive review,” arXiv preprint arXiv:2310.14735 , 2023

    2023 arXiv

  12. [20]

    Towards training reproducible deep learning models,

    B. Chen, M. Wen, Y . Shi, D. Lin, G. K. Rajbahadur, and Z. M. Jiang, “Towards training reproducible deep learning models,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 2202–2214

    2022

  13. [21]

    Humans or llms as the judge? a study on judgement biases,

    G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang, “Humans or llms as the judge? a study on judgement biases,” arXiv preprint arXiv:2402.10669, 2024

    2024 arXiv

  14. [22]

    How is chatgpt’s behavior changing over time?

    L. Chen, M. Zaharia, and J. Zou, “How is chatgpt’s behavior changing over time?” arXiv preprint arXiv:2307.09009 , 2023

    2023 arXiv

  15. [23]

    Chatbot arena: an open platform for evaluating llms by human preference,

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez et al., “Chatbot arena: an open platform for evaluating llms by human preference,” arXiv preprint arXiv:2403.04132, 2024

    2024 arXiv

  16. [24]

    Available: https://www.reddit.com/r/ClaudeAI/comments/ 1bv8ww5/claude 3 sonnet has become very lazy/

    [Online]. Available: https://www.reddit.com/r/ClaudeAI/comments/ 1bv8ww5/claude 3 sonnet has become very lazy/

  17. [25]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  18. [26]

    Evalullm: llm assisted evaluation of generative outputs,

    M. Desmond, Z. Ashktorab, Q. Pan, C. Dugan, and J. M. Johnson, “Evalullm: llm assisted evaluation of generative outputs,” in Companion Proceedings of the International Conference on Intelligent User Inter- faces, 2024

    2024

  19. [27]

    Qlora: ef- ficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: ef- ficient finetuning of quantized llms,” in Advances in Neural Information Processing Systems, 2023

    2023

  20. [28]

    Fira: fine-grained graph-based code change representation for automated com- mit message generation,

    J. Dong, Y . Lou, Q. Zhu, Z. Sun, Z. Li, W. Zhang, and D. Hao, “Fira: fine-grained graph-based code change representation for automated com- mit message generation,” in Proceedings of the International Conference on Software Engineering , 2022

    2022

  21. [29]

    Alpacafarm: a simulation framework for methods that learn from human feedback,

    Y . Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba et al. , “Alpacafarm: a simulation framework for methods that learn from human feedback,” in Advances in Neural Information Processing Systems, 2024

    2024

  22. [30]

    Bias and fairness in large language models: a survey,

    I. O. Gallegos, R. A. Rossi, J. Barrow, M. M. Tanjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, and N. K. Ahmed, “Bias and fairness in large language models: a survey,” Computational Linguistics, 2024

    2024

  23. [31]

    Self-guided noise- free data generation for efficient zero-shot learning,

    J. Gao, R. Pi, Y . Lin, H. Xu, J. Ye, Z. Wu et al. , “Self-guided noise- free data generation for efficient zero-shot learning,” arXiv preprint arXiv:2205.12679, 2022

    2022 arXiv

  24. [32]

    Llm-based nlg evaluation: current status and challenges,

    M. Gao, X. Hu, J. Ruan, X. Pu, and X. Wan, “Llm-based nlg evaluation: current status and challenges,” arXiv preprint arXiv:2402.01383 , 2024

    2024 arXiv

  25. [33]

    Fm+se vision 2030,

    A. Hassan, B. Adams, F. Khomh, N. Nagappan, and T. Zimmermann, “Fm+se vision 2030,” https://fmse.io/vision/index.html, 2024, accessed: 2024-09-24

    2024

  26. [34]

    Rethinking software engineering in the foundation model era: a curated catalogue of challenges in the development of trustworthy fmware,

    A. E. Hassan, D. Lin, G. K. Rajbahadur, K. Gallaba, F. R. C ˆogo, B. Chen, H. Zhang, K. Thangarajah, G. A. Oliva, J. Lin et al. , “Rethinking software engineering in the foundation model era: a curated catalogue of challenges in the development of trustworthy fmware,” in Compa...

    2024

  27. [35]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  28. [36]

    A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang et al. , “A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232, 2023

    2023 arXiv

  29. [37]

    Ai safety via debate,

    G. Irving, P. Christiano, and D. Amodei, “Ai safety via debate,” arXiv preprint arXiv:1805.00899, 2018

    2018 arXiv

  30. [38]

    Kejriwal, Domain-specific knowledge graph construction

    M. Kejriwal, Domain-specific knowledge graph construction. Springer, 2019

    2019

  31. [39]

    On scalable oversight with weak llms judging strong llms,

    Z. Kenton, N. Y . Siegel, J. Kram ´ar, J. Brown-Cohen, S. Albanie, J. Bulian et al. , “On scalable oversight with weak llms judging strong llms,” arXiv preprint arXiv:2407.04622 , 2024

    2024 arXiv

  32. [40]

    Debating with more persuasive llms leads to more truthful answers,

    A. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rockt ¨aschel, and E. Perez, “Debating with more persuasive llms leads to more truthful answers,” arXiv preprint arXiv:2402.06782, 2024

    2024 arXiv

  33. [41]

    Dspy: compiling declarative language model calls into state-of-the-art pipelines,

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. V . A et al. , “Dspy: compiling declarative language model calls into state-of-the-art pipelines,” in International Conference on Learning Representations, 2024

    2024

  34. [42]

    Software engineering for machine learning applications (semla) 2023,

    F. Khomh, H. Li, M. Lamothe, M. Hamdaqa, J. Cheng et al., “Software engineering for machine learning applications (semla) 2023,” https:// semla.polymtl.ca/, 2023, accessed: 2024-09-24

    2023

  35. [43]

    Understanding the effects of rlhf on llm generalisation and diversity,

    R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefen- stette, and R. Raileanu, “Understanding the effects of rlhf on llm generalisation and diversity,” arXiv preprint arXiv:2310.06452 , 2023

    2023 arXiv

  36. [44]

    On the role of knowledge graphs in explainable ai,

    F. Lecue, “On the role of knowledge graphs in explainable ai,” Semantic Web, vol. 11, no. 1, pp. 41–51, 2020

    2020

  37. [45]

    Encouraging divergent thinking in large language models through multi-agent debate,

    T. Liang, Z. He, W. Jiao, X. Wang, Y . Wang, R. Wang, Y . Yang et al., “Encouraging divergent thinking in large language models through multi-agent debate,” arXiv preprint arXiv:2305.19118 , 2023

    2023 arXiv

  38. [46]

    Rouge: a package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: a package for automatic evaluation of summaries,” in Text Summarization Branches Out , 2004

    2004

  39. [47]

    Best practices and lessons learned on synthetic data for language models,

    R. Liu, J. Wei, F. Liu, C. Si, Y . Zhang, J. Rao et al., “Best practices and lessons learned on synthetic data for language models,” arXiv preprint arXiv:2404.07503, 2024

    2024 arXiv

  40. [48]

    Calibrating llm-based evaluator,

    Y . Liu, T. Yang, S. Huang, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, and Q. Zhang, “Calibrating llm-based evaluator,” arXiv preprint arXiv:2309.13308, 2023

    2023 arXiv

  41. [49]

    Generating training data with language models: towards zero-shot language understanding,

    Y . Meng, J. Huang, Y . Zhang, and J. Han, “Generating training data with language models: towards zero-shot language understanding,” in Advances in Neural Information Processing Systems , 2022

    2022

  42. [50]

    Cider: robust consensus-based image description evaluation,

    G. Oliveira dos Santos, E. L. Colombini, and S. Avila, “Cider: robust consensus-based image description evaluation,” in Proceedings of the Workshop on Noisy User-generated Text (W-NUT 2021) , 2021

    2021

  43. [51]

    A framework for evaluating and improving requirements specifications based on the developers and testers perspective,

    A. C. Oran, G. Santos, B. Gadelha, and T. Conte, “A framework for evaluating and improving requirements specifications based on the developers and testers perspective,” Requirements Engineering, vol. 26, no. 4, pp. 481–508, 2021

    2021

  44. [52]

    Human-centered design recommendations for llm-as-a-judge,

    Q. Pan, Z. Ashktorab, M. Desmond, M. S. Cooper, J. Johnson, R. Nair, E. Daly, and W. Geyer, “Human-centered design recommendations for llm-as-a-judge,” arXiv preprint arXiv:2407.03479 , 2024

    2024 arXiv

  45. [53]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2002

    2002

  46. [54]

    V . B. Parthasarathy, A. Zafar, A. Khan, and A. Shahid, “The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research chal- lenges and opportunities,” arXiv preprint arXiv:2408.13296 , 2024

    2024 arXiv

  47. [55]

    Verbosity bias in preference labeling by large language models,

    K. Saito, A. Wachi, K. Wataoka, and Y . Akimoto, “Verbosity bias in preference labeling by large language models,” arXiv preprint arXiv:2310.10076, 2023

    2023 arXiv

  48. [56]

    Bleurt: learning robust metrics for text generation,

    T. Sellam, D. Das, and A. P. Parikh, “Bleurt: learning robust metrics for text generation,” arXiv preprint arXiv:2004.04696 , 2020

    2004 arXiv

  49. [57]

    On automatic summarization of what and why information in source code changes,

    J. Shen, X. Sun, B. Li, H. Yang, and J. Hu, “On automatic summarization of what and why information in source code changes,” in IEEE Annual Computer Software and Applications Conference (COMPSAC) , 2016

    2016

  50. [58]

    On the evaluation of neural code summarization,

    E. Shi, Y . Wang, L. Du, J. Chen, S. Han, H. Zhang et al. , “On the evaluation of neural code summarization,” in Proceedings of the international conference on software engineering , 2022

    2022

  51. [59]

    RACE: Retrieval-augmented commit message generation,

    E. Shi, Y . Wang, W. Tao, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “RACE: Retrieval-augmented commit message generation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022

    2022

  52. [60]

    On the evaluation of commit message generation models: an experimental study,

    W. Tao, Y . Wang, E. Shi, L. Du, S. Han, H. Zhang, D. Zhang, and W. Zhang, “On the evaluation of commit message generation models: an experimental study,” in2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) , 2021

    2021

  53. [61]

    Alpaca: a strong, replicable instruction-following model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, and other, “Alpaca: a strong, replicable instruction-following model,” Stanford Center for Research on Foundation Models , 2023

    2023

  54. [62]

    Judging the judges: evaluating alignment and vulnerabilities in llms-as-judges,

    A. S. Thakur, K. Choudhary, V . S. Ramayapally, S. Vaidyanathan, and D. Hupkes, “Judging the judges: evaluating alignment and vulnerabilities in llms-as-judges,” arXiv preprint arXiv:2406.12624 , 2024

    2024 arXiv

  55. [63]

    Synthetic data, real errors: how (not) to publish and use synthetic data,

    B. Van Breugel, Z. Qian, and M. Van Der Schaar, “Synthetic data, real errors: how (not) to publish and use synthetic data,” in International Conference on Machine Learning , 2023

    2023

  56. [64]

    Replacing judges with juries: evaluating llm generations with a panel of diverse models,

    P. Verga, S. Hofstatter, S. Althammer, Y . Su, A. Piktus, A. Arkhangorod- sky et al., “Replacing judges with juries: evaluating llm generations with a panel of diverse models,” arXiv preprint arXiv:2404.18796 , 2024

    2024 arXiv

  57. [65]

    Systematic evaluation of llm-as-a-judge in llm alignment tasks: explainable metrics and diverse prompt templates,

    H. Wei, S. He, T. Xia, A. Wong, J. Lin, and M. Han, “Systematic evaluation of llm-as-a-judge in llm alignment tasks: explainable metrics and diverse prompt templates,” arXiv preprint arXiv:2408.13006, 2024

    2024 arXiv

  58. [66]

    Offline energy-optimal llm serv- ing: Workload-based energy models for llm inference on heterogeneous systems,

    G. Wilkins, S. Keshav, and R. Mortier, “Offline energy-optimal llm serv- ing: Workload-based energy models for llm inference on heterogeneous systems,” arXiv preprint arXiv:2407.04014 , 2024

    2024 arXiv

  59. [67]

    Fake it till you make it: face analysis in the wild using synthetic data alone,

    E. Wood, T. Baltru ˇsaitis, C. Hewitt, S. Dziadzio, T. J. Cashman, and J. Shotton, “Fake it till you make it: face analysis in the wild using synthetic data alone,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021

    2021

  60. [68]

    Commit message generation via chatgpt: how far are we?

    Y . Wu, Y . Li, and S. Yu, “Commit message generation via chatgpt: how far are we?” in Proceedings of the IEEE/ACM First International Conference on AI Foundation Models and Software Engineering , 2024

    2024

  61. [69]

    Automatic commit message generation: a critical review and directions for future work,

    Y . Zhang, Z. Qiu, K.-J. Stol, W. Zhu, J. Zhu, Y . Tian, and H. Liu, “Automatic commit message generation: a critical review and directions for future work,” IEEE Transactions on Software Engineering , 2024

    2024

  62. [70]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” in Advances in Neural Information Processing Systems , 2023

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.