Pith. sign in

REVIEW 3 major objections 6 minor 32 references

BeSpec: Behavior-Level Specification Alignment for Code Generation

T0 review · 3 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BeSpec improves LLM code generation by repairing ambiguous intents with checkable behaviors, not by only patching code after tests fail.

desk verdict Solid empirical SE paper: behavior-level checks beat test-level alignment on hard contests, but the causal story still rests on an unvalidated premise about what candidate disagreement means. read the letter →

arxiv 2607.02949 v1 pith:AMHKP2YH submitted 2026-07-03 cs.SE

Qinghua Xu , Guancheng Wang , Boxi Yu , Lionel Briand This is my paper · ORCID
classification cs.SE
keywords specificationalignmentcodegenerationlargelanguagemodelsbehavioralintentclarificationPass@1LiveBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most LLM code generators treat a natural-language problem statement as already correct and only refine the code when tests fail. This paper argues that many failures come earlier: the statement is ambiguous or incomplete, so the model implements a coherent solution to the wrong problem. BeSpec re-expresses the intent as a structured specification, predicts small checkable behavioral properties the correct program should satisfy, then runs candidate programs on probe inputs and compares observed versus predicted behavior. When candidates disagree with a predicted property, BeSpec repairs that part of the specification and regenerates. Across three models and six settings—including contamination-free LiveCodeBench—it beats nine baselines on Pass@1 and average pass rate, and after alignment most leftover failures look algorithmic rather than specification-related.

What carries the argument

Predicted-versus-observed behavior alignment: BeSpec turns the intent into executable check functions (gold, output, input, and semantic behaviors), runs a candidate pool on shared probe scenarios, and treats systematic mismatch or disagreement on a predicted behavior as evidence that that slice of the specification is ambiguous and should be repaired.

What would settle it

On problems with known ground-truth clarifications, count how often BeSpec’s misalignment flag points to the true missing rule versus how often the predicted behavior is itself wrong or candidates merely implement different correct algorithms; if most flags are false positives, the repair loop is not driven by real specification gaps.

Watch

Extended reading notes

Core claim

Specification mismatch is a central bottleneck in LLM code generation from natural-language intents. Predicting checkable behavioral properties from the intent, executing candidates on probe inputs without full oracles, and using prediction–observation gaps to refine the specification yields higher Pass@1 and average pass rates than both code-only refinement and prior test-level specification alignment methods.

Load-bearing premise

Disagreement among candidates on a predicted checkable behavior mainly means the specification is ambiguous, not that the prediction is wrong or the probes are uninformative.

Editorial extensions

If this is right

  • After behavior-level alignment, residual failures are mostly algorithmic or resource errors rather than weak specifications.
  • Behavior-level checks can guide repair even when public samples or complete oracles are sparse.
  • Relative gains are largest on weaker open-weight models, so explicit alignment can partly offset base-model limits.
  • Improvements hold on LiveCodeBench problems released after training cutoffs, so they are unlikely to be pure memorization.
  • Candidate pool size and repair iterations trade off: larger pools need fewer refinement rounds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If some disagreements are only implementation diversity, filtering weak predicted behaviors before repair could cut wasted loops.
  • The same predict–observe–repair loop may transfer to other underspecified artifacts such as API contracts or test oracles.
  • A hybrid that first aligns the specification then runs strong code repair may close the remaining adherence gap.
  • Long contest-style statements are the natural setting; short function-completion tasks may show much smaller lift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. BeSpec addresses specification mismatch in LLM code generation by moving from test-level repair to behavior-level alignment. From a natural-language intent it extracts a five-field structured specification, predicts checkable behavioral obligations (as check(input, output) functions), generates probe inputs and a pool of candidate programs, and compares predicted vs. observed behaviors. Candidate disagreement on a predicted behavior is treated as evidence of underspecification and triggers targeted specification repair; a final behavior-grounded selection step returns one program. Evaluated with three LLMs on CodeContests, xCodeEval, APPS (plus extended suites), and contamination-free LiveCodeBench against nine baselines, BeSpec reports the highest Pass@1 and APR in every setting, with relative Pass@1 gains of 8.1%–25.3% over the strongest baseline. A failure taxonomy (LLM-as-judge, 87% agreement, κ=0.86 on 100 stratified cases) attributes most residual errors to algorithmic/adherence issues rather than weak specs, and ablations credit the behavior model and pool size.

Significance. If the results hold, the paper makes a clear contribution to LLM code generation by reframing specification alignment around checkable behavioral properties rather than full test oracles or code-only repair. Strengths include a multi-LLM, multi-benchmark evaluation with LiveCodeBench contamination control; direct comparison to the closest specification-alignment baselines (Specine, SpecFix); an ablation removing the behavior model (Table II); and a validated failure-mode analysis showing that residual failures are mostly algorithmic. These elements make the empirical claim falsifiable and useful for the community even if some causal details of the repair signal need tightening. The work is significant for competitive-programming-style generation where intents are long and ambiguous, and for resource-constrained settings where smaller models benefit most from explicit alignment.

major comments (3)
  1. [§III-E–III-F, Misalignment Identification] Sections III-E–III-F (Misalignment Identification / Specification Fixing) and the erase-game example treat predicted behavior bi as the reference and treat bi ≠ ˆbi or candidate disagreement on bi as evidence that the specification is ambiguous and should be repaired. This is the load-bearing causal premise of the paper’s claim that gains come from behavior-level alignment of true intent. The manuscript never validates predicted behaviors against gold solutions or private oracles (gold behaviors are only public samples). Incorrect predictions, weak probes X, or pure implementation diversity could produce the same disagreement signal and still yield gains via rewritten prompts and re-sampling. Please add either (i) a direct check of predicted behaviors against gold programs / private tests on a sample of problems, or (ii) a controlled experiment that injects known correct vs. deliberately
  2. [Table II / §V-C1] Table II, row Vw/o bh: removing the behavior model drops Pass@1/APR substantially, but the variant also removes behavior-guided repair, probe-based comparison, and the associated regeneration loop. It therefore confounds “useful behavior signal” with “extra sampling + iterative rewrite budget.” A fairer ablation would keep the same candidate pool size, iteration budget, and structured-specification rewrite loop while replacing behavior disagreement with a non-behavior signal (e.g., random rewrite, test-level disagreement only, or no disagreement trigger). Until that control exists, the claim that “each major component contributes positively” (abstract / RQ3) is only partially supported for the behavior model specifically.
  3. [Table I / §V-A2 / Table II] §IV-E and Table I efficiency columns: SpecFix and BeSpec both use pool size 20 and N=10, but BeSpec’s mean token cost is still high (60.3k–122.1k) and time is 364–1132s per problem. The ablation (Table II) shows most accuracy is reached by n≈10 with ~1.6–1.7 iterations, yet RQ1 reports the expensive configuration. For the central claim of practical superiority over Specine/SpecFix, please report Pass@1/APR at a matched or lower token budget (e.g., n=10, early-stop as in the ablation) against Specine and SpecFix under the same budget, so gains are not partly an artifact of unequal compute.
minor comments (6)
  1. [Table II caption] Table II caption: “CCANDxCEARE SHORT FOR CodeContentsANDxCodeEval” — “CodeContents” should be “CodeContests”; spacing/typos in the caption should be cleaned.
  2. [Fig. 1] Figure 1 is dense; the Predicted vs. Observed pipelines and the alignment matrix are hard to parse at print size. Consider splitting into a pipeline overview and a small worked matrix for the erase-game example.
  3. [§IV-B Datasets] §IV-B: APPS and xCodeEval use stratified samples of 300; state the random seed and release the exact problem IDs for reproducibility.
  4. [Fig. 1 / §III-E] Notation: predicted behaviors B vs. observed ˆBj and bi vs. ˆbi are introduced clearly in III-E, but the matrix description in Fig. 1 uses mixed b$ / b* notation that does not match the text. Align figure labels with §III notation.
  5. [§V-B2] §V-B2: report the full judge prompt and the confusion matrix for the 100-case human validation (not only 87% / κ=0.86) so readers can assess class-wise reliability of “weak spec” vs. adherence subtypes.
  6. [§VII-B] Related work (§VII) cites SpecAlign and ReaCoder as unpublished without packages; a short sentence on how BeSpec would differ mechanistically if those packages appear would help future readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: BeSpec’s reported gains are measured on independent private-test oracles, not on agreement with its own predicted behaviors.

full rationale

BeSpec is an empirical code-generation pipeline, not a first-principles derivation. Predicted behaviors B are LLM-extracted checkable properties used only to drive specification repair and candidate selection (III-C–III-G). Pass@1 and APR are scored exclusively against held-out private tests on CodeContests, xCodeEval, APPS/APPS-Eval, and LiveCodeBench (IV-B–IV-C, Table I); those oracles are not defined by B, so success is not forced by construction. Ablation Vw/o bh (Table II) and the failure-mode taxonomy (RQ2) are independent of the evaluation metric. Author self-citations in Related Work (mutation-guided testing, multi-agent JUnit generation) are background and not load-bearing uniqueness claims. There is no fitted parameter renamed as a prediction, no self-definitional loop between X and Y, and no uniqueness theorem imported from the authors. The skeptic concern that candidate disagreement may reflect wrong predicted behaviors rather than true ambiguity is a causal/validity issue, not circularity of the reported result.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

This is an empirical systems paper. The load-bearing content is a method design plus measured gains, not a closed-form derivation. Free parameters are engineering knobs (pool size, iteration budget, temperature). Axioms are domain assumptions about ambiguous contest intents and about LLM-predicted checks being useful signals. Invented entities are the paper’s intermediate artifacts (structured specification, behavioral obligations as check functions, probe scenarios), not physical objects; they have independent operational handles via execution but no external existence beyond the pipeline.

free parameters (4)
  • candidate_pool_size_n = 20
    Set to 20 to match SpecFix for fair comparison; ablation shows accuracy depends strongly on n (Table II).
  • max_refinement_iterations_N = 10
    Maximum repair iterations for BeSpec and baselines; early stopping also used. Directly bounds search cost and opportunity for specification repair.
  • LLM_temperature = 0.8
    Sampling temperature for all LLM calls, following prior practice; affects candidate diversity and behavior prediction.
  • APPS_xCodeEval_sample_size = 300
    300 stratified test problems each, chosen for cost/generalizability rather than full official test sets.
assumptions (4)
  • domain assumption Natural-language programming intents are often ambiguous or underspecified, so correct code for a wrong interpretation is a primary failure mode.
    Stated in the abstract and Introduction; motivates specification alignment over code-only repair.
  • domain assumption Checkable behavioral properties are easier for an LLM to predict from the intent than full solution oracles for arbitrary probe inputs.
    Core design claim in Sections I and III-C; justifies behavior-level over test-level alignment.
  • ad hoc to paper Disagreement among candidates on a predicted behavior indicates that the current specification induces multiple interpretations of that behavior and should be repaired.
    Operational rule in Misalignment Identification (III-E) and Specification Fixing (III-F); gold behaviors are the exception.
  • domain assumption Held-out private tests on contest benchmarks are a valid measure of program correctness for comparing methods.
    Standard SE/code-generation evaluation assumption used throughout Section IV–V.
invented entities (3)
  • structured specification (five fields: io_format, constraints, rules, edge_cases, interpretation)
    purpose: Central editable contract that BeSpec extracts, repairs, and uses for generation and behavior prediction.
    Introduced in Section III-B as the maintained object of alignment; not a prior standard artifact with independent external evidence beyond this pipeline.
  • predicted behavioral obligations as check(input, output) functions
    purpose: Decompose intended behavior into executable properties for alignment without full oracles.
    Core of the Predicted Behavior Pipeline (III-C); falsifiable only inside the method via execution, not as external scientific entities.
  • probe input scenarios for behavior observation
    purpose: Shared inputs that expose agreement/disagreement among candidates on predicted behaviors.
    Generated in Observed Behavior Pipeline (III-D); method-internal scaffolding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BeSpec: Behavior-Level Specification Alignment for Code Generation." pith.science (2026). https://pith.science/paper/AMHKP2YH

@misc{pith2026260702949,
  author       = {Pith},
  title        = {Pith review of: BeSpec: Behavior-Level Specification Alignment for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AMHKP2YH}},
  note         = {Machine review of arXiv:2607.02949}
}
read the original abstract

LLMs have made substantial progress on automated code generation from natural-language descriptions of desired behavior (intent). Most existing methods improve generated programs through execution-guided code refinement: they generate a candidate solution, execute it, and patch the implementation using feedback, while leaving the underlying specification unchanged. This workflow implicitly assumes that the LLM's understanding of the intent is already correct and complete. In practice, however, intents are often ambiguous or underspecified. As a result, even a capable model may produce a correct implementation of the wrong intent, making specification mismatch a central bottleneck. This paper presents BeSpec, a behavioral model-based approach to specification alignment. BeSpec treats the task description as partial evidence about the intended behavior of the correct program. It first builds an explicit behavioral model, which are checkable properties that valid outputs must satisfy. BeSpec then generates candidate programs, executes them on probe inputs, and compares their observed behavior with the predicted behaviors. When observed behavior does not match the predicted behaviors, BeSpec either refines the specification or rejects the candidate program. We evaluate BeSpec with three LLMs on four benchmarks: CodeContests, xCodeEval, APPS, and the contamination-free LiveCodeBench. Against nine baselines, BeSpec achieves the highest Pass@1 and average pass rate across all settings, improving average Pass@1 over the strongest baseline by 8.1%--25.3% relative across the three LLMs. A failure analysis shows that after alignment, most remaining errors stem from algorithmic difficulty rather than misunderstood specifications, while ablation studies confirm that each major component of BeSpec contributes positively.

Figures

Figures reproduced from arXiv: 2607.02949 by the authors.

Figure 1
Figure 1. Overview of BESPEC, illustrated with the running example. C. The Predicted Behavior Pipeline This pipeline starts with the structured specification and then asks the LLM to predict a set of m intended behaviors, B = {b1, b2, . . . , bm}. These behaviors model how the gold solution (correct program) should behave, without asking the LLM to generate it directly. This is easier than solving the full problem because a s… view at source ↗
Figure 2
Figure 2. Problem distribution by the number of methods that solve each [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Failure mode analysis of BESPEC, Specine, and SpecFix on hard problems. interpretation [28] of Cohen’s κ statistic [29], this indicates strong agreement. This validation supports using the LLM judge for the full failure set, consistent with prior LLM judge protocols [30] [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 6 linked inside Pith

  1. [1]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  2. [2]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosmaet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    Competition-level code generation with AlphaCode,

    Y . Li, D. Choi, J. Chung, N. Kushmanet al., “Competition-level code generation with AlphaCode,”Science, vol. 378, no. 6624, pp. 1092– 1097, 2022

    2022

  4. [4]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”ACM Transactions on Software Engineering and Methodology (TOSEM), 2026

    2026

  5. [5]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 2023, pp. 31–53

    2023

  6. [6]

    Automated repair of ambiguous problem descriptions for LLM-based code generation,

    H. Jia, R. Morris, H. Ye, F. Sarro, and S. Mechtaev, “Automated repair of ambiguous problem descriptions for LLM-based code generation,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025

    2025

  7. [7]

    Aligning requirement for large language model’s code generation,

    Z. Tian and J. Chen, “Aligning requirement for large language model’s code generation,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026

    2026

  8. [8]

    Fixing large language models’ speci- fication misunderstanding for better code generation,

    Z. Tian, J. Chen, and X. Zhang, “Fixing large language models’ speci- fication misunderstanding for better code generation,” inProceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE), 2025

    2025

Show all 32 references
  1. [9]

    ClarifyGPT: A framework for enhancing LLM-based code generation via requirements clarification,

    F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang, “ClarifyGPT: A framework for enhancing LLM-based code generation via requirements clarification,”Proceedings of the ACM on Software Engineering (FSE), 2024

    2024

  2. [10]

    Bridging the gap between user intent and llm: A requirement alignment approach for code generation,

    J. Li, R. Bai, Y . Luo, Y . Zhang, W. Yang, Z. Sun, T. Zhao, D. Jin, L. Li, and Z. Jin, “Bridging the gap between user intent and llm: A requirement alignment approach for code generation,”arXiv preprint arXiv:2604.16198, 2026

    2026 arXiv

  3. [11]

    Specalign: Efficient specification-grounded alignment of large language models via synthetic data,

    W. Wang, Y . Huang, Z. Yuan, H. Bao, S. Du, Y . Ma, Y . Zhao, Y . Ye, and X. Zhang, “Specalign: Efficient specification-grounded alignment of large language models via synthetic data,”arXiv preprint arXiv:2606.16276, 2026

    2026 arXiv

  4. [12]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” inInternational Conference on Learning Repre- sentations (ICLR), 2024

    2024

  5. [13]

    Is self-repair a silver bullet for code generation?

    T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, “Is self-repair a silver bullet for code generation?” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  6. [14]

    AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui, “AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,”arXiv preprint arXiv:2312.13010, 2023

    2023 arXiv

  7. [15]

    Self-collaboration code gener- ation via ChatGPT,

    Y . Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code gener- ation via ChatGPT,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 33, no. 7, 2024

    2024

  8. [16]

    A pair programming framework for code generation via multi-plan exploration and feedback- driven refinement,

    H. Zhang, W. Cheng, Y . Wu, and W. Hu, “A pair programming framework for code generation via multi-plan exploration and feedback- driven refinement,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2024

    2024

  9. [17]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,”ACM Transactions on Software Engineering and Methodology (TOSEM), 2025

    2025

  10. [18]

    Mutation-Guided Unit Test Generation With a Large Language Model ,

    G. Wang, Q. Xu, L. Briand, and K. Liu, “ Mutation-Guided Unit Test Generation With a Large Language Model ,”IEEE Transactions on Software Engineering, vol. 52, no. 05, pp. 1657–1671, May

  11. [19]

    Available: https://doi.ieeecomputersociety.org/10.1109/ TSE.2026.3682975

    [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ TSE.2026.3682975

    2026

  12. [20]

    Hallucination to consensus: Multi-agent llms for end-to-end junit test generation,

    Q. Xu, G. Wang, L. Briand, and K. Liu, “Hallucination to consensus: Multi-agent llms for end-to-end junit test generation,” ACM Trans. Softw. Eng. Methodol., Mar. 2026. [Online]. Available: https://doi.org/10.1145/3803418

    2026 doi

  13. [21]

    TOGLL: Correct and Strong Test Oracle Generation with LLMs,

    S. B. Hossain and M. Dwyer, “TOGLL: Correct and Strong Test Oracle Generation with LLMs,” 5 2024. [Online]. Available: http://arxiv.org/abs/2405.03786

    2024 arXiv

  14. [22]

    Quantum machine learning-based test oracle for autonomous mobile robots,

    X. Wang, Q. Xu, P. Arcaini, S. Ali, and T. Peyrucain, “Quantum machine learning-based test oracle for autonomous mobile robots,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025, pp. 3695–3706

    2025

  15. [23]

    CodeT: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “CodeT: Code generation with generated tests,” inInternational Con- ference on Learning Representations (ICLR), 2023

    2023

  16. [24]

    Enhancing large language mod- els in coding through multi-perspective self-consistency,

    B. Huang, S. Lu, X. Wan, and N. Duan, “Enhancing large language mod- els in coding through multi-perspective self-consistency,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  17. [25]

    XCodeEval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval,

    M. A. M. Khan, M. S. Bari, X. L. Do, W. Wang, M. R. Parvez, and S. Joty, “XCodeEval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval,” inProceedings of the 62nd Annual Meeting of the Association for ...

    2024

  18. [26]

    Measuring coding challenge competence with APPS,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring coding challenge competence with APPS,” inAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021

    2021

  19. [27]

    LiveCodeBench: Holistic and contamination-free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “LiveCodeBench: Holistic and contamination-free evaluation of large language models for code,” in International Conference on Learning Representations (ICLR), 2025

    2025

  20. [28]

    OpenRouter: The unified interface for LLMs,

    OpenRouter, “OpenRouter: The unified interface for LLMs,” https: //openrouter.ai/, 2026, accessed: 2026-06-24

    2026

  21. [29]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,”Biometrics, vol. 33, no. 1, pp. 159–174, 1977

    1977

  22. [30]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,”Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960

    1960

  23. [31]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” inAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Ben...

    2023

  24. [32]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinanet al., “Self-refine: Iterative refinement with self-feedback,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.