ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

Haofei Yu; Jiaxuan You; Zhaochen Hong

REVIEW 5 major objections 5 minor 35 references

ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ConsistencyChecker claims a benchmark-free tree score ranks LLMs in line with WMT 2024 translation metrics.

desk verdict A genuinely new evaluation instrument with an under-powered external validation: the WMT correlations rest on six models and likely one outlier. read the letter →

arxiv 2506.12376 v2 pith:7DYUQX7O submitted 2025-06-14 cs.AI cs.CL

Zhaochen Hong , Haofei Yu , Jiaxuan You This is my paper

classification cs.AIcs.CL

keywords self-consistencytree-basedevaluationbenchmark-freemachinetranslationcodegenerationLLMround-triptransformation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ConsistencyChecker sets out to measure how well an LLM preserves meaning or function across a chain of reversible transformations, such as translating text out of English and back, or rewriting code into a functionally equivalent form. It builds a self-consistency tree whose nodes are LLM outputs and whose edges are paired inverse prompts, then scores a model by averaging the similarity between root and leaf over a forest of such trees. The paper's central evidence is that this score, computed without any paired WMT data, ranks models in close agreement with WMT 2024 auto-ranking metrics: Pearson correlations above $r > 0.8$ for the Czech–Ukrainian and English–Chinese pairs, and $r > 0.7$ with WMT metrics in four of five language pairs. If the claim holds, it offers a cheap, leakage-resistant way to compare LLMs on translation and code tasks without curated reference benchmarks.

What carries the argument

The central object is the self-consistency tree. Nodes are LLM-generated states $(c, I)$, where $c$ is text or code and $I$ is a shared set of test inputs (empty for translation, twenty inputs for code); edges are pairs of inverse operations $(f_p, f'_p)$ defined by prompts; a path is a sequence of such round trips. The score $C_n(\mathcal{F})$ averages, over a forest of $M$ trees, the similarity between root and end node after $n$ steps, and the paper fixes $n=3$, $M=10$. This machinery lets a single framework treat translation and programming uniformly: for translation, similarity is computed on the returned text; for code, on execution outputs for $I$.

What would settle it

Re-run the comparison on the same WMT language pairs using a different embedding model, or with human translation-quality judgments as the target: if ConsistencyChecker rankings diverge from WMT's CometKiwi and AutoRank correlations when the embedding model changes, the reported $r > 0.8$ is an artifact of NV-Embed-v2 rather than of the tree construction.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a model's ability to survive many inverse transformations is a usable proxy for translation quality and code correctness. Starting from evaluator-generated root texts, ConsistencyChecker prompts the model under test through branches such as English→French→English and English→German→English, and, for code, through prompts to rewrite a solution by a different method. Each branch creates a path; the consistency score for path length $n$ is the average similarity $\text{sim}(v_1, v_n)$ between start and end nodes, using either NV-Embed-v2 cosine similarity or BLEU, and the forest-level score $C_n(\mathcal{F})$ averages this over ten root trees. The headline result is that $C_3(\mathcal{F})$ correlates with WMT 2024's CometKiwi and AutoRank metrics with Pearson $r > 0.8$ for the Czech–Ukrainian and English–Chinese pairs, even though none of the WMT parallel data is used.

Load-bearing premise

The authors themselves state that the study covers only translation and code and relies on automated similarity rather than human judgment; within that scope, the load-bearing premise is that embedding cosine similarity (or BLEU) and execution-output comparison faithfully detect whether meaning or function survived a round trip.

Editorial extensions

If this is right

For machine translation, the consistency score promises a reference-free alternative to expensive quality-estimation systems like CometKiwi and AutoRank, since it needs only an evaluator LLM to generate roots and the model under test to perform round trips.
For programming, the same score measures whether code survives rewrites into functionally equivalent forms, giving a leakage-resistant signal for models trained on public code repositories.
The framework can rank eight models of different families and sizes on both tasks, and scores decline with path length, so it also exposes how quickly a model loses information as transformation chains get longer.
Because root nodes are dynamically generated by an evaluator model, the evaluation procedure can be re-run on new topics without reusing examples, reducing the risk of benchmark contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the correlation with WMT rankings holds beyond the five tested pairs, the cheapest use of ConsistencyChecker would be in low-resource language pairs where no parallel corpus exists: an evaluator LLM can write the roots in any language it knows, and the score needs only round-trip outputs, not references.
A harder test the paper does not run: replace NV-Embed-v2 with another embedding model and see whether the WMT correlations persist; if they do not, the reported agreement is partly a property of that specific similarity measure.
The framework's dependence on an evaluator model means its scores could inherit the evaluator's blind spots or biases, so a valuable follow-up would be to vary evaluator identity and language coverage and compare the resulting rankings.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the actual contribution: ConsistencyChecker formalizes a tree of paired inverse operations and aggregates path-level similarity into a score. Round-trip translation and self-consistency sampling are known, but the tree structure, the path/forest aggregation, and the extension to code transformations with execution-based checks make this a new evaluation instrument, not a rehash. The work is clearly written, the code is released, and the WMT 2024 comparison is the right kind of external check. The ablations on path length and evaluator model are useful.

The soft spots are real but not fatal. The headline r>0.8 correlations are computed over six models. With n=6, one outlier can drive the result; in Czech-Ukrainian, Phi-3-Medium is far below the other five, and removing it likely collapses the correlation. The paper reports Pearson coefficients without confidence intervals, leave-one-out, or any rank-based agreement, even though AutoRank is ordinal. That is a load-bearing gap: the abstract's 'highly correlated' claim is not established by the reported numbers. The choice of n=3 is mentioned as based on preliminary experiments that are not shown. The code-execution failure protocol is undocumented: what happens when generated code times out or raises? This matters for the programming task scores. There is also a silent protocol change: for WMT language pairs involving English, the trees are chains with out-degree 1 and height 12; for pairs without English, they use out-degree 3 and height 3. That may be justifiable, but it is not discussed and makes the per-pair results less comparable. Finally, calling the evaluator-generated roots 'dynamic' is a stretch, since the same ten roots are reused for all models; leakage is reduced, not eliminated.

None of this kills the central idea. The framework is sensible, cheap, and worth testing more thoroughly. I would send this to review, with the expectation of major revisions: add uncertainty quantification and robustness checks for the WMT correlations, document the execution failure handling, justify or pre-register the tree-shape choices, and soften the leakage claim.

Referee Report

5 major / 5 minor

Summary. The paper proposes ConsistencyChecker, an evaluation framework that measures LLM consistency over trees of reversible transformations. A root node is generated by an evaluator LLM; edges are pairs of inverse operations (e.g., translate English to French and back, or rewrite code into an equivalent form); and a forest-level consistency score C_n(F) averages the similarity between the root and the final node over all n-step paths. Experiments on machine translation and AI-assisted programming report that larger models receive higher scores, and that consistency scores computed without WMT paired data correlate strongly (Pearson r > 0.7 or > 0.8) with WMT 2024 metrics such as CometKiwi and AutoRank across five language pairs. The authors also provide an open-source implementation and document licenses and model budgets in the appendices.

Significance. The central idea is attractive and genuinely benchmark-free: dynamically generated roots avoid benchmark leakage, and the WMT 2024 comparison is an independent external anchor rather than a fitted target. The release of code and the detailed appendices on artifacts, licenses, and computational resources are strengths. If the headline correlations survived a more careful statistical treatment, ConsistencyChecker would be a useful low-cost complement to reference-based MT evaluation and a plausible template for other reversible-transformation tasks. However, the main empirical support rests on correlations computed over only six models, with no confidence intervals or leave-one-out analysis, and the WMT protocol mixes different tree structures across language pairs. These issues are load-bearing for the paper's central claim rather than cosmetic, so the evidence currently supports conditional acceptance with substantial revision.

major comments (5)

[Section 7, Table 3; Appendix I, Table 9 and Figure 5] The claim that ConsistencyChecker rankings are 'highly correlated' with WMT 2024 metrics is supported only by Pearson correlations over six models. At n=6, the reported r≈0.8 has a 95% confidence interval that easily includes zero, and the Czech-Ukrainian coefficient is visibly driven by the single Phi-3-Medium point: Table 9 gives C3=44.9 and CometKiwi=0.425 for Phi-3-Medium, while the other five models span only 95.9–98.1 and 0.661–0.683, respectively. The paper also applies Pearson to AutoRank, which is ordinal, and among the top five Czech-Ukrainian models the ConsistencyChecker order (Gemini before GPT-4) already disagrees with AutoRank's tie (both GPT-4 and Gemini receive 2.0). Please add confidence intervals, leave-one-out correlations, rank-based coefficients with explicit tie handling, and scatter plots for every language pair; without these, the abstract's 'r > 0.7' claim is not statistically established.
[Appendix F; Section 4] The WMT comparison protocol is inconsistent across language pairs. For pairs involving English, Appendix F sets out-degree=1 and height=12, which produces a chain rather than a branching tree, while for pairs without English it uses out-degree=3 and height=3. Consequently the C1–C3 values in Table 9 are computed over different structures, and the 'tree-based' part of the framework is not uniformly exercised in the headline correlation experiment. Moreover, Section 4 motivates the choice n=3 partly by computational cost, yet the WMT setting uses height 12 for English pairs; please justify this deviation or rerun the comparison with a uniform tree protocol.
[Section 4, Section 5, Appendix J] The code-execution failure protocol is undocumented. In Appendix J, the Qwen-2.5-7B tree contains a Level-3 node labeled 'None' with similarity 0.0520 to the root, which strongly suggests that the model failed to produce executable code. The paper never specifies how exec(c, I) handles syntax errors, runtime exceptions, the 2-second time limit mentioned in Section 5, or empty outputs, nor whether failed nodes are excluded or scored as dissimilar. Because the programming-task score in Section 4 is defined as similarity between execution outputs, this choice directly determines whether non-executability is treated as functional inconsistency, and it materially affects Table 2. Please document and justify the failure-handling protocol.
[Section 4, Section 6, Appendix F] The final metric C3(F) depends on several free design choices—n=3, M=10, temperature 0.6, and the NV-Embed-v2 similarity model—whose selection is attributed to 'preliminary experiments' that are never reported. Since the paper's main conclusion is about rankings, the authors should provide sensitivity analyses showing that model rankings (especially the WMT correlations) are stable under variations of n (e.g., 1, 2, 3, 4), M (e.g., 5, 10, 20), and generation temperature. Without this, it is unclear whether the reported correlations are a property of the consistency concept or an artifact of the specific configuration.
[Section 9, Figure 5] The statement that BLEU-based consistency scores achieve Pearson correlations above 0.7 with three WMT metrics in four of five language pairs is not directly verifiable from Figure 5, where several BLEU bars for English-German appear to fall below 0.7 (the y-axis begins at 0.5). Please provide the exact correlation matrix for all five pairs, all three WMT metrics, and C1/C2/C3, and state explicitly how many of the 45 pair-metric-depth combinations meet the 0.7 threshold.

minor comments (5)

[Appendix F] There is a typo: 'withoiut' should be 'without'.
[Appendix B] The text refers to 'LLaMA-3.1-79B', which appears to be a typo for 'LLaMA-3.1-70B'.
[Figure 2] The example code 'def mult(a, b): answer = a * b; return ans' contains an undefined variable 'ans'; it should presumably be 'return answer'.
[Section 2] The subsection heading 'Formal verification method' is singular and slightly awkward; 'Formal verification methods' would read better.
[Abstract and Section 7] The abstract states 'r > 0.7' while Table 3 is described as showing correlations 'above 0.8'; please harmonize these thresholds so the abstract accurately reflects the strongest supported claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ConsistencyChecker scores are defined independently and validated against WMT, with no fitted parameters or self-citation chain.

full rationale

The derivation of ConsistencyChecker scores is self-contained: the tree construction (Algorithm 1), node-pair similarity, path-level, tree-level, and forest-level scores are defined purely in terms of the evaluated model's own outputs under reversible operation pairs, with fixed hyperparameters (n=3, M=10, NV-Embed-v2) that are not fitted to WMT scores. The WMT 2024 comparison is an external validation: Table 3 and Appendix I report Pearson correlations between independently computed consistency scores and CometKiwi/AutoRank/MetricX, and no parameter of ConsistencyChecker is estimated from those metrics. There are no load-bearing self-citations (the reference list contains no works by the present authors), no imported uniqueness theorem, and no ansatz smuggled in via citation; round-trip translation is properly credited to prior external work. Claims such as 'our consistency scores... correlate strongly with WMT 2024 auto-ranking' are empirical correlations, not identities by construction. The reviewer's concerns about n=6, outlier leverage, and undocumented failure handling are soundness or robustness criticisms, not circularity. Hence no circular step can be exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method depends on hand-set design choices rather than fitted parameters: n=3, M=10, out-degree 3, temperature 0.6, and the similarity model. The main unproven inputs are domain assumptions: reversible transformation pairs isolate consistency, embedding similarity captures semantic preservation, and evaluator-generated roots are fair across models. No new physical entities are introduced.

free parameters (5)

Path length n = 3
Selected in Section 4 as the consistency depth for the main score; no held-out search is reported.
Number of trees M = 10
Forest size in Section 6; chosen for cost and diversity, not derived.
Out-degree k (operation pairs) = 3 for main experiments, 1 for English-language WMT pairs
Number of inverse operation pairs per node; varied across experiments in Appendix F.
Generation temperature = 0.6
All LLM generations use temperature 0.6 (Appendix F); affects variance of all scores.
Similarity metric and embedding model = NV-Embed-v2 cosine similarity, with BLEU as ablative alternative
The choice of similarity measure determines the consistency values; no independent human calibration is provided.

assumptions (5)

domain assumption Inverse operation pairs exist and approximately restore the input under an ideal model, f'_p(f_p(c)) ≈ c.
Section 3.1, Eq. (1)-(2); the interpretation of score drops as inconsistency assumes deviations from identity reflect actual semantic or functional drift.
domain assumption Embedding cosine similarity or BLEU monotonically reflects semantic preservation.
Section 4 and Discussion; the authors themselves acknowledge in Limitations that automated metrics may not fully reflect human judgment.
domain assumption Dynamically generated root nodes and operation prompts from the evaluator LLM are equally fair across all evaluatee models.
Section 6; the ablation with two evaluators is reassuring, but a possible self-preference for the Qwen family is not fully ruled out.
domain assumption For code, execution outputs on shared test inputs fully determine functional equivalence.
Sections 4 and 5; the generation of the 20 test inputs is not detailed, and the handling of syntax errors, timeouts, or non-deterministic programs is not specified.
standard math Standard graph-theoretic definitions of tree, path, and forest.
Section 3.2; used without proof and not controversial.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities." pith.science (2026). https://pith.science/paper/7DYUQX7O

@misc{pith2026250612376,
  author       = {Pith},
  title        = {Pith review of: ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DYUQX7O}},
  note         = {Machine review of arXiv:2506.12376}
}

read the original abstract

Evaluating consistency in large language models (LLMs) is crucial for ensuring reliability, particularly in complex, multi-step interactions between humans and LLMs. Traditional self-consistency methods often miss subtle semantic changes in natural language and functional shifts in code or equations, which can accumulate over multiple transformations. To address this, we propose ConsistencyChecker, a tree-based evaluation framework designed to measure consistency through sequences of reversible transformations, including machine translation tasks and AI-assisted programming tasks. In our framework, nodes represent distinct text states, while edges correspond to pairs of inverse operations. Dynamic and LLM-generated benchmarks ensure a fair assessment of the model's generalization ability and eliminate benchmark leakage. Consistency is quantified based on similarity across different depths of the transformation tree. Experiments on eight models from various families and sizes show that ConsistencyChecker can distinguish the performance of different models. Notably, our consistency scores-computed entirely without using WMT paired data-correlate strongly (r > 0.7) with WMT 2024 auto-ranking, demonstrating the validity of our benchmark-free approach. Our implementation is available at: https://github.com/ulab-uiuc/consistencychecker.

Figures

Figures reproduced from arXiv: 2506.12376 by the authors.

**Figure 1.** Overview of the ConsistencyChecker. It shows a self-consistency tree for evaluating LLMs on machine translations. The root node (r) is the initial English sentence generated by the evaluator. Subsequent nodes (vi) are produced by the evaluatee using transformation pairs, such as English→Spanish→English (f1, f′ 1 ) and English→German→English (f2, f′ 2 ). The evaluation framework supports multilingual translations (… view at source ↗

**Figure 2.** Key concepts in the self-consistency tree (operation, node, and edge). It provides a concrete example of the self-consistency tree in the AI-assisted programming task. The node v contains a function that returns the product of two positive integers, and I is its set of inputs. The prompt p asks the evaluatee LLM to rewrite the code in node v to use a looped sum instead of just multiplying, while p ′ asks the same LL… view at source ↗

**Figure 3.** Ablation study on path lengths. For most models and for both translation and programming tasks, the consistency score becomes smaller when the path becomes longer. while LLaMA-3.1-70B outperforms LLaMA-3.1- 8B by 38.2%. These results underscore the importance of model scale for robust performance on complex transformation tasks. ConsistencyChecker reaches high correlation with metrics in WMT 2024 without using its … view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Comparison of two self-consistency trees with different performances. This diagram illustrates how Qwen-2.5-7B (left) and Qwen-2.5-72B (right) performed under the same root node and inverse operation pairs, in machine translation tasks. The diagram shows a simplified …

**Figure 5.** Figure 5: Correlation between ConsistencyChecker evaluation and metrics in WMT 2024 benchmarks. The three figures on the left column from top to bottom correspond to the Czech-Ukrainian, English-German, and English-Japanese language pairs. The two figures on the right column fro…

**Figure 6.** Figure 6: Example of root nodes and leaf nodes in a self-consistency tree. from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of modern financial syst…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages

[1]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

arXiv 2024
[2]

Anthropic. 2024. https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf The claude 3 model family: Opus, sonnet, haiku . In Anthropic Model Card

work page 2024
[3]

Lo \"i c Barrault, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias M \"u ller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. https://doi.org/10.18653/v1/W19-5301 Findings of the 2019 conference on machine translation ( WMT 19...

doi:10.18653/v1/w19-5301 2019
[4]

Clark Barrett and Cesare Tinelli. 2018. Satisfiability modulo theories. Handbook of model checking, pages 305--343

work page 2018
[5]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

arXiv 2020
[6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021
[7]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

work page 2024
[8]

Edmund M Clarke, Orna Grumberg, and Doron Peled. 1999. Model checking. MIT press

work page 1999

Show all 35 references

[9]

Malin, Sricharan Kumar, and Jiaxin Zhang

Wendi Cui, Zhuohang Li, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar, and Jiaxin Zhang. 2024. https://doi.org/10.18653/v1/2024.emnlp-industry.25 Divide-conquer-reasoning for consistency evaluation and automatic improvement of large language models . In Proceed...

2024 doi
[10]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[11]

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=sD93GOzH3i5 Measuring coding challenge competence with APPS . In Thirty-fifth Co...

2021
[12]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[13]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024 arXiv
[14]

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024 b . A survey on large language models for code generation. arXiv preprint arXiv:2406.00515

2024 arXiv
[15]

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. Metricx-24: The google submission to the wmt 2024 metrics shared task. arXiv preprint arXiv:2410.03983

2024 arXiv
[16]

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. 2024. Preliminary wmt24 ranking of general mt systems and llms. arXiv preprint arXiv:2407.19884

2024 arXiv
[17]

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. https://arxiv.org/abs/2401.16745 Mt-eval: A multi-turn capabilities evaluation benchmark for large language models . Preprint, arXiv:2401.16745

2024 arXiv
[18]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

2004
[19]

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

2021
[20]

Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2023. Beyond accuracy: Evaluating self-consistency of code large language models with identitychain. arXiv preprint arXiv:2310.14053

2023 arXiv
[21]

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. https://arxiv.org/abs/2402.06196 Large language models: A survey . Preprint, arXiv:2402.06196

2024 arXiv
[22]

Tobias Nipkow, Markus Wenzel, and Lawrence C. Paulson. 2002. Isabelle/HOL: a proof assistant for higher-order logic. Springer-Verlag, Berlin, Heidelberg

2002
[23]

OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

2024 arXiv
[24]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

2002
[25]

Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G

Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André F. T. Martins. 2023. https://arxiv.org/abs/2309.11925 Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task . Preprint, ...

2023 arXiv
[26]

Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. https://aclanthology.org/2006.amta-papers.25/ A study of translation edit rate with targeted human annotation . In Proceedings of the 7th Conference of the Association for Machine Translation...

2006
[27]

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, A...

2024 arXiv
[28]

Menno van Zaanen and Simon Zwarts. 2006. https://doi.org/10.1007/11941439_149 Unsupervised measurement of translation quality using multi-engine, bi-directional translation . In Proceedings of the 19th Australian Joint Conference on Artificial Intelligence: Advances in Artific...

2006 doi
[29]

Gomez, ukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000–6010, ...

2017
[30]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

2022 arXiv
[31]

Wong, and Rui Wang

Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. 2024. https://arxiv.org/abs/2410.13640 Latent space chain-of-embedding enables output-free llm self-evaluation . Preprint, arXiv:2410.13640

2024 arXiv
[32]

Warren Weaver. 1952. Translation. In Proceedings of the Conference on Mechanical Translation

1952
[33]

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674

2023 arXiv
[34]

Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. https://arxiv.org/abs/2404.18824 Benchmarking benchmark leakage in large language models . Preprint, arXiv:2404.18824

2024 arXiv
[35]

Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. 2025. https://arxiv.org/abs/2502.06215 Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks ...

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

arXiv 2024

[2] [2]

Anthropic. 2024. https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf The claude 3 model family: Opus, sonnet, haiku . In Anthropic Model Card

work page 2024

[3] [3]

Lo \"i c Barrault, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias M \"u ller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. https://doi.org/10.18653/v1/W19-5301 Findings of the 2019 conference on machine translation ( WMT 19...

doi:10.18653/v1/w19-5301 2019

[4] [4]

Clark Barrett and Cesare Tinelli. 2018. Satisfiability modulo theories. Handbook of model checking, pages 305--343

work page 2018

[5] [5]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

arXiv 2020

[6] [6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021

[7] [7]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

work page 2024

[8] [8]

Edmund M Clarke, Orna Grumberg, and Doron Peled. 1999. Model checking. MIT press

work page 1999