SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation

Jiao Sun; Nanyun Peng; Yufei Tian; Zizhao Zhang

REVIEW 4 major objections 5 minor 33 references

SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SkillVerse turns LLM critiques into a hierarchical skill tree that improves in-context learning by 25% and predicts model weaknesses with 55% success.

desk verdict A genuinely new hierarchical evaluation framework with solid clustering validation, but the unvalidated parser and a sign-flipped headline claim mean the downstream numbers should not be trusted yet. read the letter →

arxiv 2506.00319 v1 pith:7OPC42PY submitted 2025-05-31 cs.CL

Yufei Tian , Jiao Sun , Nanyun Peng , Zizhao Zhang This is my paper

classification cs.CL

keywords LLMevaluationdendrogramatomicjudgmentin-contextlearningweaknesspredictionhierarchicalclusteringLLM-as-judgeskilldiagnosis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SkillVerse, an unsupervised framework that turns free-form LLM critiques of model responses into a hierarchical 'dendrogram' of atomic judgments. Each atomic judgment is a single, non-decomposable statement of the form model + succeeded/partially succeeded/failed + specific task, and the dendrogram organizes these by semantic similarity, so the tree can be sliced at any granularity to yield skills with computed proficiency scores. The authors claim this structure gives accurate, actionable estimates of model strengths and weaknesses, and they demonstrate two uses: a tree-search that selects contrastive few-shot demonstrations and improves in-context learning by 25% relative to C-ICL, and an auto-discovery pipeline in which a reasoning LLM reads the proficiency report and hypothesizes weakness scenarios, achieving a 55% success rate on those tasks versus 77% for uninformed guesses. The point is to show that structured, interpretable skill diagnosis from unstructured traffic data is feasible and immediately useful for model development.

What carries the argument

The machinery is the atomic judgment and the dendrogram. An atomic judgment is a strict triple, subject (model), verb (succeed, partially succeed, or fail), object (a specific task such as 'identify cities in Eastern Europe'), that the framework parses from free-form critiques. Only the object is embedded, using Google's Text Embedding API, and agglomerative clustering builds a dendrogram from semantic distances; horizontal slices at chosen thresholds produce nested skill clusters, and each cluster's proficiency is the ratio of positive atomic judgments. The same tree powers two new algorithms: a tree-search that prunes branches where the target model already succeeds (success rate above a threshold) and re-ranks the remaining contrastive examples by relevance and benefit, and a weakness-extrapolation pipeline that feeds a proficiency report to a separate reasoner. The dendrogram's key work is to make the critiques searchable and generalizable, bringing together semantically similar tasks and providing a compact, granular capability report.

What would settle it

Manually inspect a random sample of 200 atomic judgments and compare them against a human-annotated gold parse of the corresponding critiques, measuring agreement; if parser accuracy is low, the dendrogram and all downstream claims collapse. Alternatively, run SkillVerse with human-written critiques on the same response pairs and check whether the cluster structure and the reported 25% in-context learning improvement persist.

Watch

Extended reading notes

Core claim

The central claim is that the dendrogram of atomic judgments faithfully represents model capabilities, and that this representation is not just descriptive but operative: it can be searched to pick few-shot demonstrations that are both relevant and challenging, and it can be digested by a strong reasoner to extrapolate to unseen failure modes. The paper validates the clustering against human similarity judgments (Pearson correlation 0.643, true positive rate 0.916, true negative rate 0.88) and the anchoring step against human gold merges (precision 0.926, recall 0.980). Downstream, the tree-search selection achieves a 25% relative improvement over the C-ICL baseline across GPT-4o, Gemini-1.5-pro, and Gemini-1.5-flash on IFEval and ChatbotArena, and the reasoner's informed weakness hypotheses hit only 55% success, 22% lower than the uninformed baseline, meaning the predicted tasks are indeed difficult for the target model. The paper also reports that SkillVerse identifies inverse-scaling capabilities, where larger models underperform smaller ones, and predicts such cases with an average scaling gain of 0.5% versus 10.6% for uninformed predictions.

Load-bearing premise

The whole framework assumes the LLM judge's free-form critiques, after parsing into atomic judgments, accurately reflect what the model actually did wrong or right; the paper never validates this parser.

Editorial extensions

If this is right

SkillVerse's tree-search demonstration selection yields a 25% relative improvement over standard contrastive in-context learning on IFEval and ChatbotArena across three models.
The reasoner with SkillVerse's proficiency report predicts weaknesses that are 22% more challenging than uninformed predictions, with success rates of 55% versus 77% on those tasks.
The framework identifies inverse-scaling capabilities, where larger models underperform smaller ones, with an average scaling gain of only 0.5% on predicted tasks compared to 10.6% without performance data.
SkillVerse reveals fine-grained strengths and weaknesses that leaderboards mask, for instance GPT-4-turbo outperforming GPT-4o on SQL queries, file processing, and music tasks despite being an older release.
The dendrogram can be sliced at arbitrary granularity, so the same diagnosis serves both coarse-grained and fine-grained analyses, from broad categories like coding down to specific tasks like writing a riddle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension of the demonstration-selection logic is model routing: once a new prompt is located in the dendrogram, it could be sent to the model whose cluster proficiency is highest, turning the skill map into an inference-time dispatcher.
The inverse-scaling findings suggest a testable hypothesis that strict-format and constraint-heavy tasks form a recurring failure family across model series trained on similar data, so the dendrogram could serve as an early-warning system for such tasks.
The causal role of the hierarchy itself could be isolated by ablating the proficiency report to a flat, non-hierarchical skill list and measuring whether the weakness-prediction gain disappears; the paper does not run this ablation.
A robustness check worth trying is to run the full pipeline with human-written critiques on a small response set and compare the resulting dendrogram and downstream gains, which would separate the framework's contribution from the judge model's reliability.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The new thing here is real: unsupervised agglomerative clustering of LLM critiques parsed into atomic judgments (subject-verb-object) to build a dendrogram, with controlled granularity via slicing. That combination is not in the flat-clustering or predefined-attribute prior work they cite, and the two applications—dendrogram-guided ICL and weakness extrapolation—are sensible extensions. I also give them credit for validating the clustering and anchoring steps with human annotations: Pearson 0.643 on similarity judgments, TPR/TNR around 0.9, and anchoring precision/recall near 0.93/0.98. The checkable-rubric supplement for format and calculation tasks is a pragmatic touch. The inverse-scaling observations are interesting, if not deeply analyzed.

But there are three real soft spots, in increasing order of concern. First, the paper explicitly says the only source of error is the clustering, which is wrong: the free-form critiques are parsed into atomic judgments with no precision/recall or human agreement reported for that parser. Everything downstream—the dendrogram, the proficiency ratios, the ICL gains, the weakness predictions—inherits whatever errors the parser makes. That is load-bearing. Second, the abstract says the 55% weakness-prediction success rate is 22% higher than baseline, while the body and Figure 8 say it is 22% lower. A sign flip on a headline number is not a typo; it changes the conclusion. Third, they cite Yan et al. (2021) for C-ICL, but that paper is about zero-shot object detection; the actual contrastive in-context learning work is Mo et al. (2024). That mis-citation makes me doubt the care with which other baselines are handled.

The weakness-prediction experiment also uses a small post-hoc-selected task set, and the inter-critic agreement of 0.65 is moderate. These are fixable with more data and a parser validation. The core framework is still plausible and worth a referee's time—it is a within-subfield advance, not a paradigm shift, and the missing parser check is a clear gap rather than a dead end. I would send it to peer review, with the expectation that the authors add parser validation, fix the contradiction, and correct the citation.

Who is this for? People working on LLM evaluation, model debugging, or few-shot selection. It deserves a serious referee, but not unconditional acceptance as-is.

Referee Report

4 major / 5 minor

Summary. SkillVerse constructs an unsupervised hierarchical skill tree from LLM critiques: free-form critiques are parsed into atomic judgments (Subject+Verb+Object), embedded, agglomeratively clustered, and sliced at multiple granularities to yield model proficiency estimates. The framework is evaluated via human similarity judgments of cluster structure and human-verified anchoring of independently built trees. Two downstream applications are reported: tree-guided selection of contrastive few-shot demonstrations, claimed to yield a 25% relative improvement over C-ICL, and LLM-generated weakness hypotheses for unseen tasks, where informed predictions achieve a 55% success rate versus 77% for uninformed guesses, with a complementary inverse-scaling prediction result.

Significance. If the central pipeline is reliable, SkillVerse offers a useful alternative to benchmark scores: it provides granular, interpretable model diagnostics with user-controlled granularity. The paper's strongest evidence is the human validation of the clustering (Pearson r=0.643 on similarity ratings; TPR=0.916 and TNR=0.883) and of the anchoring procedure (precision 0.926, recall 0.980). The two applications—ICL demonstration selection and weakness prediction—are concrete and falsifiable, and the inverse-scaling findings are interesting. However, the framework's input is free-form LLM critiques, and the atomic-judgment parsing step is unvalidated, so the overall significance of the framework depends on adding that missing evidence.

major comments (4)

[§2.3 and §3.2] Section 2.3 introduces atomic judgments (Subject+Verb+Object) extracted from free-form critiques, and Section 3.2 states that "the only potential source of error in our framework arises from the unsupervised clustering process." This is not accurate: the parsing step is itself a potential error source, and no precision, recall, or human agreement is reported for it. Since the dendrogram, proficiency ratios, ICL demonstration selection, and weakness hypotheses are all computed from these parsed judgments, an error in this step would propagate to every downstream claim. Please validate the parser against human-annotated atomic judgments, or otherwise bound its error rate.
[Abstract and §5.3] The abstract states that SkillVerse predicts weaknesses with a 55% success rate, "22% higher than without SkillVerse." In Section 5.3 and Figure 8, however, the informed hypothesis success rate is 0.55, 22% lower than the uninformed guess of 0.77. Because a lower success rate on hypothesized weakness tasks is the intended demonstration of predictive ability, the abstract's direction is incorrect and must be fixed.
[§5.2–§5.3] Section 5.3 draws strong conclusions from a small, filtered set: after the "practical relevance" re-ranking, only 10 tasks remain for the inverse-scaling analysis (Figure 9), and the KS test in Table 3 is based on 5 rows. The 0.5% versus 10.6% average performance-gap comparison in Tables 4–5 is reported without per-task variance or confidence intervals. Please report the full selected task lists, the exact filtering rule, and uncertainty measures so readers can assess whether the claimed effect is robust.
[§4.2 and Figure 6] The 25% relative improvement over C-ICL is not defined in Section 4.2. Figure 6 plots improvement relative to direct generation, not relative to C-ICL, and the per-condition improvements vary widely, from about 6.7% (GPT-4o, Chatbot Arena) to 40% (Gemini-1.5-Flash, IFEval). State the aggregation rule (e.g., average over all conditions) and report per-condition values with error bars, since the headline number depends on how the comparison is aggregated.

minor comments (5)

[References and §4.1] The citation for C-ICL is incorrect: the text cites Yan et al. (2021), but the reference list entry is a zero-shot object detection paper. The contrastive in-context learning method appears to correspond to Mo et al. (2024), which is already cited in the related work; please fix the citation.
[§2.3] There is a typo in Section 2.3: "serve as act as the building blocks" should read "serve as the building blocks."
[§4] Section 4's introduction contains "how the the uncovered model proficiency" with a duplicated article; please proofread the manuscript for similar issues.
[§2.3 and Figure 2] "Easter Europe" appears in Figure 2 and Section 2.3 and should be "Eastern Europe"; also, "scaler score" in §4.1 should be "scalar score."
[§3.2] Section 3.2 reports inter-annotator agreement as Pearson 0.88, but the main text says there were 1,590 annotations with 3 annotators; please specify whether this is a pairwise average or another statistic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the dendrogram is definitional, and the ICL and weakness-prediction claims are tested on held-out tasks.

full rationale

SkillVerse's core construction is definitional rather than derivational: an atomic judgment is defined as a Subject+Verb+Object claim extracted from critiques, and a skill cluster's proficiency is defined as the ratio of positive atomic judgments in that cluster. These definitions do not make a predicted quantity equal to an input by construction. The two downstream claims are evaluated out-of-sample. For in-context learning, demonstrations are selected by tree search using content relevance and the benefit C(r1)-C(r2) (Section 4.1), but the reported 25% relative improvement is measured on 150 held-out prompts per dataset (Section 4.2) and compared against C-ICL and similarity-only baselines under the same evaluation; the benefit score is not a parameter fitted to the held-out evaluation set. For weakness prediction, a reasoning LLM receives the proficiency report, hypothesizes new tasks, humans curate prompts for those tasks, and success rates are then measured on the newly collected prompts (Sections 5.1-5.2); the uninformed baseline isolates the contribution of the proficiency data. The predictions are therefore not the same objects as the training judgments. The paper's stated assumption that 'the critiques are reliable' (Section 3.2) and the Limitation paragraph acknowledging that LLM judges 'might introduce errors' are genuine missing-validation and correctness risks, especially because the atomic-judgment parser is never precision/recall-checked, but they are not circularity: nothing in the paper equates an output to an input by construction. There are no load-bearing self-citations or imported uniqueness theorems; the cited related work is external. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The framework's central output is a dendrogram built from LLM critiques parsed into atomic judgments. The main free parameters are clustering and anchoring thresholds. The key assumptions are the reliability of the judge and the accuracy of parsing, both of which the paper acknowledges but does not independently verify.

free parameters (4)

tau (anchor centroid threshold) = not specified
Grid-searched on validation set in the anchoring procedure (Section 2.4).
epsilon (anchor overlap threshold) = not specified
Grid-searched on validation set in the anchoring procedure (Section 2.4).
T (pruning threshold for ICL skill selection) = not specified
Used in Step 2 of the few-shot selection method (Section 4.1) to prune easy branches; value not given.
dendrogram slicing level = not specified
Chosen via elbow method to determine number of clusters (Section 3.1).

assumptions (3)

domain assumption LLM critiques are reliable indicators of model errors
Invoked throughout; admitted as a limitation in the Limitation section and in Section 3.2 the paper assumes critiques are reliable.
domain assumption Atomic judgments can be accurately parsed from free-form critiques
Section 2.3 describes the parsing but provides no validation of parser accuracy.
domain assumption Text embedding similarity corresponds to task-skill similarity
Used for clustering (Section 2.3); partially validated via human annotation (Pearson 0.643), but this is a moderate correlation.

invented entities (1)

atomic judgment
purpose: A structured triplet (subject, verb, object) representing one indivisible critique, used as the clustering unit
Introduced in Section 2.3; no external falsifiable handle beyond the paper's own examples.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation." pith.science (2026). https://pith.science/paper/7OPC42PY

@misc{pith2026250600319,
  author       = {Pith},
  title        = {Pith review of: SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OPC42PY}},
  note         = {Machine review of arXiv:2506.00319}
}

read the original abstract

As language models evolve to tackle complex, multifaceted tasks, their evaluation must adapt to capture this intricacy. A granular, skill-specific understanding of model capabilities can empower researchers to make informed model development plans. In this paper, we introduce SkillVerse, an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities. With LLM as a judge, SkillVerse first critiques the model responses, and then organizes them into a hierarchical structure termed dendrogram. Given proficiency at arbitrary levels of granularity, SkillVerse is flexible to produce insights of behaviors of modern large models. We also demonstrate its efficacy in two downstream tasks: 1) improving model in-context learning by 25% using a tree-search algorithm to select more informative few-shot demonstrations, and 2) accurately predicting new model weaknesses with a 55% success rate, 22% higher than without SkillVerse.

Figures

Figures reproduced from arXiv: 2506.00319 by the authors.

**Figure 1.** Up: The input-output flow of SKILLVERSE. Skill-specific critiques are extracted, structured into a dendrogram, and sliced at varying granularities to reveal nested clusters of skills and model proficiency. Bottom: Versatile applications of SKILLVERSE, from selecting informative few-shot demonstrations to uncovering hidden model weaknesses. requires manual inspections, which is both timeconsuming and costly. These … view at source ↗

**Figure 2.** The overall framework of SKILLVERSE. A set of critiques on model responses is parsed into atomic judgments and organized using bottom-up clustering into a dendrogram, which is then unfolded at varying levels of granularity to form nested clusters, allowing for detailed analysis of model proficiency. Thanks to the hierarchical structure, this novel pipeline is highly flexible in interpreting model capabilities. VERSE… view at source ↗

**Figure 3.** Upper right: A dendrogram produced by SKILLVERSE on a combination of two datasets: ChatbotArena (Chiang et al., 2024) and IF Eval (Zhou et al., 2023). Main figure: Multiple layers of nested clusters that represent model proficiencies from coarse-grained to fine-grained, by horizontally slicing the dendrogram at different levels. For each group, we can then calculate skill-level model proficiency based on the atomic … view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Comparing the strengths and weaknesses of [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: The dendrogram produced by SKILLVERSE helps to selects more informative few-shot demonstrations by considering both relevance and challenges posed to the target model. In contrast, previous methods selected ‘...output a Python script in OOP for a banditinspired appro…

**Figure 6.** Figure 6: Improvement of different in-context learning [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: The process to extrapolate to model deficien [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: The performance of Gemini-1.5-flash under [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 10.** Figure 10: An example of the Gemini-1.5-flash’s capa [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 11.** Figure 11: The capability report comparing Gemini-1.5-pro, Claude3.5-Sonnet, and GPT-4o on ChatbotArena [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

**Figure 12.** Figure 12: Figure [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

**Figure 14.** Figure 14: Annotation Guideline: Phrase Similarity Annotation Instructions Page 2 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 14 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Introducing claude 3.5 sonnet . Accessed: 2024-12-14

work page 2024
[4]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

2024
[5]

Liang Chao, Wei Xiang, and Bang Wang. 2024. In-context contrastive learning for event causality identification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 868--881

work page 2024
[7]

Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023 b . https://arxiv.org/abs/2311.09277 Contrastive chain-of-thought prompting . Preprint, arXiv:2311.09277

arXiv 2023
[8]

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In International Conference on Machine Learning, pages 8359--8388. PMLR

work page 2024
[9]

Google. 2024 a . https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . arXiv preprint arXiv:2403.05530

arXiv 2024

Show all 33 references

[10]

Google. 2024 b . Text embeddings api. https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings. Accessed: 2024-12-14

2024
[11]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[12]

Xiaonan Jing, Srinivas Billa, and Danny Godbout. 2024. On a scale from 1 to 5: Quantifying hallucination in faithfulness evaluation. arXiv preprint arXiv:2410.12222

2024 arXiv
[13]

Akira Kawabata and Saku Sugawara. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.905 Rationale-aware answer verification by pairwise self-evaluation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16178--16196, Miami, Flori...

2024 doi
[14]

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939

2024 arXiv
[15]

Hashimoto

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

2023
[16]

Adian Liusie, Potsawee Manakul, and Mark Gales. 2024 a . https://aclanthology.org/2024.eacl-long.8 LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models . In Proceedings of the 18th Conference of the European Chapter of t...

2024
[17]

Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark Gales. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.389 Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons . In Proceedings of the 2024 Conference on Empirical Methods in ...

2024 doi
[18]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceed...

2023
[19]

Inverse scaling: When bigger isn't better

Ian R McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, et al. Inverse scaling: When bigger isn't better. Transactions on Machine Learning Research
[20]

Ying Mo, Jiahao Liu, Jian Yang, Qifan Wang, Shun Zhang, Jingang Wang, and Zhoujun Li. 2024. C-icl: contrastive in-context learning for information extraction. arXiv preprint arXiv:2402.11254

2024 arXiv
[21]

Mazda Moayeri, Vidhisha Balachandran, Varun Chandrasekaran, Safoora Yousefi, Thomas Fel, Soheil Feizi, Besmira Nushi, Neel Joshi, and Vibhav Vineet. 2024. Unearthing skill-level insights for understanding trade-offs of foundation models. arXiv preprint arXiv:2410.13826

2024 arXiv
[22]

Vishvak Murahari, Ameet Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, and Ashwin Kalyan. 2024. Qualeval: Qualitative evaluation for model improvement. In Proceedings of the 2024 Conference of the North American Chapter of the Association for ...

2024
[23]

Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, Vu Le, Nick McKenna, Carina Suzana Negreanu, Chris Parnin, and Advait Sarkar. 2025. Evaluating the evaluator: Measuring llms’ adherence to task evaluation instructions. In Proceedings of the AAAI Conference on Artificial ...

2025
[24]

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. https://doi.org/10.1162/tacl_a_00660 Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies . Transactions of the Associ...

2024 doi
[25]

Llm-as-a-judge & reward model: What they can and cannot do

Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do
[26]

Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pag...

2024
[27]

Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024. Self-taught evaluators. arXiv preprint arXiv:2408.02666

2024 arXiv
[28]

Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04592

2023 arXiv
[29]

Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.92 LLMR efine: Pinpointing and refining large language models via fine-grained actionable f...

2024 doi
[30]

Caixia Yan, Xiao Chang, Minnan Luo, Huan Liu, Xiaoqin Zhang, and Qinghua Zheng. 2021. https://api.semanticscholar.org/CorpusID:237491964 Semantics-guided contrastive network for zero-shot object detection . IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:1530--1544

2021
[31]

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020

2024 arXiv
[32]

Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. 2024. In-context principle learning from mistakes. arXiv preprint arXiv:2402.05403

2024 arXiv
[33]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

2023
[34]

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Introducing claude 3.5 sonnet . Accessed: 2024-12-14

work page 2024

[4] [4]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

2024

[5] [5]

Liang Chao, Wei Xiang, and Bang Wang. 2024. In-context contrastive learning for event causality identification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 868--881

work page 2024

[6] [7]

Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023 b . https://arxiv.org/abs/2311.09277 Contrastive chain-of-thought prompting . Preprint, arXiv:2311.09277

arXiv 2023

[7] [8]

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In International Conference on Machine Learning, pages 8359--8388. PMLR

work page 2024

[8] [9]

Google. 2024 a . https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . arXiv preprint arXiv:2403.05530

arXiv 2024