{"id":"71c3caee-9c20-4233-a083-c8b2e0a74d2e","arxiv_id":"2506.00319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SkillVerse clusters LLM critiques into a dendrogram of skills and uses it to improve in-context learning and predict unseen model weaknesses.","lead":"SkillVerse is a system that turns LLM judge critiques into a hierarchical skill tree, revealing a model's granular strengths and weaknesses. The paper reports that this tree improves few-shot example selection by 25% and predicts where models will fail better than an uninformed baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim inherits a single unvalidated component: free-form critiques are parsed into atomic judgments with no accuracy check, so the dendrogram and all downstream numbers may rest on corrupted skill labels.","rationale":"The paper's main contribution is a hierarchy built on atomic judgments; if those are unreliable, all downstream claims lose grounding. The clustering evaluation cannot compensate because it measures pairwise similarity of extracted objects, not whether the extraction is correct. The reader's weakest_assumption identifies exactly this gap. We agree with the conditional verdict: the framework is plausible and the clustering validation is real, but acceptance should be contingent on parser validation or on showing downstream results are insensitive to parser errors.","tokens_in":17498,"tokens_out":8537,"duration_ms":80479,"concrete_test":"Take 200 critiques from the pipeline. Have two annotators independently produce gold atomic judgments (model, success/fail, task object) following the paper's syntax, resolve disagreements, and measure parser precision/recall/F1. Then recompute proficiency for one dendrogram slice and re-run the ICL selection (Section 4) and weakness prediction (Section 5) on a held-out subset using only gold atomic judgments. If parser F1 is below, say, 0.90, or if the 25% relative gain and 22% gap move by more than a few points, the reported headline numbers are not robust without a validated parser.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SkillVerse's core output is a dendrogram built from atomic judgments (Subject + Verb + Object) extracted from LLM critiques. Section 3.2 states: Assuming the critiques are reliable and given that the success rates are algorithmically calculated, the only potential source of error in our framework arises from the unsupervised clustering process. This explicitly skips the parser. The paper validates clustering and anchoring with human annotations (Pearson 0.643; TPR 0.916 / TNR 0.883; anchoring precision 0.926 / recall 0.980), but no precision, recall, or human agreement is reported for atomic-judgment extraction itself. The only critique-model reliability evidence is a Pearson 0.65 agreement between Gemini-1.5-Pro and GPT-4o on 1,000 pairs, which is moderate and does not establish that extracted judgments match ground truth. If the parser splits one critique into multiple claims, merges distinct claims, or assigns the wrong object or verb, the embeddings, cluster membership, and proficiency ratios are all corrupted. Because the 25% ICL improvement and the 55% vs. 77% weakness-prediction results are computed from the same tree, this single unvalidated step is load-bearing. This is a missing-support concern, not an internal contradiction: the error could be small, but the paper provides no evidence either way.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SkillVerse constructs an unsupervised hierarchical skill tree from LLM critiques: free-form critiques are parsed into atomic judgments (Subject+Verb+Object), embedded, agglomeratively clustered, and sliced at multiple granularities to yield model proficiency estimates. The framework is evaluated via human similarity judgments of cluster structure and human-verified anchoring of independently built trees. Two downstream applications are reported: tree-guided selection of contrastive few-shot demonstrations, claimed to yield a 25% relative improvement over C-ICL, and LLM-generated weakness hypotheses for unseen tasks, where informed predictions achieve a 55% success rate versus 77% for uninformed guesses, with a complementary inverse-scaling prediction result.","tokens_in":17809,"tokens_out":5340,"duration_ms":45850,"significance":"If the central pipeline is reliable, SkillVerse offers a useful alternative to benchmark scores: it provides granular, interpretable model diagnostics with user-controlled granularity. The paper's strongest evidence is the human validation of the clustering (Pearson r=0.643 on similarity ratings; TPR=0.916 and TNR=0.883) and of the anchoring procedure (precision 0.926, recall 0.980). The two applications—ICL demonstration selection and weakness prediction—are concrete and falsifiable, and the inverse-scaling findings are interesting. However, the framework's input is free-form LLM critiques, and the atomic-judgment parsing step is unvalidated, so the overall significance of the framework depends on adding that missing evidence.","major_comments":[{"comment":"Section 2.3 introduces atomic judgments (Subject+Verb+Object) extracted from free-form critiques, and Section 3.2 states that \"the only potential source of error in our framework arises from the unsupervised clustering process.\" This is not accurate: the parsing step is itself a potential error source, and no precision, recall, or human agreement is reported for it. Since the dendrogram, proficiency ratios, ICL demonstration selection, and weakness hypotheses are all computed from these parsed judgments, an error in this step would propagate to every downstream claim. Please validate the parser against human-annotated atomic judgments, or otherwise bound its error rate.","section":"§2.3 and §3.2"},{"comment":"The abstract states that SkillVerse predicts weaknesses with a 55% success rate, \"22% higher than without SkillVerse.\" In Section 5.3 and Figure 8, however, the informed hypothesis success rate is 0.55, 22% lower than the uninformed guess of 0.77. Because a lower success rate on hypothesized weakness tasks is the intended demonstration of predictive ability, the abstract's direction is incorrect and must be fixed.","section":"Abstract and §5.3"},{"comment":"Section 5.3 draws strong conclusions from a small, filtered set: after the \"practical relevance\" re-ranking, only 10 tasks remain for the inverse-scaling analysis (Figure 9), and the KS test in Table 3 is based on 5 rows. The 0.5% versus 10.6% average performance-gap comparison in Tables 4–5 is reported without per-task variance or confidence intervals. Please report the full selected task lists, the exact filtering rule, and uncertainty measures so readers can assess whether the claimed effect is robust.","section":"§5.2–§5.3"},{"comment":"The 25% relative improvement over C-ICL is not defined in Section 4.2. Figure 6 plots improvement relative to direct generation, not relative to C-ICL, and the per-condition improvements vary widely, from about 6.7% (GPT-4o, Chatbot Arena) to 40% (Gemini-1.5-Flash, IFEval). State the aggregation rule (e.g., average over all conditions) and report per-condition values with error bars, since the headline number depends on how the comparison is aggregated.","section":"§4.2 and Figure 6"}],"minor_comments":[{"comment":"The citation for C-ICL is incorrect: the text cites Yan et al. (2021), but the reference list entry is a zero-shot object detection paper. The contrastive in-context learning method appears to correspond to Mo et al. (2024), which is already cited in the related work; please fix the citation.","section":"References and §4.1"},{"comment":"There is a typo in Section 2.3: \"serve as act as the building blocks\" should read \"serve as the building blocks.\"","section":"§2.3"},{"comment":"Section 4's introduction contains \"how the the uncovered model proficiency\" with a duplicated article; please proofread the manuscript for similar issues.","section":"§4"},{"comment":"\"Easter Europe\" appears in Figure 2 and Section 2.3 and should be \"Eastern Europe\"; also, \"scaler score\" in §4.1 should be \"scalar score.\"","section":"§2.3 and Figure 2"},{"comment":"Section 3.2 reports inter-annotator agreement as Pearson 0.88, but the main text says there were 1,590 annotations with 3 annotators; please specify whether this is a pairwise average or another statistic.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with a clearly stated pipeline, but the missing parser validation is a serious gap that the authors should address. The abstract/body direction error and the incorrect C-ICL citation are easy fixes. I believe the manuscript can be made acceptable with a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: unsupervised agglomerative clustering of LLM critiques parsed into atomic judgments (subject-verb-object) to build a dendrogram, with controlled granularity via slicing. That combination is not in the flat-clustering or predefined-attribute prior work they cite, and the two applications—dendrogram-guided ICL and weakness extrapolation—are sensible extensions. I also give them credit for validating the clustering and anchoring steps with human annotations: Pearson 0.643 on similarity judgments, TPR/TNR around 0.9, and anchoring precision/recall near 0.93/0.98. The checkable-rubric supplement for format and calculation tasks is a pragmatic touch. The inverse-scaling observations are interesting, if not deeply analyzed.\n\nBut there are three real soft spots, in increasing order of concern. First, the paper explicitly says the only source of error is the clustering, which is wrong: the free-form critiques are parsed into atomic judgments with no precision/recall or human agreement reported for that parser. Everything downstream—the dendrogram, the proficiency ratios, the ICL gains, the weakness predictions—inherits whatever errors the parser makes. That is load-bearing. Second, the abstract says the 55% weakness-prediction success rate is 22% higher than baseline, while the body and Figure 8 say it is 22% lower. A sign flip on a headline number is not a typo; it changes the conclusion. Third, they cite Yan et al. (2021) for C-ICL, but that paper is about zero-shot object detection; the actual contrastive in-context learning work is Mo et al. (2024). That mis-citation makes me doubt the care with which other baselines are handled.\n\nThe weakness-prediction experiment also uses a small post-hoc-selected task set, and the inter-critic agreement of 0.65 is moderate. These are fixable with more data and a parser validation. The core framework is still plausible and worth a referee's time—it is a within-subfield advance, not a paradigm shift, and the missing parser check is a clear gap rather than a dead end. I would send it to peer review, with the expectation that the authors add parser validation, fix the contradiction, and correct the citation.\n\nWho is this for? People working on LLM evaluation, model debugging, or few-shot selection. It deserves a serious referee, but not unconditional acceptance as-is.","headline":"A genuinely new hierarchical evaluation framework with solid clustering validation, but the unvalidated parser and a sign-flipped headline claim mean the downstream numbers should not be trusted yet.","tokens_in":18294,"tokens_out":3715,"would_cite":false,"duration_ms":33838,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillVerse turns LLM critiques into a hierarchical skill tree that improves in-context learning by 25% and predicts model weaknesses with 55% success.","keywords":["LLM evaluation","dendrogram","atomic judgment","in-context learning","weakness prediction","hierarchical clustering","LLM-as-judge","skill diagnosis"],"falsifier":"Manually inspect a random sample of 200 atomic judgments and compare them against a human-annotated gold parse of the corresponding critiques, measuring agreement; if parser accuracy is low, the dendrogram and all downstream claims collapse. Alternatively, run SkillVerse with human-written critiques on the same response pairs and check whether the cluster structure and the reported 25% in-context learning improvement persist.","tokens_in":17332,"feed_emoji":"🌳","tokens_out":7030,"duration_ms":59862,"temperature":0.7,"pith_summary":"The paper introduces SkillVerse, an unsupervised framework that turns free-form LLM critiques of model responses into a hierarchical 'dendrogram' of atomic judgments. Each atomic judgment is a single, non-decomposable statement of the form model + succeeded/partially succeeded/failed + specific task, and the dendrogram organizes these by semantic similarity, so the tree can be sliced at any granularity to yield skills with computed proficiency scores. The authors claim this structure gives accurate, actionable estimates of model strengths and weaknesses, and they demonstrate two uses: a tree-search that selects contrastive few-shot demonstrations and improves in-context learning by 25% relative to C-ICL, and an auto-discovery pipeline in which a reasoning LLM reads the proficiency report and hypothesizes weakness scenarios, achieving a 55% success rate on those tasks versus 77% for uninformed guesses. The point is to show that structured, interpretable skill diagnosis from unstructured traffic data is feasible and immediately useful for model development.","feed_headline":"SkillVerse tree predicts LLM weaknesses, lifts few-shot learning 25%","feed_subtitle":"LLM critiques become a dendrogram of atomic judgments that guides demonstration selection and failure prediction.","key_machinery":"The machinery is the atomic judgment and the dendrogram. An atomic judgment is a strict triple, subject (model), verb (succeed, partially succeed, or fail), object (a specific task such as 'identify cities in Eastern Europe'), that the framework parses from free-form critiques. Only the object is embedded, using Google's Text Embedding API, and agglomerative clustering builds a dendrogram from semantic distances; horizontal slices at chosen thresholds produce nested skill clusters, and each cluster's proficiency is the ratio of positive atomic judgments. The same tree powers two new algorithms: a tree-search that prunes branches where the target model already succeeds (success rate above a threshold) and re-ranks the remaining contrastive examples by relevance and benefit, and a weakness-extrapolation pipeline that feeds a proficiency report to a separate reasoner. The dendrogram's key work is to make the critiques searchable and generalizable, bringing together semantically similar tasks and providing a compact, granular capability report.","core_discovery":"The central claim is that the dendrogram of atomic judgments faithfully represents model capabilities, and that this representation is not just descriptive but operative: it can be searched to pick few-shot demonstrations that are both relevant and challenging, and it can be digested by a strong reasoner to extrapolate to unseen failure modes. The paper validates the clustering against human similarity judgments (Pearson correlation 0.643, true positive rate 0.916, true negative rate 0.88) and the anchoring step against human gold merges (precision 0.926, recall 0.980). Downstream, the tree-search selection achieves a 25% relative improvement over the C-ICL baseline across GPT-4o, Gemini-1.5-pro, and Gemini-1.5-flash on IFEval and ChatbotArena, and the reasoner's informed weakness hypotheses hit only 55% success, 22% lower than the uninformed baseline, meaning the predicted tasks are indeed difficult for the target model. The paper also reports that SkillVerse identifies inverse-scaling capabilities, where larger models underperform smaller ones, and predicts such cases with an average scaling gain of 0.5% versus 10.6% for uninformed predictions.","pith_inferences":["A direct extension of the demonstration-selection logic is model routing: once a new prompt is located in the dendrogram, it could be sent to the model whose cluster proficiency is highest, turning the skill map into an inference-time dispatcher.","The inverse-scaling findings suggest a testable hypothesis that strict-format and constraint-heavy tasks form a recurring failure family across model series trained on similar data, so the dendrogram could serve as an early-warning system for such tasks.","The causal role of the hierarchy itself could be isolated by ablating the proficiency report to a flat, non-hierarchical skill list and measuring whether the weakness-prediction gain disappears; the paper does not run this ablation.","A robustness check worth trying is to run the full pipeline with human-written critiques on a small response set and compare the resulting dendrogram and downstream gains, which would separate the framework's contribution from the judge model's reliability."],"forward_implications":["SkillVerse's tree-search demonstration selection yields a 25% relative improvement over standard contrastive in-context learning on IFEval and ChatbotArena across three models.","The reasoner with SkillVerse's proficiency report predicts weaknesses that are 22% more challenging than uninformed predictions, with success rates of 55% versus 77% on those tasks.","The framework identifies inverse-scaling capabilities, where larger models underperform smaller ones, with an average scaling gain of only 0.5% on predicted tasks compared to 10.6% without performance data.","SkillVerse reveals fine-grained strengths and weaknesses that leaderboards mask, for instance GPT-4-turbo outperforming GPT-4o on SQL queries, file processing, and music tasks despite being an older release.","The dendrogram can be sliced at arbitrary granularity, so the same diagnosis serves both coarse-grained and fine-grained analyses, from broad categories like coding down to specific tasks like writing a riddle."],"supporting_citations":[{"why":"Supplies IFEval's verifiable instructions and the 25 checkable instruction types that SkillVerse adapts for its programmatic verifiers.","marker":"Zhou et al. (2023)"},{"why":"Provides ChatbotArena prompts and leaderboard context, the primary real-world dataset for clustering and experiments.","marker":"Chiang et al. (2024)"},{"why":"Provides the text-embedding API used to vectorize atomic-judgment objects for agglomerative clustering.","marker":"Google (2024b)"},{"why":"Defines contrastive in-context learning (C-ICL), the baseline SkillVerse's tree-search claims a 25% relative improvement over.","marker":"Yan et al. (2021)"},{"why":"Supplies the 'learning from principles' baseline and the general mistake-based ICL setup that SkillVerse extends.","marker":"Zhang et al. (2024)"},{"why":"Gives the similarity-only demonstration-selection method used as an ablation for SkillVerse's relevance-plus-difficulty ranking.","marker":"Mo et al. (2024)"},{"why":"Documents inverse scaling, the phenomenon SkillVerse identifies and predicts at fine granularity.","marker":"McKenzie et al."}],"fun_headline_variants":["Tree of LLM judgments boosts few-shot learning 25%","SkillVerse dendrogram finds model weaknesses, picks better demos","LLM self-critiques as tree lift in-context learning by 25%","Skill tree from LLM critique predicts failures, improves demos","Dendrogram of atomic judgments enhances LLM few-shot learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole framework assumes the LLM judge's free-form critiques, after parsing into atomic judgments, accurately reflect what the model actually did wrong or right; the paper never validates this parser.","fun_headline_variants_meta":{"raw":{"variants":["Tree of LLM judgments boosts few-shot learning 25%","SkillVerse dendrogram finds model weaknesses, picks better demos","LLM self-critiques as tree lift in-context learning by 25%","Skill tree from LLM critique predicts failures, improves demos","Dendrogram of atomic judgments enhances LLM few-shot learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00045,"raw_usage":{"total_tokens":2264,"prompt_tokens":935,"completion_tokens":1329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1238}},"tokens_in":551,"tokens_out":1329,"duration_ms":13157,"temperature":1.0,"reasoning_tokens":1238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:07:14.088939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually inspect a random sample of 200 atomic judgments and compare them against a human-annotated gold parse of the corresponding critiques, measuring agreement; if parser accuracy is low, the dendrogram and all downstream claims collapse. Alternatively, run SkillVerse with human-written critiques on the same response pairs and check whether the cluster structure and the reported 25% in-context learning improvement persist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines contrastive in-context learning (C-ICL), the baseline SkillVerse's tree-search claims a 25% relative improvement over."}],"review_version":1}