{"id":"e0336786-42f6-4ca0-905d-5a7d8b37c0ec","arxiv_id":"2506.00482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"BenchHub is an automatically categorized, customizable LLM benchmark suite covering 303K questions across 38 benchmarks in English and Korean.","lead":"BenchHub bundles 303,000 questions from 38 AI benchmarks and labels each one by subject, skill, and culture using an automated classifier. It lets users build custom test sets and shows that model leaderboard rankings shift with the mix of topics included.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Categorizer target accuracy (49.4%) and subject error (12.9%) exceed the 10% corruption tested in §5.2, so the paper's ranking-variation claims are not yet shown robust.","rationale":"The infrastructure contribution is real: the authors release 303K categorized questions, the categorizer model, and code, and the robustness analysis is a good-faith attempt to address label noise. However, the central empirical claim relies on the categorizer's labels being accurate enough that observed ranking shifts reflect true domain differences rather than misclassification artifacts. Table 7 shows the target label is at 49.4% accuracy—near chance for a multiclass problem—and the subject label is at 87.1%, implying a 12.9% error rate. Section 5.2 only tests corruption up to 10% and explicitly claims negligible disruption only up to 1.5%. The actual error rates lie outside the tested envelope, and the corruption model is random rather than systematic. Therefore the paper has not demonstrated that the ranking variations in §4.1–4.3 are robust to the categorizer's real error distribution. This is the weakest link in the central claim. The reader identified the target-accuracy issue; we extend it to subject accuracy and note the random-corruption limitation. A conditional verdict is appropriate: the paper should either extend the robustness analysis to realistic error rates or validate a sample of labels with human annotators and re-run the key analyses.","tokens_in":30909,"tokens_out":6088,"duration_ms":55285,"concrete_test":"Extend the §5.2 robustness ablation to corruption rates matching the actual misclassification rates from Table 7: 12.9% for subject labels and 50.6% for target labels, using both random corruption and a systematic corruption model that flips between confusable subject pairs (e.g., chemistry vs. biology). Compute Spearman rank correlation and Wilcoxon signed-rank p-values between corrupted and uncorrupted model rankings at each rate. If rankings remain stable (e.g., Spearman > 0.9 and non-significant Wilcoxon) at these rates, the central claim stands; if not, re-run §4.1 and §4.2 on a human-validated label subset to determine whether the observed ranking variation is a labeling artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim—that model rankings vary by subject and that benchmark composition affects rankings—depends on the BenchHub-Cat-7B categorizer's labels being accurate enough that observed shifts reflect true domain differences. Section 5.2's robustness analysis only tests corruption rates from 0% to 10% and reports negligible disruption only up to 1.5% corruption. However, Table 7 reports subject accuracy of 0.871 (12.9% error) and target accuracy of 0.494 (50.6% error). Thus the actual operating point for subject labels already exceeds the tested range, and target labels are roughly five times the tested maximum. Moreover, the ablation corrupts labels randomly, whereas real categorizer errors are likely systematic (e.g., confusion between adjacent subjects), which can induce larger ranking shifts than random noise. Consequently, the observed ranking fluctuations in §4.1–4.3, and especially the culture-specific customized evaluation in §4.3(b), could be partly an artifact of label noise rather than true domain-specific differences; the paper's own evidence does not rule this out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BenchHub, a benchmark infrastructure that aggregates 303K questions from 38 English and Korean datasets and classifies them at the sample level into skills, subjects, and target types using a fine-tuned Qwen-2.5-7B categorizer (BenchHub-Cat-7B). The authors evaluate seven or fourteen LLMs from several families and report that (i) model rankings fluctuate across coarse-grained subject categories, (ii) sampling strategies that mimic different benchmark compositions significantly change rankings, and (iii) customized subsets (STEM-focused English evaluation and a Korean math-teaching-agent scenario) yield different leaderboards. Section 5.2 attempts to show robustness of rankings to categorization errors via a corruption ablation up to 10% corruption.","tokens_in":31285,"tokens_out":2429,"duration_ms":25953,"significance":"If the central findings hold, BenchHub would be a practically useful and reproducible infrastructure for domain-aware and customizable LLM evaluation, and the demonstration that benchmark composition can alter rankings is an important caution for leaderboard interpretation. The paper's strengths include a public dataset and model release, a clearly described pipeline, and an unusually extensive set of evaluations across model families and languages. However, the validity of the empirical claims depends heavily on the accuracy of the automatic categorizer, and the paper's own robustness evidence does not cover the categorizer's actual operating point, especially for target labels. With additional validation at realistic error rates, structured noise, and per-subset statistical measures, the contribution could be solid; in its current form the load-bearing robustness claim is not established.","major_comments":[{"comment":"The corruption ablation only tests misclassification rates from 0% to 10%, but the categorizer's reported subject accuracy is 0.871 (12.9% error) and target accuracy is 0.494 (50.6% error). Consequently, the subject-label operating point lies above the tested range, and the target-label operating point lies roughly five times beyond the maximum tested corruption. Because the culture-specific 'Local' subsets and the customized Korean math-teaching evaluation in Section 4.3(b) rely directly on target labels, the paper does not currently demonstrate that those rankings are robust to real categorizer errors. The authors should repeat the ablation at the actual error rates and, ideally, with structured corruption that mimics realistic confusions between adjacent categories rather than purely random label corruption.","section":"Section 5.2 and Table 7"},{"comment":"There is a potential circularity in the composition experiments: the same BenchHub-Cat-7B categorizer is used to label MMLU, MixEval, and Chatbot Arena, and those labels define the category distributions that are then used to construct the sampling setups in Section 4.2. If the categorizer has systematic biases (for example, over- or under-labeling particular subject classes), those biases are baked into the very compositions that are shown to change rankings. The paper should validate the distributional labels against human annotation or an independent classifier, or at minimum analyze the sensitivity of the Section 4.2 conclusions to plausible bias patterns.","section":"Section 2 and Section 4.2"},{"comment":"Fine-grained category accuracies are reported without confidence intervals or sample sizes, and several rows appear to be based on extremely small numbers of items (for example, the Korean 'Psychology' row shows values of 0.000 and 1.000 across models, and Section 2 mentions KorNAT EN has only 24 samples). Without per-category sample counts and uncertainty estimates, the fine-grained rank fluctuations in Figure 6 and the customized-evaluation conclusions in Table 1 may not be statistically distinguishable from noise. The authors should provide sample sizes and confidence intervals, and should consider restricting claims for categories with very few items.","section":"Appendix F, Tables 9 and 10"},{"comment":"The text states that pairwise Wilcoxon signed-rank tests confirm all pairs of sampling setups significantly differ in average, 'except for random sampling versus sampling according to MixEval distribution (p < 0.01).' This is internally contradictory: if the exception has p < 0.01, it is not an exception. The intended result is presumably that random versus MixEval does not differ significantly, or that all pairs differ at p < 0.01; the sentence must be corrected and the associated p-values and multiple-testing correction should be reported explicitly.","section":"Section 4.2, final paragraph"}],"minor_comments":[{"comment":"The figure labels contain typos such as 'Humanities and Social Sciencce' and 'T ech'; these should be corrected for readability.","section":"Figures 2 and 3"},{"comment":"The word 'interpertable' should be 'interpretable'.","section":"Section 6"},{"comment":"The notation is inconsistent between 'Marin eng.' in Table 9 and 'Marine eng.' in Table 10 and the taxonomy in Table 6; unify the spelling.","section":"Appendix F and Table 2"},{"comment":"The sentence 'We select 6,644 and 6,485 examples for English and Korean, respectively' would benefit from a brief explanation of the selection criterion relative to the total 303K samples, since the numbers are much smaller than the full corpus.","section":"Section 4.1"},{"comment":"The description of the synthetic rationale generation in Appendix D.1 is important but appears only in the appendix; a short summary in the main text of how the classifier was trained and validated would help readers assess the accuracy numbers without opening the appendix.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper uses several datasets authored by members of the same research groups that produced the paper, and the categorizer labels the reference benchmarks that are then used to demonstrate composition effects. This is not inherently disqualifying, but it increases the burden on independent validation of the categorizer and of the distributional claims. An external or human-labeled validation set for subject and target labels would substantially strengthen the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BenchHub is a real artifact and should be reviewed, but the load-bearing robustness argument has a hole you should know about. The subject-level categorization is fine for many purposes (87.1% accuracy), yet the corruption ablation in §5.2 stops at 10% while the actual subject error is 12.9%, and target accuracy is 49.4% — far below what you'd want for filtering culture-specific sets. The authors claim resilience only up to 1.5% corruption, so the culture-specific parts of the paper (especially the Korean math teaching agent demo) are not supported by their own robustness data. That is the main soft spot, and the stress-test note has it right. Real categorizer errors are also likely systematic (adjacent subjects, ambiguous target), not the random corruption they simulate.\n\nWhat the paper does well: it ships. The data, code, and categorizer are public; 303K samples across 38 benchmarks is a useful aggregation, especially the Korean/English pair. The 14-model sampling study with 50 simulations per setup is a serious piece of work, and the Friedman/Wilcoxon analysis is appropriate. The observation that MMLU variants differ in subject mix and that rankings shuffle across categories is a useful, concrete restatement of the composition problem. Credit is due for the sample-wise multi-label classification, which goes beyond earlier benchmark-level taxonomies.\n\nWhere it is soft, beyond the categorizer gap: the same classifier that labels MMLU, MixEval, and Chatbot Arena is also used to build the sampling distributions, so there is a mild circularity — if the classifier has systematic bias, the \"canonical\" distributions inherit it. That does not kill the ranking-shift finding, because subject accuracy is decent and the effect is large, but it should be acknowledged. Appendix F has tiny category samples (KorNAT EN has 24 questions; Psychology shows one or two examples per model in the Korean table) and no confidence intervals. Also, the target classifier at 49.4% means the \"Local\" subsets and the culture-specific evaluation need human validation before anyone trusts them.\n\nWho should read this: people building custom evaluation sets, and researchers studying benchmark composition. The infrastructure is the contribution, not the empirical surprise.\n\nMy recommendation: send it to peer review, but with a requirement to either validate the categorizer labels against human annotations (at least for target and the categories used in the demos), or to restrict the claims to subject-level and skill-level results. The paper is honest about its limitations, and the artifacts are valuable enough to justify serious referee time.","headline":"A solid infrastructure paper with a real categorizer-robustness gap; the benchmark is worth using, but the culture-specific claims need human validation before I'd trust them.","tokens_in":31666,"tokens_out":3228,"would_cite":true,"duration_ms":34682,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Model rankings in LLM benchmarks shift significantly with the subject composition of the test set, and BenchHub's sample-level categorization makes that dependence visible and usable for customized evaluation.","keywords":["LLM evaluation","benchmark suite","domain-aware evaluation","benchmark composition","sample-wise categorization","culture-specific evaluation","model ranking","taxonomy"],"falsifier":"Take a sample of the BenchHub questions, have human annotators assign subject and target labels, rebuild the STEM-only English set and the Korean math-and-culture teaching-agent set from those gold labels, and re-rank the same models: if the customized rankings collapse toward the stratified rankings, or if the Friedman test across the four sampling strategies in Section 4.2 loses significance under the gold labels, the paper's composition claim would be shown to be an artifact of classifier noise rather than a property of benchmarks.","tokens_in":30722,"feed_emoji":"📊","tokens_out":14318,"duration_ms":111578,"temperature":0.7,"pith_summary":"This paper claims that a model's rank on an LLM leaderboard is not a stable property of the model but depends on what mix of subjects the test set happens to contain. To make that dependence visible and controllable, the authors build BenchHub, a repository that merges 303K questions from 38 English and Korean benchmarks and labels every question by skill (knowledge, reasoning, value/alignment), subject (64 fine-grained categories under six coarse ones), and cultural target (General or Local). Across seven model families they show rankings fluctuate sharply by category — Llama-3.3-70B, for instance, sits sixth in Science and Technology yet first in Culture and Social Intelligence — and that the sampling strategy used to compose a test set changes rankings significantly (Friedman test, $p < 0.01$). If the paper is right, single-number benchmark scores are composition-dependent, and fair model comparison requires domain-aware, customizable evaluation of the kind BenchHub supports.","feed_headline":"Swap the subject mix and a different LLM tops the leaderboard","feed_subtitle":"A 303K-question suite labels every item by subject and culture, showing how test mix tilts model rankings.","key_machinery":"The load-bearing object is the sample-wise taxonomy together with the automated categorizer that assigns its labels. Each question receives a skill label (knowledge, reasoning, or value/alignment), one or more subject labels drawn from 64 fine-grained categories organized under six coarse-grained subjects (Science, Technology, Humanities and Social Science, Arts and Sports, Culture, Social Intelligence), and a target label (General, or Local with a specific culture such as Korea or the United States). The categorizer, BenchHub-Cat-7B, is a Qwen-2.5-7B model fine-tuned on GPT-4o-generated synthetic rationales, and it is what lets the authors reclassify whole existing benchmarks sample by sample, quantify how subject distribution shifts model rankings, and construct customized evaluation sets. The distribution experiments in Section 4.2 then use these labels to resample 5K-question test sets under four strategies and test ranking shifts with the Friedman and Wilcoxon tests.","core_discovery":"On the paper's own terms, the central discovery is that benchmark composition — the distribution of subject categories in a test set — substantially changes which LLM appears to be best. The authors show that existing holistic benchmarks already encode very different compositions: MMLU and MixEval are more than half Humanities and Social Sciences, Chatbot Arena contains only 25.5% HASS, and the Korean KMMLU is 76.1% STEM. They then demonstrate experimentally that the same set of models is ranked differently depending on whether the test set is assembled by random sampling, stratified sampling, or sampling to match the category distributions of Chatbot Arena or MixEval, with the differences statistically significant at $p < 0.01$. BenchHub is presented as the infrastructure that surfaces this composition effect: because every question carries skill, subject, and target labels, a user can assemble a purpose-built test set — a STEM-only English set or a Korean math-and-culture teaching-agent set — and obtain rankings that differ from, and better match the intended use than, a generic stratified evaluation.","pith_inferences":["If the composition effect holds, it carries beyond English and Korean: any aggregated leaderboard in any language or modality silently encodes a subject-distribution choice, so multilingual and multimodal suites deserve the same kind of composition audit rather than a single averaged rank.","The paper's own Table 7 points to a stress point the robustness section does not cover: the target-culture label is accurate only about half the time, while the corruption analysis runs only to 10% misclassification. A fair reading suggests the culture-specific customized evaluations should be re-run with human-verified target labels to confirm they measure cultural knowledge rather than classific","One concrete extension an independent team could run: take the released BenchHub data, relabel a sample with human annotators, and check whether the Section 4.2 sampling-strategy differences and the Section 4.3 customized rankings survive; positive results would strengthen the composition thesis considerably."],"forward_implications":["Single-number leaderboard scores should no longer be read as model quality: the same models rank differently when the test set's subject mix changes, so a reported score needs a stated composition to be interpretable.","New datasets can be fed through the automated pipeline (reformatting, metadata assignment, sample-level classification, merging) and become part of BenchHub without manual curation, keeping evaluations current as models and benchmarks evolve.","Application-specific evaluation becomes practical: a team building a Korean math teaching agent can assemble a weighted test set of math, education, and Korea-culture questions and obtain a ranking that generic stratified sampling does not provide.","The categorized distribution statistics give benchmark designers a direct view of which subjects are over- and under-represented in current evaluation suites, identifying gaps worth filling with new datasets.","Small categorization noise is tolerable: the robustness study finds that corruption rates up to about 1.5% leave model rankings essentially undisturbed, so the framework does not require perfect labels to be useful."],"supporting_citations":[{"why":"Supplies the MMLU benchmark whose subject distribution is analyzed and whose questions enter the aggregated English set, anchoring the claim that holistic benchmarks are composition-skewed.","marker":"[16]"},{"why":"Provides the MixEval distribution used as one sampling strategy in Section 4.2 and the LLM-assisted answer-extraction approach adopted for short-form evaluation.","marker":"[41]"},{"why":"Provides the Chatbot Arena human-preference benchmark whose category distribution serves as the second sampling strategy against which ranking shifts are measured.","marker":"[7]"},{"why":"The survey of culture-related benchmarks that guides the selection of culture-specific datasets included in BenchHub.","marker":"[44]"},{"why":"Supplies KMMLU, the Korean benchmark whose 76.1% STEM composition supports the cross-lingual distribution comparison and the Korean evaluation subset.","marker":"[60]"},{"why":"Supplies the regular-expression answer extraction used to score multiple-choice questions in the ranking experiments.","marker":"[38]"},{"why":"Supplies the Qwen3 model family, several of whose members are among the 14 models in the Section 4.2 sampling-strategy experiments.","marker":"[72]"},{"why":"Supplies Qwen2.5, the base model fine-tuned into the BenchHub-Cat-7B categorizer that assigns the sample-wise labels.","marker":"[73]"}],"fun_headline_variants":["Benchmark mix changes which LLM ranks best","Swap subject mix, get a different top LLM","303K questions reveal test set skews rankings","Domain-aware evaluation reshuffles LLM leaderboards","BenchHub: swap the mix, flip the leaderboard"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the automated labels — subject categories and especially the cultural-target label, which is correct only about half the time — being accurate enough that the reported ranking shifts and culture-specific scores reflect what the questions actually measure, rather than noise in the classification.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark mix changes which LLM ranks best","Swap subject mix, get a different top LLM","303K questions reveal test set skews rankings","Domain-aware evaluation reshuffles LLM leaderboards","BenchHub: swap the mix, flip the leaderboard"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000439,"raw_usage":{"total_tokens":2227,"prompt_tokens":943,"completion_tokens":1284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1209}},"tokens_in":559,"tokens_out":1284,"duration_ms":10696,"temperature":1.0,"reasoning_tokens":1209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:03:17.663208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of the BenchHub questions, have human annotators assign subject and target labels, rebuild the STEM-only English set and the Korean math-and-culture teaching-agent set from those gold labels, and re-rank the same models: if the customized rankings collapse toward the stratified rankings, or if the Friedman test across the four sampling strategies in Section 4.2 loses significance under the gold labels, the paper's composition claim would be shown to be an artifact of classifier noise rather than a property of benchmarks.","supporting_citations":[{"cited_title":"MixEval: Deriving wisdom of the crowd from LLM benchmark mixtures","cited_arxiv_id":null,"evidence_quote":"Provides the MixEval distribution used as one sampling strategy in Section 4.2 and the LLM-assisted answer-extraction approach adopted for short-form evaluation."},{"cited_title":"Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I","cited_arxiv_id":null,"evidence_quote":"Provides the Chatbot Arena human-preference benchmark whose category distribution serves as the second sampling strategy against which ranking shifts are measured."},{"cited_title":"KMMLU: Measuring massive multitask language understanding in Korean","cited_arxiv_id":null,"evidence_quote":"Supplies KMMLU, the Korean benchmark whose 76.1% STEM composition supports the cross-lingual distribution comparison and the Korean evaluation subset."}],"review_version":1}