Introduces the first community-governed unified JSON schema and crowdsourced repository for AI evaluation results, with converters and a database spanning 22,235 models and 2,273 benchmarks.
arXiv preprint arXiv:2511.04689 , year=
8 Pith papers cite this work. Polarity classification is still indexing.
8
Pith papers citing it
years
2026 8representative citing papers
Using a 1PL IRT model on real cultural questions across 13 locales, the study identifies a local-language knowledge-access advantage masked by lower proficiency in raw accuracy.
FailureScope clusters evaluation probes by cross-model failure patterns via LOMO to produce stable taxonomies that generalize across single-turn, multi-turn, and adversarial regimes, with reported metrics of Kendall's tau 0.81 and AUC 0.88.
An IRT-based detector identifies mislabeled examples in LLM benchmarks at 95% precision in the top 200 cases, outperforming supervised classifiers and revealing reward-model specialization on style over facts.
ECC calibrates semantic embeddings with model comparisons via Bradley-Terry profiles and mixture weights to cluster queries by latent LLM capabilities, claiming 17-18 point gains in ranking quality over baselines.
Dynamic Boundary Evaluation locates each LLM's performance boundary at ~50% pass probability via a calibrated item bank and Skill-Guided Boundary Search algorithm to enable unified, adaptive evaluations across safety, capability, and truthfulness.
DualEval jointly calibrates LLM abilities and item difficulties/sharpness in a shared latent space using static labels and reward-model scores to unify benchmark and arena-style evaluation.
citing papers explorer
-
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Introduces the first community-governed unified JSON schema and crowdsourced repository for AI evaluation results, with converters and a database spanning 22,235 models and 2,273 benchmarks.
-
The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
Using a 1PL IRT model on real cultural questions across 13 locales, the study identifies a local-language knowledge-access advantage masked by lower proficiency in raw accuracy.
-
FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses
FailureScope clusters evaluation probes by cross-model failure patterns via LOMO to produce stable taxonomies that generalize across single-turn, multi-turn, and adversarial regimes, with reported metrics of Kendall's tau 0.81 and AUC 0.88.
-
Auditing LLM Benchmarks with Item Response Theory
An IRT-based detector identifies mislabeled examples in LLM benchmarks at 95% precision in the top 200 cases, outperforming supervised classifiers and revealing reward-model specialization on style over facts.
-
Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
ECC calibrates semantic embeddings with model comparisons via Bradley-Terry profiles and mixture weights to cluster queries by latent LLM capabilities, claiming 17-18 point gains in ranking quality over baselines.
-
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
Dynamic Boundary Evaluation locates each LLM's performance boundary at ~50% pass probability via a calibrated item bank and Skill-Guided Boundary Search algorithm to enable unified, adaptive evaluations across safety, capability, and truthfulness.
-
DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
DualEval jointly calibrates LLM abilities and item difficulties/sharpness in a shared latent space using static labels and reward-model scores to unify benchmark and arena-style evaluation.
- Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration