{"id":"721615ae-16a8-40a0-b273-0cebf4d35d11","arxiv_id":"2602.12424","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RankLLM ranks LLMs by jointly estimating question difficulty and model competency through a damped random walk on a model-question graph, claiming 90% agreement with human difficulty judgments.","lead":"This paper introduces RankLLM, a graph-based method that scores both LLMs and benchmark questions by propagating difficulty and competency signals between them. It reports that the resulting rankings agree with human difficulty judgments on 90% of pairwise comparisons and reorder models by performance on hard questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90% human-agreement headline is pool-relative: RankLLM difficulty is a stationary distribution over a particular 30-model pool, and Fig. 7 shows homogeneous pools drop consensus to 38.6–64.3%, so the central claim does not generalize without a pool-selection criterion.","rationale":"The reader's weakest assumption — that RankLLM scores are pool-relative — is exactly the most load-bearing concern I find. The abstract's headline claim ('achieves 90% agreement with human judgments') and the concluding claim that RankLLM is 'human-aligned' both rest on a single 30-model pool. The paper's own Figure 7 demonstrates that the result does not survive even a coarse change to size-homogeneous pools, with consensus dropping to 38.6–64.3%. This is not a speculative failure mode; it is an empirical boundary condition reported in the same paper. Table 5 further shows that removing half the pool shifts question-difficulty rankings with Spearman 0.938, so the difficulty scores themselves are not stable across reasonable pool perturbations. The central claim is therefore best read as 'RankLLM, when run on this particular mixed-size pool, agrees with human consensus on 70 pairs' — not as a general property of the method. Other issues raised in the reader's rationale (per-rater agreement around 63%, an undisclosed damping parameter, and a technically flawed convergence proof in Appendix C) are real but secondary; they would affect confidence and reproducibility, but the pool-dependence concern is the one that most directly controls whether the abstract's central claim generalizes. Because the method may still be useful as a relative, within-pool ranking tool, I do not recommend rejection; the verdict should remain CONDITIONAL, with the added requirement that the authors either provide a principled pool-selection protocol or substantially qualify the 90% human-alignment claim. Since this matches the reader's weakest assumption, no verdict change is needed.","tokens_in":28283,"tokens_out":14159,"duration_ms":130608,"concrete_test":"Using the existing response matrix and the 70 human-labeled pairs, draw 100 random subsets of the 30-model pool with k=5, 10, and 15 models removed, stratified by size (small/medium/large) to mimic the diversity manipulation in Fig. 7. For each subset, rerun RankLLM with the paper's damping and tolerance settings and compute (i) the question-difficulty Spearman correlation with the full-pool scores and (ii) the consensus metric from Eq. (9) against the same human majority labels. If the 95% CI of consensus for k=15 extends below 70%, or if the question-difficulty ρ distribution overlaps the homogeneous-pool range, the 90% claim is pool-specific and the abstract must be qualified. If consensus remains near 90% across all stratified subsets, the pool-dependence concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RankLLM's central output — question difficulty πQ and model competency πM — is not a property of questions/models alone: Equations (2)–(3) define scores as the stationary distribution of a random walk whose transition matrices are built from the success/failure pattern of one specific set of 30 models. The paper's own Fig. 7 is the decisive stress test: replacing the full pool with size-homogeneous pools changes human consensus from 90.0% to 64.3%, 61.4%, and 38.6%, with Cohen's κ collapsing from 0.80 to 0.13–0.29. Table 5 shows the same fragility on the question side: deleting 15 of 30 models lowers the question-difficulty Spearman correlation to 0.938 (SD 0.019), so the ordering of questions is materially affected by pool composition. The claimed 90% agreement with human judgments is therefore not a validation of RankLLM as a general difficulty estimator; it is a validation of the particular curated pool used here. The method provides no formal criterion for selecting model pool size or composition, and the paper's 'Whole' pool is just one arbitrary instantiation. Any user with a different pool — e.g., all frontier models — can obtain substantially different difficulty rankings and, potentially, different model orderings. Unless the method is accompanied by a pool-selection rule or a pooling-invariant formulation, the title's 'quantifying question difficulty' is not supported in general.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RankLLM defines question difficulty and model competency as the joint stationary distribution of a damped random walk on a bipartite graph whose edges encode which models solved or failed which benchmark questions. The paper evaluates the method on six benchmarks with 30 models and 35,550 questions, reporting 90% consensus with human difficulty judgments, faster convergence than IRT baselines, stable question and model rankings under model-pool and dataset perturbations, and a controlled simulation in which only RankLLM recovers a target ordering. The proposed contribution is to make item-level difficulty an explicit, jointly estimated quantity rather than a fixed property of benchmark items.","tokens_in":28635,"tokens_out":12545,"duration_ms":121734,"significance":"If the claims hold, RankLLM is a practically useful and refreshingly simple alternative to IRT-style difficulty estimation: it is non-parametric, interpretable, and cheap (a few matrix-vector products per iteration), and it directly addresses a real gap in benchmark evaluation. The paper is unusually transparent in its reproducibility statement, licensing appendix, and detailed evaluation protocols, which is a clear strength. The fixed-point formulation in Eqs. (1)-(3) is a legitimate way to operationalize difficulty, and the pool-dependence of the scores is not by itself circular or invalid. However, the current manuscript overstates what is established: the formal convergence proof is flawed, the headline human-alignment figure is pool-relative and statistically thin, and the sensitivity claim rests largely on a single small simulation. With targeted but substantive revisions, the paper could become a solid contribution.","major_comments":[{"comment":"The proof of Theorem C.1 is not valid as written. In Step 2, the combined teleportation vector v = [1_Q/Q; 1_M/M] has total mass 2, so the matrix T = alpha P^T + (1-alpha) v 1^T/(Q+M) is not stochastic (column sums are alpha + 2(1-alpha)/(Q+M), not 1). The irreducibility, aperiodicity, and Perron-Frobenius arguments therefore do not apply to this T. Since Theorem C.1 is the only formal justification for the claimed guarantee of convergence to a unique stationary distribution, this is load-bearing. The alternating contraction view suggests the theorem is true, but the proof must be replaced or repaired (e.g., by normalizing v by 2, or by proving the composition of the two updates is a contraction).","section":"Appendix C, Eqs. (6)-(7)"},{"comment":"The abstract's headline '90% agreement with human judgments' is pool-relative. The paper itself reports that replacing the full 30-model pool with size-homogeneous pools drops consensus to 64.3%, 61.4%, and 38.6% with kappa = 0.29, 0.27, and 0.13 (Fig. 7), and that removing 15 of 30 models lowers question-difficulty Spearman correlation to 0.938 (Table 5). Because Eqs. (2)-(3) define difficulty as the stationary distribution over a specific model pool, there is no formal basis for claiming that RankLLM outputs 'the' difficulty of a question. The paper should either supply a pool-selection criterion, characterize conditions under which scores are approximately pool-invariant, or explicitly qualify the central claim as conditional on a diverse pool.","section":"§3.3; Fig. 7; Table 5"},{"comment":"The human-validation evidence is thin for a central claim. It consists of 70 pairwise judgments made by 20 evaluators, with no confidence intervals, no per-pair measure of rater agreement, and no significance test comparing RankLLM to the IRT baselines. The Consensus metric of Eq. (9) is majority agreement over raters, which can mask low individual agreement; indeed, individual RankLLM alignments in Table 2 range from 37.1% to 74.3%. The statement that RankLLM 'outperforms' IRT baselines should be supported by paired tests, effect sizes, and uncertainty intervals, and the result should be reported with and without the two author-evaluators.","section":"§3.3; Eq. (9); Table 2"},{"comment":"The controlled simulation that supports the sensitivity claim ('only RankLLM recovers the ground truth') is a single run with 5 hypothetical models and 100 questions constructed from an easy/medium/hard grid. There are no repetitions, no variation in the difficulty mixture, no noise injection into the response matrix, and no exploration of the number of models or questions. Given that some IRT baselines also recover parts of the target ordering, one favorable configuration is not enough to establish that RankLLM is reliably more sensitive than IRT. Please provide a small sensitivity study, multiple random replicates, or at least a discussion of how robust the comparison is to the chosen simulation parameters.","section":"§4; Table 6"}],"minor_comments":[{"comment":"The text in §3.1 says '26 mainstream LLMs,' while the abstract and Table 12 list 30 models. Please reconcile.","section":"§3.1; Table 12"},{"comment":"Kendall's tau between RankLLM scores and accuracy is reported as 0.8492 in §3.2 but as 0.876 (p<0.001) in Table 11; Section 1's phrasing ('adjacent changes consistent with Kendall's tau = 0.8492') is also unclear. Please make the reported correlation consistent and specify exactly which ranking pair the coefficient refers to.","section":"§3.2; Appendix G"},{"comment":"Table 8 includes a row for alpha = 1, but the method is defined for alpha in (0,1) and the ergodicity argument relies on damping. For an undamped bipartite walk, the chain is periodic, so the reported convergence for alpha = 1 appears to contradict the proof sketch in Appendix C. Either remove this row or explain why the special initialization converges.","section":"Table 8; §2.3"},{"comment":"Equation (1) states the score definitions 'proportional to' the propagation sums, but it omits the damping/teleportation terms of Eqs. (2)-(3). As written, Eq. (1) is not the exact fixed-point equation; please mark it as an intuitive definition or include the constant terms.","section":"Eq. (1) vs Eqs. (2)-(3)"},{"comment":"Table 3 reports full-pool convergence of 0.00597 s, while Table 5 reports 0.0079 s after removing one model with a 14.09% 'reduction.' The relationship between these numbers is unclear; please define the comparison baseline and explain the variability.","section":"Tables 3, 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is marked as 'Published as a conference paper at ICLR 2026.' If the journal version is expected to go beyond the conference paper, the editor may wish to clarify the incremental contribution expected. The technical issues above are substantive but, in my view, repairable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful core: RankLLM is a practical way to get difficulty-aware rankings of LLMs. The idea—a damped random walk on the bipartite graph of models and questions, where solving a hard question boosts a model and failing an easy one boosts the question—is simple, converges in milliseconds, and gives sensible re-rankings. The empirical work is solid in its own terms: 30 models, 35,550 questions across six benchmarks, comparisons against IRT baselines, robustness checks, and a human study. The finding that mixed-size model pools align much better with human difficulty judgments (90% consensus vs. 39–64% for homogeneous pools) is a real and useful insight. I'd cite the empirical findings, and the method itself is likely to be used by the community.\n\nBut there are several problems. The convergence proof in Appendix C is simply wrong as written: the teleportation vector v has total mass 2, so the transition matrix T is not stochastic and the Perron–Frobenius argument doesn't apply. The alternating update equations (2)–(3) do not define the single Markov chain T described in the proof. The algorithm probably does converge—the empirical data look clean—but the proof needs a fix or a rewrite.\n\nSecond, the damping factor α is never disclosed in the main text or appendix. The reproducibility statement says config files are included, but the paper doesn't state the value used in the experiments. That's a basic transparency issue.\n\nThird, the 90% human agreement is majority-vote consensus over 70 pairs, not per-rater agreement (which averages around 63%). The abstract's '90% agreement with human judgments' is technically true but easy to misread, and the small number of pairs gives no confidence intervals.\n\nFourth, and most structural, difficulty is pool-relative. The scores are the stationary distribution of a walk over a particular set of 30 models. The paper's own Figure 7 shows that with size-homogeneous pools the consensus drops to 38.6–64.3%, and Table 5 shows that removing 15 models still shifts question-difficulty Spearman to 0.938. That's okay if you treat RankLLM as a tool for ranking within a defined pool, but the title says 'quantifying question difficulty' as if it were an intrinsic property. The paper needs a clear statement of the pool-dependence and, ideally, guidance on what pools give stable results.\n\nFinally, the method is essentially damped HITS/PageRank on a bipartite graph. Not citing that lineage is a miss.\n\nThese are all addressable. The central idea is sound and the experimental work is substantial. This deserves a serious referee—it should go to peer review, with expectations of a revision that fixes the proof, discloses α, tempers the human-agreement framing, and adds a pool-dependence discussion.","headline":"RankLLM is a genuinely useful empirical paper, but its convergence proof is wrong, α is undisclosed, and the 90% human agreement is consensus-by-pool—all fixable in revision.","tokens_in":29153,"tokens_out":4964,"would_cite":true,"duration_ms":42430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RankLLM assigns every benchmark question a difficulty score and every model a competency score through one coupled random walk, and reports 90% agreement with human judgments of difficulty.","keywords":["RankLLM","LLM evaluation","question difficulty","model competency","bidirectional score propagation","bipartite graph","item response theory","benchmark ranking"],"falsifier":"Run RankLLM on a homogeneous pool of 30 models of one size class on the same 70 human-judged pairs: if consensus agreement falls toward or below the 50% baseline (as the paper's own Fig. 7 suggests at 38.6–64.3%), the difficulty scores are pool-relative artifacts, not recoverable human difficulty.","tokens_in":28171,"feed_emoji":"📊","tokens_out":5372,"duration_ms":48228,"temperature":0.7,"pith_summary":"Benchmark scores today treat every question as equally informative, so a model that nails hard reasoning and one that only handles routine items look the same if their hit rates match. This paper claims to fix that by deriving per-question difficulty and per-model competency from the same response matrix, using a damped random walk that passes score back and forth between questions and models. Solving a hard question raises a model's competency; failing an easy question raises the question's difficulty. Across 30 models and 35,550 questions, the resulting difficulty scores match human consensus 90% of the time, beat IRT baselines, and reorder close models according to performance on hard questions. If correct, leaderboards would no longer treat every correct answer as equal.","feed_headline":"Difficulty scores match humans 90% and re-rank LLMs","feed_subtitle":"A coupled random walk over models and questions weights hard questions more, exposing strengths raw accuracy hides.","key_machinery":"The load-bearing object is a damped random walk on a directed bipartite graph G = (Q ∪ M, E_Comp ∪ E_Fail). Two row-stochastic transition matrices encode the walks: P_{Q→M} normalizes success edges by the number of solvers S(q), and P_{M→Q} normalizes failure edges by the number of failures F(m). A damping factor α ∈ (0,1) teleports the walker to a uniform state, making the chain ergodic and guaranteeing a unique stationary distribution via Perron-Frobenius. That stationary distribution simultaneously yields difficulty scores π_Q and competency scores π_M, so difficulty and competency are defined, not by annotation, but as the equilibrium of mutual reinforcement.","core_discovery":"The central claim is that question difficulty and model competency are not separate quantities but two sides of one stationary distribution. RankLLM builds a directed bipartite graph whose edges are successes and failures between models and questions, then iterates two reciprocal updates: a question's difficulty mass flows to the models that solved it, divided by the number of solvers, and a model's competency mass flows to the questions it failed, divided by the number of failures. The damped updates converge to a unique stationary distribution (a consequence of Perron-Frobenius), giving each question a difficulty score and each model a competency score. The authors show that these difficul","pith_inferences":["Beyond the paper: difficulty scores are pool-relative, so the same question can change difficulty when the model set changes; cross-benchmark comparisons need a matched pool, and the paper's own stability numbers (ρ = 0.938 after removing 15 of 30 models) show the ranking is stable but not invariant.","Beyond the paper: the method's resemblance to eigenvector-style centralities suggests it could be applied to partial-credit or continuous scores (the paper sketches this) and to non-binary interaction graphs such as logits or confidence scores, where the same fixed-point iteration would yield soft difficulty estimates.","Beyond the paper: the 90% human agreement was obtained on same-category pairs judged by 20 evaluators; a natural test is whether agreement persists across categories and evaluator expertise levels, since the protocol allowed raters to skip unfamiliar items.","Beyond the paper: because the difficulty signal flows only from failures and successes, the method cannot distinguish a question that is hard because of missing knowledge from one that is hard because of reasoning complexity; separating those two sources would require auxiliary information beyond the response matrix."],"forward_implications":["Leaderboards can report a difficulty score for each question at no extra labeling cost, since the score is a byproduct of the models' own responses.","Closely matched models can be re-ranked: a model with lower raw accuracy but more correct answers on hard questions can outrank a model with higher overall accuracy, so flat accuracy tables may hide real capability differences.","Dataset difficulty profiles become measurable and comparable—MATH and MMLU-Pro show wider difficulty tails, while GSM8K and HellaSwag skew easy—which informs which benchmarks are good at separating models.","The computation scales linearly in Q×M and converged in about 0.006 seconds for 30 models and 35,550 questions, making it practical to recompute difficulty as new models are added.","If the model pool is diverse, open-weight models alone estimate difficulty almost as well as the full pool (Spearman 0.96), suggesting difficulty-aware evaluation need not depend on proprietary APIs."],"fun_headline_variants":["Weighted LLM ranking: questions with difficulty scores","90% human accord on difficulty, then re-rank LLMs","Coupling model and question scores exposes true LLM strength","Difficulty-aware ranking beats raw accuracy on 35k questions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that question difficulty can be inferred from the pattern of successes and failures of the specific pool of models being used, and that a diverse pool is available; the paper itself reports that homogeneous pools drop human consensus to 38.6–64.3%, so the 90% alignment does not survive a narrow pool.","fun_headline_variants_meta":{"raw":{"variants":["Weighted LLM ranking: questions with difficulty scores","90% human accord on difficulty, then re-rank LLMs","Coupling model and question scores exposes true LLM strength","Difficulty-aware ranking beats raw accuracy on 35k questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1218,"prompt_tokens":706,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":457}},"tokens_in":450,"tokens_out":512,"duration_ms":5344,"temperature":1.0,"reasoning_tokens":457,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:48:38.261695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RankLLM on a homogeneous pool of 30 models of one size class on the same 70 human-judged pairs: if consensus agreement falls toward or below the 50% baseline (as the paper's own Fig. 7 suggests at 38.6–64.3%), the difficulty scores are pool-relative artifacts, not recoverable human difficulty.","supporting_citations":[],"review_version":1}