RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

Kaidi Xu; Kai Zhang; Lichao Sun; Neil Zhenqiang Gong; Qingsong Wen; Ruoxi Chen; Xiangliang Zhang; Xingjian Hu; Yixin Liu; Yue Huang

REVIEW 4 major objections 5 minor 7 references

RankLLM assigns every benchmark question a difficulty score and every model a competency score through one coupled random walk, and reports 90% agreement with human judgments of difficulty.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:48 UTC pith:D6K7TQ5B

load-bearing objection RankLLM is a genuinely useful empirical paper, but its convergence proof is wrong, α is undisclosed, and the 90% human agreement is consensus-by-pool—all fixable in revision. the 4 major comments →

arxiv 2602.12424 v2 pith:D6K7TQ5B submitted 2026-02-12 cs.CL cs.AI

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

Ziqian Zhang , Xingjian Hu , Yue Huang , Kai Zhang , Ruoxi Chen , Yixin Liu , Qingsong Wen , Kaidi Xu

show 3 more authors

Xiangliang Zhang Neil Zhenqiang Gong Lichao Sun

This is my paper

classification cs.CL cs.AI

keywords RankLLMLLM evaluationquestion difficultymodel competencybidirectional score propagationbipartite graphitem response theorybenchmark ranking

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Benchmark scores today treat every question as equally informative, so a model that nails hard reasoning and one that only handles routine items look the same if their hit rates match. This paper claims to fix that by deriving per-question difficulty and per-model competency from the same response matrix, using a damped random walk that passes score back and forth between questions and models. Solving a hard question raises a model's competency; failing an easy question raises the question's difficulty. Across 30 models and 35,550 questions, the resulting difficulty scores match human consensus 90% of the time, beat IRT baselines, and reorder close models according to performance on hard questions. If correct, leaderboards would no longer treat every correct answer as equal.

Core claim

The central claim is that question difficulty and model competency are not separate quantities but two sides of one stationary distribution. RankLLM builds a directed bipartite graph whose edges are successes and failures between models and questions, then iterates two reciprocal updates: a question's difficulty mass flows to the models that solved it, divided by the number of solvers, and a model's competency mass flows to the questions it failed, divided by the number of failures. The damped updates converge to a unique stationary distribution (a consequence of Perron-Frobenius), giving each question a difficulty score and each model a competency score. The authors show that these difficul

What carries the argument

The load-bearing object is a damped random walk on a directed bipartite graph G = (Q ∪ M, E_Comp ∪ E_Fail). Two row-stochastic transition matrices encode the walks: P_{Q→M} normalizes success edges by the number of solvers S(q), and P_{M→Q} normalizes failure edges by the number of failures F(m). A damping factor α ∈ (0,1) teleports the walker to a uniform state, making the chain ergodic and guaranteeing a unique stationary distribution via Perron-Frobenius. That stationary distribution simultaneously yields difficulty scores π_Q and competency scores π_M, so difficulty and competency are defined, not by annotation, but as the equilibrium of mutual reinforcement.

Load-bearing premise

The load-bearing premise is that question difficulty can be inferred from the pattern of successes and failures of the specific pool of models being used, and that a diverse pool is available; the paper itself reports that homogeneous pools drop human consensus to 38.6–64.3%, so the 90% alignment does not survive a narrow pool.

What would settle it

Run RankLLM on a homogeneous pool of 30 models of one size class on the same 70 human-judged pairs: if consensus agreement falls toward or below the 50% baseline (as the paper's own Fig. 7 suggests at 38.6–64.3%), the difficulty scores are pool-relative artifacts, not recoverable human difficulty.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Leaderboards can report a difficulty score for each question at no extra labeling cost, since the score is a byproduct of the models' own responses.
Closely matched models can be re-ranked: a model with lower raw accuracy but more correct answers on hard questions can outrank a model with higher overall accuracy, so flat accuracy tables may hide real capability differences.
Dataset difficulty profiles become measurable and comparable—MATH and MMLU-Pro show wider difficulty tails, while GSM8K and HellaSwag skew easy—which informs which benchmarks are good at separating models.
The computation scales linearly in Q×M and converged in about 0.006 seconds for 30 models and 35,550 questions, making it practical to recompute difficulty as new models are added.
If the model pool is diverse, open-weight models alone estimate difficulty almost as well as the full pool (Spearman 0.96), suggesting difficulty-aware evaluation need not depend on proprietary APIs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: difficulty scores are pool-relative, so the same question can change difficulty when the model set changes; cross-benchmark comparisons need a matched pool, and the paper's own stability numbers (ρ = 0.938 after removing 15 of 30 models) show the ranking is stable but not invariant.
Beyond the paper: the method's resemblance to eigenvector-style centralities suggests it could be applied to partial-credit or continuous scores (the paper sketches this) and to non-binary interaction graphs such as logits or confidence scores, where the same fixed-point iteration would yield soft difficulty estimates.
Beyond the paper: the 90% human agreement was obtained on same-category pairs judged by 20 evaluators; a natural test is whether agreement persists across categories and evaluator expertise levels, since the protocol allowed raters to skip unfamiliar items.
Beyond the paper: because the difficulty signal flows only from failures and successes, the method cannot distinguish a question that is hard because of missing knowledge from one that is hard because of reasoning complexity; separating those two sources would require auxiliary information beyond the response matrix.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

RankLLM is a genuinely useful empirical paper, but its convergence proof is wrong, α is undisclosed, and the 90% human agreement is consensus-by-pool—all fixable in revision.

read the letter

First, the useful core: RankLLM is a practical way to get difficulty-aware rankings of LLMs. The idea—a damped random walk on the bipartite graph of models and questions, where solving a hard question boosts a model and failing an easy one boosts the question—is simple, converges in milliseconds, and gives sensible re-rankings. The empirical work is solid in its own terms: 30 models, 35,550 questions across six benchmarks, comparisons against IRT baselines, robustness checks, and a human study. The finding that mixed-size model pools align much better with human difficulty judgments (90% consensus vs. 39–64% for homogeneous pools) is a real and useful insight. I'd cite the empirical findings, and the method itself is likely to be used by the community.

But there are several problems. The convergence proof in Appendix C is simply wrong as written: the teleportation vector v has total mass 2, so the transition matrix T is not stochastic and the Perron–Frobenius argument doesn't apply. The alternating update equations (2)–(3) do not define the single Markov chain T described in the proof. The algorithm probably does converge—the empirical data look clean—but the proof needs a fix or a rewrite.

Second, the damping factor α is never disclosed in the main text or appendix. The reproducibility statement says config files are included, but the paper doesn't state the value used in the experiments. That's a basic transparency issue.

Third, the 90% human agreement is majority-vote consensus over 70 pairs, not per-rater agreement (which averages around 63%). The abstract's '90% agreement with human judgments' is technically true but easy to misread, and the small number of pairs gives no confidence intervals.

Fourth, and most structural, difficulty is pool-relative. The scores are the stationary distribution of a walk over a particular set of 30 models. The paper's own Figure 7 shows that with size-homogeneous pools the consensus drops to 38.6–64.3%, and Table 5 shows that removing 15 models still shifts question-difficulty Spearman to 0.938. That's okay if you treat RankLLM as a tool for ranking within a defined pool, but the title says 'quantifying question difficulty' as if it were an intrinsic property. The paper needs a clear statement of the pool-dependence and, ideally, guidance on what pools give stable results.

Finally, the method is essentially damped HITS/PageRank on a bipartite graph. Not citing that lineage is a miss.

These are all addressable. The central idea is sound and the experimental work is substantial. This deserves a serious referee—it should go to peer review, with expectations of a revision that fixes the proof, discloses α, tempers the human-agreement framing, and adds a pool-dependence discussion.

Referee Report

4 major / 5 minor

Summary. RankLLM defines question difficulty and model competency as the joint stationary distribution of a damped random walk on a bipartite graph whose edges encode which models solved or failed which benchmark questions. The paper evaluates the method on six benchmarks with 30 models and 35,550 questions, reporting 90% consensus with human difficulty judgments, faster convergence than IRT baselines, stable question and model rankings under model-pool and dataset perturbations, and a controlled simulation in which only RankLLM recovers a target ordering. The proposed contribution is to make item-level difficulty an explicit, jointly estimated quantity rather than a fixed property of benchmark items.

Significance. If the claims hold, RankLLM is a practically useful and refreshingly simple alternative to IRT-style difficulty estimation: it is non-parametric, interpretable, and cheap (a few matrix-vector products per iteration), and it directly addresses a real gap in benchmark evaluation. The paper is unusually transparent in its reproducibility statement, licensing appendix, and detailed evaluation protocols, which is a clear strength. The fixed-point formulation in Eqs. (1)-(3) is a legitimate way to operationalize difficulty, and the pool-dependence of the scores is not by itself circular or invalid. However, the current manuscript overstates what is established: the formal convergence proof is flawed, the headline human-alignment figure is pool-relative and statistically thin, and the sensitivity claim rests largely on a single small simulation. With targeted but substantive revisions, the paper could become a solid contribution.

major comments (4)

[Appendix C, Eqs. (6)-(7)] The proof of Theorem C.1 is not valid as written. In Step 2, the combined teleportation vector v = [1_Q/Q; 1_M/M] has total mass 2, so the matrix T = alpha P^T + (1-alpha) v 1^T/(Q+M) is not stochastic (column sums are alpha + 2(1-alpha)/(Q+M), not 1). The irreducibility, aperiodicity, and Perron-Frobenius arguments therefore do not apply to this T. Since Theorem C.1 is the only formal justification for the claimed guarantee of convergence to a unique stationary distribution, this is load-bearing. The alternating contraction view suggests the theorem is true, but the proof must be replaced or repaired (e.g., by normalizing v by 2, or by proving the composition of the two updates is a contraction).
[§3.3; Fig. 7; Table 5] The abstract's headline '90% agreement with human judgments' is pool-relative. The paper itself reports that replacing the full 30-model pool with size-homogeneous pools drops consensus to 64.3%, 61.4%, and 38.6% with kappa = 0.29, 0.27, and 0.13 (Fig. 7), and that removing 15 of 30 models lowers question-difficulty Spearman correlation to 0.938 (Table 5). Because Eqs. (2)-(3) define difficulty as the stationary distribution over a specific model pool, there is no formal basis for claiming that RankLLM outputs 'the' difficulty of a question. The paper should either supply a pool-selection criterion, characterize conditions under which scores are approximately pool-invariant, or explicitly qualify the central claim as conditional on a diverse pool.
[§3.3; Eq. (9); Table 2] The human-validation evidence is thin for a central claim. It consists of 70 pairwise judgments made by 20 evaluators, with no confidence intervals, no per-pair measure of rater agreement, and no significance test comparing RankLLM to the IRT baselines. The Consensus metric of Eq. (9) is majority agreement over raters, which can mask low individual agreement; indeed, individual RankLLM alignments in Table 2 range from 37.1% to 74.3%. The statement that RankLLM 'outperforms' IRT baselines should be supported by paired tests, effect sizes, and uncertainty intervals, and the result should be reported with and without the two author-evaluators.
[§4; Table 6] The controlled simulation that supports the sensitivity claim ('only RankLLM recovers the ground truth') is a single run with 5 hypothetical models and 100 questions constructed from an easy/medium/hard grid. There are no repetitions, no variation in the difficulty mixture, no noise injection into the response matrix, and no exploration of the number of models or questions. Given that some IRT baselines also recover parts of the target ordering, one favorable configuration is not enough to establish that RankLLM is reliably more sensitive than IRT. Please provide a small sensitivity study, multiple random replicates, or at least a discussion of how robust the comparison is to the chosen simulation parameters.

minor comments (5)

[§3.1; Table 12] The text in §3.1 says '26 mainstream LLMs,' while the abstract and Table 12 list 30 models. Please reconcile.
[§3.2; Appendix G] Kendall's tau between RankLLM scores and accuracy is reported as 0.8492 in §3.2 but as 0.876 (p<0.001) in Table 11; Section 1's phrasing ('adjacent changes consistent with Kendall's tau = 0.8492') is also unclear. Please make the reported correlation consistent and specify exactly which ranking pair the coefficient refers to.
[Table 8; §2.3] Table 8 includes a row for alpha = 1, but the method is defined for alpha in (0,1) and the ergodicity argument relies on damping. For an undamped bipartite walk, the chain is periodic, so the reported convergence for alpha = 1 appears to contradict the proof sketch in Appendix C. Either remove this row or explain why the special initialization converges.
[Eq. (1) vs Eqs. (2)-(3)] Equation (1) states the score definitions 'proportional to' the propagation sums, but it omits the damping/teleportation terms of Eqs. (2)-(3). As written, Eq. (1) is not the exact fixed-point equation; please mark it as an intuitive definition or include the constant terms.
[Tables 3, 5] Table 3 reports full-pool convergence of 0.00597 s, while Table 5 reports 0.0079 s after removing one model with a 14.09% 'reduction.' The relationship between these numbers is unclear; please define the comparison baseline and explain the variability.

Circularity Check

1 steps flagged

Core RankLLM fixed-point equations are legitimate and the human-alignment claim is externally grounded; the one circular element is the Section 4 simulation, whose ground truth is built from the same easy/medium/hard construct RankLLM is designed to estimate.

specific steps

self definitional [Section 4 (Case Study), Tables 6-7]
"Models were configured to establish ground truth ranking: M1 > M2 > M4 > M5 > M3. M1/2 had equal accuracy, but M1 answered more hard questions. M4/5 had equal accuracy, but M4 performed better on medium-difficulty items. ... Table 7: Easy 70 Mean 18.24; Medium 21 Mean 73.44; Hard 9 Mean 98.59."

The simulated ground truth is defined by performance on pre-labeled easy/medium/hard items. RankLLM's competency score πm is, by Eq. (1), a weighted sum of the difficulty scores πq of solved questions, and πq is derived from the same binary response matrix that those labels generated. Table 7 shows RankLLM's estimated difficulty immediately recovers the configured categories (Easy≈18, Medium≈73, Hard≈99). Thus the 'validation' that RankLLM recovers the configured model ranking and outperforms IRT in this experiment is a consistency check of its own weighting scheme, not an independent test: the success criterion (hard/medium performance) is the same construct the method encodes.

full rationale

RankLLM's central equations (1)-(3) are a legitimate fixed-point/stationary-distribution formulation, not a fit to the target result; the method defines difficulty as model failure and competency as success-weighted difficulty, and the paper's 90% agreement with human judgments is an external, independent check that keeps the core claim from being circular. No load-bearing self-citation appears in the derivation; the convergence argument relies on the external Perron-Frobenius theorem. The only notable circular element is Section 4's controlled simulation, where the 'ground truth' ranking is constructed from the same easy/medium/hard distinction that RankLLM is designed to estimate, so that experiment confirms internal consistency rather than external validity. Pool-dependence (Fig. 7, Table 5) is a limitation and generalizability concern, not a circularity. Overall the central claim has independent support, so the score reflects one partial, non-central circular validation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

The central claim rests on treating difficulty as a pool-relative latent trait, on a mixed model pool, and on a small human-judgment validation. The algorithm itself is a standard damped random walk with one free hyperparameter (α) whose value is not disclosed.

free parameters (2)

Damping factor α = not reported in main text
Controls the teleportation term in the random walk; all difficulty and competency scores depend on it. Table 8 sweeps α but the experimental results do not state which value was used.
Convergence tolerance ε = not reported
Stops the iteration and affects the reported iteration count (9) and reported convergence times, though not materially the final scores.

axioms (4)

domain assumption Question difficulty is a scalar latent trait recoverable from binary model successes/failures.
The paper operationalizes difficulty purely as the pattern of model failures (§2.1, Eq. 1); no external definition of difficulty is used beyond human validation.
domain assumption The 30-model pool is sufficiently diverse for the stationary distribution to approximate human difficulty judgments.
Fig. 7 shows homogeneous pools give 38.6–64.3% human consensus, so the 90% result depends on the specific mixed pool rather than the method alone.
domain assumption Universally solved or universally failed questions can be excluded without biasing the analysis.
§2.2 removes such questions ('only 2%') and assigns them conceptual extremes; this exclusion is necessary for graph connectivity but could affect score normalization.
domain assumption Human pairwise difficulty judgments on 70 pairs by 20 raters are valid ground truth.
§3.3/H.3 rely on pairwise comparisons; the sample is small, includes two authors, and allows skipping, so the consensus label is a weak external benchmark.

invented entities (2)

Question difficulty score π_q independent evidence
purpose: Latent per-question scalar used to weight model rankings.
Validated against human pairwise difficulty judgments (§3.3), giving a falsifiable external handle.
Model competency score π_m independent evidence
purpose: Difficulty-weighted scalar for ranking LLMs.
Can be checked against held-out human rankings or future benchmark performance, and it reorders models relative to accuracy (§3.2).

pith-pipeline@v1.3.0-alltime-deepseek · 28000 in / 16629 out tokens · 145310 ms · 2026-08-02T23:48:38.261695+00:00 · methodology

0 comments

read the original abstract

Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.

Figures

Figures reproduced from arXiv: 2602.12424 by Kaidi Xu, Kai Zhang, Lichao Sun, Neil Zhenqiang Gong, Qingsong Wen, Ruoxi Chen, Xiangliang Zhang, Xingjian Hu, Yixin Liu, Yue Huang, Ziqian Zhang.

**Figure 2.** Figure 2: Detailed process demonstration of score propagation in RankLLM, which includes Evalua [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: RankLLM scores and accuracies of models, both normalized to a fixed maximum of 100 to [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 5.** Figure 5: The proportion of Easy/Medium/Hard questions within correctly answered samples across Llama/Yi variants. lies in solving difficult questions. In contrast, RankLLM explicitly incorporates quantified question difficulty, enabling a more nuanced evaluation of model performance. The positive correlation between RankLLM scores and accuracy (Kendall’s Tau τ = 0.8492; see Appendix G) aligns with our expectations,… view at source ↗

**Figure 6.** Figure 6: Correlation analysis across model group combinations (Pro. means proprietary, and OW [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

**Figure 8.** Figure 8: Difficulty distribution grouped by model size. A diverse model selection in RankLLM mitigates extremes in difficulty estimation of questions. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗

**Figure 9.** Figure 9: Question difficulty distributions for six benchmarks. Difficulty distribution varies by dataset. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗

**Figure 10.** Figure 10: Impact of Random Model Subset Removal on RankLLM Stability. Top panel shows [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗

**Figure 11.** Figure 11: Distribution of correctly answered questions by models across the whole dataset (leave [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗

**Figure 12.** Figure 12: Absolute model rank difference distribution. The distribution of absolute rank changes (see Figure 12) reveals that while a majority of models experience small shifts (6 models, or 20%, have no rank change, a notable portion shift by 1 or more positions. These larger shifts are particularly interesting as they point to models whose performance on questions of varying difficulty is disproportionately aff… view at source ↗

**Figure 13.** Figure 13: Mean and Maximum Rank Displacement of RankLLM scores compared to Accuracy [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 3 linked inside Pith

[1]

Yi: Open foundation models by 01.ai, 2025

AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhi...

Pith/arXiv arXiv 2025
[2]

Question: What is the value of the inflection point off(x) = 10 lnx x2 ? A. 2.000 B. 1.587 C. 0.693 D. 1.203 E. 3.014 F. 2.718 G. 4.000 H. 3.142 I. 1.000 J. 2.301 Which question is more difficult? Enter 0 if unable to judge, otherwise enter 1 or 2: 32 Published as a conference paper at ICLR 2026 K DISCLOSURE OFLLM USAGE We used large language models solel...

2026
[3]

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V

URLhttps://arxiv.org/abs/2206.04615. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/ abs/2210.09261. Gemini Team, Petko Georgiev, Ving Ia...

Pith/arXiv arXiv 2022
[6]

Question: Evaluate the expression (751−745) + (748−742) + (745−739) + (742−736) +· · ·+ (499−493) + (496−490)
[2023]

Ofir Ben Shoham and Nadav Rappoport

URLhttps://arxiv.org/abs/2311.12022. Ofir Ben Shoham and Nadav Rappoport. Medconceptsqa: Open source medical concepts qa bench- mark, 2024. URLhttps://arxiv.org/abs/2405.07348. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

Pith/arXiv arXiv 2024
[2024]

assess models on multi-disciplinary language understanding and reasoning, while Big-Bench Hard (BBH) (Suzgun et al., 2022) evaluates their multi-step reasoning and algorithmic reasoning abilities. The GSM8K (Cobbe et al., 2021) benchmark tests performance on grade-school-level mathematics, and HumanEval (Chen et al., 2021) measures a model’s capability in...

2022
[2025]

10") that matches an option, respond with the corresponding option letter (e.g.,

are under Apache 2.0.Microsoft Phi Models, such as those described by Abdin et al. (2024), are generally released under the permissive MIT License. Compliance StatementTo the best of the authors’ knowledge, and based on the diligent research of the licenses and terms detailed herein, the use of all software, datasets, benchmarks, and language models in th...

2024

[1] [1]

Yi: Open foundation models by 01.ai, 2025

AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhi...

Pith/arXiv arXiv 2025

[2] [2]

Question: What is the value of the inflection point off(x) = 10 lnx x2 ? A. 2.000 B. 1.587 C. 0.693 D. 1.203 E. 3.014 F. 2.718 G. 4.000 H. 3.142 I. 1.000 J. 2.301 Which question is more difficult? Enter 0 if unable to judge, otherwise enter 1 or 2: 32 Published as a conference paper at ICLR 2026 K DISCLOSURE OFLLM USAGE We used large language models solel...

2026

[3] [3]

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V

URLhttps://arxiv.org/abs/2206.04615. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/ abs/2210.09261. Gemini Team, Petko Georgiev, Ving Ia...

Pith/arXiv arXiv 2022

[4] [6]

Question: Evaluate the expression (751−745) + (748−742) + (745−739) + (742−736) +· · ·+ (499−493) + (496−490)

[5] [2023]

Ofir Ben Shoham and Nadav Rappoport

URLhttps://arxiv.org/abs/2311.12022. Ofir Ben Shoham and Nadav Rappoport. Medconceptsqa: Open source medical concepts qa bench- mark, 2024. URLhttps://arxiv.org/abs/2405.07348. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

Pith/arXiv arXiv 2024

[6] [2024]

assess models on multi-disciplinary language understanding and reasoning, while Big-Bench Hard (BBH) (Suzgun et al., 2022) evaluates their multi-step reasoning and algorithmic reasoning abilities. The GSM8K (Cobbe et al., 2021) benchmark tests performance on grade-school-level mathematics, and HumanEval (Chen et al., 2021) measures a model’s capability in...

2022

[7] [2025]

10") that matches an option, respond with the corresponding option letter (e.g.,

are under Apache 2.0.Microsoft Phi Models, such as those described by Abdin et al. (2024), are generally released under the permissive MIT License. Compliance StatementTo the best of the authors’ knowledge, and based on the diligent research of the licenses and terms detailed herein, the use of all software, datasets, benchmarks, and language models in th...

2024