{"id":"b39296a9-770f-49a2-8a67-8fae7d350999","arxiv_id":"2506.01048","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An IRT-based router that models each LLM's latent ability and each query's difficulty outperforms RouterBench on cost-performance reward across ID and OOD benchmarks.","lead":"This paper applies Item Response Theory, a psychometrics method, to route each user query to the LLM predicted to answer it best at a good price. The router beats a strong benchmark on 12 datasets and also reports model abilities and query difficulties as explanations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-split, no-error-bar evaluation leaves the headline performance gain over RouterBench within sampling error; the 'higher performance' claim is not yet established.","rationale":"The reader's weakest_assumption concerns the sufficiency of BERT embeddings and the MIRT dot-product form (Eq. 4-5). I do not dispute that this is an assumption, but the paper partially addresses it: NIRT-Router (Eq. 7-9) adds a neural interaction layer and performs comparably, and the embedding-model ablation (Table 8) shows BERT is not a bottleneck. Thus the model form is not the weakest link. The weakest link is the lack of uncertainty quantification: the headline numbers come from a single split with no error bars, and the margins over the strongest baseline (RouterBench) are small enough to be within sampling error. The paper's own Table 3 shows Top-1 routing accuracy of only 2.72%, which underscores that the router's score differences are often tiny and the performance outcome could be driven by averaging over many near-ties. A bootstrap or multi-split analysis would settle whether the reported 0.66 pp and 1.62 pp advantages are real. This concern does not change the reader's CONDITIONAL verdict; it reinforces it.","tokens_in":20480,"tokens_out":18170,"duration_ms":181619,"concrete_test":"Re-run the ID and OOD routing evaluation for MIRT-Router and RouterBench at alpha=0.8 over 10 random 70/30 splits (or bootstrap the test set 1000 times), and report the mean and 95% bootstrap CI of the performance difference. If the CI includes 0 in either scenario, the 'higher performance' claim is unsupported; if the CI excludes 0, the claim is corroborated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Tables 1-2, which report a single 70/30 split with no error bars, confidence intervals, or significance tests. At alpha=0.8, MIRT-Router beats RouterBench by 0.66 percentage points on ID (80.67 vs 80.01) and 1.62 points on OOD (87.12 vs 85.50). Given test sizes of roughly 10,470 (ID) and 3,160 (OOD) and proportions near 0.8, the standard error of each difference is about 0.5-0.9 pp, so neither gap reaches the 95% confidence threshold (z about 1.2 and 1.8). The paper reports no variance across seeds or splits despite these small margins, making the 'higher response performance' component of the central claim unverified. The consistent direction across settings is suggestive, but without uncertainty estimates the comparative claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IRT-Router, a multi-LLM routing framework that models each LLM as a latent-ability vector and each query as a difficulty/discrimination item using Item Response Theory. Two variants are presented: MIRT-Router, based on a multidimensional logistic model, and NIRT-Router, based on a neural cognitive diagnosis model with predefined ability dimensions. A semantic-similarity warm-up mechanism is added for cold-start queries. The method is evaluated on 20 LLMs and 12 datasets, with in-distribution and out-of-distribution tests, and compared against RouterBench, HybridLLM, RouteLLM, and single-model baselines. The authors report that IRT-Router achieves higher performance at lower cost than the baselines, and they present interpretability analyses of learned LLM abilities and query difficulties.","tokens_in":20717,"tokens_out":6662,"duration_ms":72478,"significance":"If the empirical claims hold, the paper makes a useful contribution: it brings a principled psychometric model to LLM routing, provides a compact and cheap predictor, and includes an out-of-distribution evaluation, a new-LLM generalization experiment, a warm-up ablation, and publicly released code. These are concrete strengths that go beyond a purely methodological proposal. The main weaknesses are statistical: the headline comparisons rest on a single split without error bars, the margins over RouterBench are small relative to sampling error, and the interpretability evidence is partly anecdotal and partly affected by parameter identifiability. The approach is nevertheless well motivated and the central idea is testable, so the paper merits a major revision rather than rejection.","major_comments":[{"comment":"The headline comparative claim is not statistically supported. All results come from a single 70/30 split with no error bars, confidence intervals, or significance tests. At α=0.8, MIRT-Router exceeds RouterBench by 0.66 percentage points on ID (80.67% vs. 80.01%) and by 1.62 points on OOD (87.12% vs. 85.50%). With ID test size about 10,470 and OOD test size about 3,160, the approximate standard error of each difference is 0.5–1.0 percentage points, so neither gap reaches 95% confidence. Please report results over multiple seeds or splits, or provide significance tests, before claiming that IRT-Router outperforms baselines.","section":"§5, §6, Tables 1–2"},{"comment":"The query-difficulty evidence is internally inconsistent with the model definition. In Eq. (5), P = σ(−a_i^T θ_j + b_i), so a larger b_i makes a query harder. In Fig. 5, the Level 2 query has difficulty −0.047 and the Level 4 query has difficulty −0.117; under Eq. (5) the Level 4 query is therefore easier, contradicting the caption's claim that higher Difficulty means more challenging. Please clarify the sign convention or re-examine the difficulty extraction, because as written the figure does not demonstrate that learned difficulty matches level labels.","section":"§6.2, Fig. 5, Eq. (5)"},{"comment":"The interpretability claims are weakened by lack of identifiability constraints on MIRT-Router's parameters. The predicted probability depends only on the product a_i^T θ_j + b_i, so the transformations θ_j → θ_j + d, b_i → b_i − a_i^T d, and θ_j → c θ_j, a_i → a_i/c leave all predictions unchanged. Without anchor items, regularization, or explicit constraints, the absolute ability values shown in Fig. 4 and the difficulty values in Fig. 5 are not uniquely identified. The authors should state what identifies the scale and location of θ and b, or present the interpretability results as relative rather than absolute.","section":"§4.2.1, Eqs. (4)–(5)"},{"comment":"The warm-up hyperparameter λ is analyzed on the OOD test set itself. Table 9 reports ID and OOD rewards for λ ∈ {0, 0.1, 0.2, 0.3, 0.4} and uses these OOD values to conclude that larger λ improves cold-start generalization. Since the OOD test set is used to select/justify λ, the reported OOD gains of the warm-up mechanism are partially in-sample. Please use a validation split or nested tuning to demonstrate that the warm-up benefit holds for truly unseen queries.","section":"§4.4, Table 9 (Appendix B)"},{"comment":"The Top-k routing accuracy results raise concerns about routing quality. With 20 candidate LLMs, random Top-1 selection has an expected accuracy of 5%, but MIRT-Router reports Top-1 accuracy of only 2.72% on ID and 2.15% on OOD. The explanation that many models have similar scores needs quantitative support: please report the distribution of score gaps to the oracle-best model, the number of near-optimal ties, and Top-k accuracy under a relaxed optimality threshold. As presented, this table does not support the claim that IRT-Router routes effectively.","section":"§6.2, Table 3"}],"minor_comments":[{"comment":"There are typos in the table header ('Iutput') and in the appendix heading ('Cadidate LLMs' should be 'Candidate LLMs').","section":"Table 5, Appendix A.2"},{"comment":"The dimensional notation in the NIRT interaction is confusing: Eq. (7) defines x_ij as a vector but Eq. (8) writes W1 x_ij^T; please clarify the row/column convention and the shapes of W1 and b1.","section":"Eq. (7)–(8)"},{"comment":"The dataset name 'CEV AL' is inconsistent with the reference to C-EVAL; please unify the spelling.","section":"Table 7"},{"comment":"The example query text appears with a rendering artifact ('About:miscellaneousQuestion'); please format the figure caption or example properly.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the central idea is promising, but the statistical evidence for the headline claim is currently insufficient. I would ask the authors to add uncertainty quantification, fix the difficulty-sign inconsistency, address identifiability, and reassess the warm-up evaluation. The very low Top-1 routing accuracy in Table 3 also deserves careful scrutiny before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: IRT-Router is a sensible, well-tested application of Item Response Theory to multi-LLM routing. The cost-efficiency result is solid; the claim that it beats RouterBench on raw performance is not yet established, because the reported margins sit within sampling error. Still, the paper deserves referee time.\n\nWhat's new: the combination of MIRT and NCDM-style neural IRT as the scoring function inside a router. I don't see that in the cited literature—IRT is applied to model evaluation, leaderboards, and LLM capability assessment, and existing routers use BERT classifiers, MF, GNNs, or KNN. The equations are standard psychometrics, so the novelty is in the application, not the math. Code is provided.\n\nWhat it does well: the experimental scope is broad—20 LLMs, 12 datasets, ID and OOD evaluation, a cold-start ablation, and a new-LLM generalization check. The embedding-average warm-up is simple and the ablation shows it helps, more so for NIRT. The limitation section is candid: short benchmark queries, and the router's insensitivity to alpha is acknowledged. They don't oversell.\n\nThe soft spots are real but addressable. All results are from one 70/30 split with no error bars. At alpha=0.8 the ID gap over RouterBench is 0.66 pp and the OOD gap 1.62 pp; with test sizes near 10k and 3k, both are within roughly two standard errors. So 'higher performance' is unverified; 'comparable performance at roughly a third of the cost' is verified and is the stronger practical claim anyway. The warm-up weight lambda is selected on the OOD test set (Table 9), which is test-set leakage; it should be chosen on a validation split or via inner cross-validation. The interpretability evidence is weak: two cherry-picked MATH queries for difficulty and post-hoc narratives for ability vectors, which are fit on training data and then used to explain behavior—not circular for the routing claim, but too thin to support the interpretability headline. Top-1 routing accuracy is 2.7% ID, which they explain by near-ties and cost weighting, but it's low enough to suggest the scoring function is only roughly calibrated. And they never test against a more expressive predictor, so the MIRT dot-product form is a maintained assumption, not a tested one.\n\nNet: the central mechanism—LLMs as test-takers, queries as items—earns its keep as a practical router. The paper needs multiple seeds and error bars, a clean validation split for lambda, and a more systematic interpretability evaluation. If those are added, it's a solid applied paper. Worth sending to peer review.","headline":"A useful, honestly-framed application of IRT to LLM routing with a solid cost-efficiency result, but the headline performance advantage over RouterBench is within sampling error and needs seeds and error bars.","tokens_in":21242,"tokens_out":3132,"would_cite":false,"duration_ms":30724,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An Item-Response-Theory router gives each LLM a 25-dimensional ability and each query a difficulty, routing to the best quality-per-cost model; the paper reports higher performance than GPT-4o alone at 1/30 the cost.","keywords":["LLM routing","Item Response Theory","multidimensional IRT","latent ability","query difficulty","cost-performance trade-off","cold start","interpretability"],"falsifier":"On a held-out set of queries where all 20 candidate LLMs actually respond to every query, compute the rank correlation between IRT-Router's predicted score $S(q_i, M_j)$ and the empirically measured reward of routing $q_i$ to $M_j$; if the correlation is near zero, the logistic IRT interaction is not capturing query-model fit and the routing decisions built on it are miscalibrated.","tokens_in":20291,"feed_emoji":"🧠","tokens_out":13554,"duration_ms":110535,"temperature":0.7,"pith_summary":"The paper tries to establish that a router can treat LLMs as test-takers and queries as test items: each model gets a 25-dimensional latent ability vector, each query gets difficulty and discrimination vectors, and a logistic function predicts how well that model will answer that query. The router then sends each query to the model that maximizes predicted quality minus a fixed cost term. If this works, it matters because routing becomes cheaper and more explainable than always calling the most powerful model, and because the learned ability and difficulty values provide a reason for each routing choice. On the in-distribution test set with performance priority, the paper reports MIRT-Router reaches 80.67% average performance at a total cost of $0.42, where RouterBench reaches 80.01% at $1.15 and GPT-4o alone reaches 77.53% at $12.93. The paper's own limitation section notes that benchmark queries are short and that the router's sensitivity to the performance-versus-cost weight is limited.","feed_headline":"Psychometric router beats GPT-4o's quality at 1/30 the cost","feed_subtitle":"Giving each LLM a 25-point ability profile and each query a difficulty score lets easy questions go to cheap models.","key_machinery":"The load-bearing object is the logistic IRT interaction\n$$\\hat{P}(q_i, M_j) = \\frac{1}{1+\\exp(-a_i^\\top \\theta_{M_j} + b_i)},$$\nwhere $\\theta_{M_j}\\in\\mathbb{R}^{25}$ is the latent ability vector of model $M_j$, produced by a linear map from a BERT embedding of the model's text profile, and $a_i$ and $b_i$ are discrimination and difficulty parameters produced by linear maps from the query's BERT embedding. Predicted performance is combined with a fixed cost $C(M_j)$ through the score $S=\\alpha \\hat{P}-\\beta C$, and the query is sent to the model with the highest score. MIRT-Router uses the logistic form directly; NIRT-Router replaces it with a neural interaction layer that uses a per-query relevance vector, obtained by clustering training queries and asking an LLM which of 25 predefined abilities each cluster needs. A warm-up step blends each new query embedding with the averaged embeddings of its five nearest training neighbors to reduce cold-start error.","core_discovery":"The central claim is that the response quality of an LLM on a query follows an IRT measurement model, and that the parameters of that model can be learned from query embeddings and LLM profile embeddings alone. On the paper's 20-model, 12-dataset evaluation, both MIRT-Router and NIRT-Router outperform the strongest routing baseline, RouterBench, on the performance-cost reward in every tested setting, while spending a fraction of the cost of always using GPT-4o. The learned parameters also carry the interpretability story: ability vectors of larger models dominate smaller ones in most dimensions, learned query difficulty tracks human difficulty labels, and hard queries are routed to stronger models while easy queries go to cheaper, adequate ones.","pith_inferences":["Editorial extension: the learned ability vectors could serve as a 'model zoo' registry, letting a newly released LLM be inserted by embedding its public description alone; the paper evaluates only one held-out LLM and does not test this workflow.","Editorial extension: since the score is a linear combination of predicted performance and a fixed cost, the same router can be repriced for different deployments (for example, free self-hosted models) by changing $C(M_j)$; the paper notes this possibility but does not test it.","Editorial extension: comparing the router against an oracle that always picks the empirically best model for each test query would reveal how much headroom remains; the reported top-1 routing accuracy is low (2.72% in-distribution), suggesting the gap may be substantial.","Editorial extension: the warm-up step is a form of transductive learning, so one could test whether actively selecting a small set of training queries for new domains improves OOD routing more than the fixed k-nearest-neighbor averaging."],"forward_implications":["A service choosing among many LLM APIs can use a lightweight per-query predictor plus fixed-cost scores, avoiding the need to call every candidate model before deciding.","Routing decisions can be explained to users with the model's estimated ability and the query's estimated difficulty, rather than an opaque classifier score.","The warm-up mechanism makes the router usable when new queries arrive online, and the paper reports its out-of-distribution reward increases with a larger blending weight $\\lambda$.","The same framework can be applied to a newly released LLM by encoding its text profile, though the paper reports limited accuracy on one held-out model (ACC 0.67) and treats improvement as future work.","Because the predictor is a compact factor model, its parameters can serve as a reusable map of which models are strong in which ability dimensions."],"supporting_citations":[{"why":"Supplies the Item Response Theory framework and the monotonicity assumption that the router's interpretability rests on.","marker":"Woodruff and Hanson (1996)"},{"why":"Defines multidimensional IRT and the logistic ability-difficulty interaction that MIRT-Router implements.","marker":"Reckase (2009)"},{"why":"NCDM provides the neural interaction and relevance-vector design that NIRT-Router adapts.","marker":"Wang et al. (2020)"},{"why":"RouterBench is the strongest multi-LLM routing baseline and the Predictive Router strategy that IRT-Router is compared against.","marker":"Hu et al. (2024)"},{"why":"GraphRouter supplies the Performance, Total Cost, and Reward metrics used in the evaluation.","marker":"Feng et al. (2024)"},{"why":"HybridLLM is the BERT-based routing baseline that the paper's routers are compared with.","marker":"Ding et al. (2024)"},{"why":"RouteLLM is the binary large-vs-small routing baseline whose setup the paper contrasts with multi-LLM routing.","marker":"Ong et al. (2024)"}],"fun_headline_variants":["Psychometric router: cheap LLMs for easy queries, save 97% cost","IRT-Router: learn LLM ability and query difficulty, then route","Item Response Theory meets LLM routing: interpretable and cost-effective","Route to the right LLM with psychometrics, not just benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The router's predictions stand or fall on the assumption that a single 25-number ability profile per model, plus a per-query difficulty level and a 25-number sensitivity profile, combined through one logistic formula, can predict how well any model will answer any query—including out-of-distribution queries the router never trained on.","fun_headline_variants_meta":{"raw":{"variants":["Psychometric router: cheap LLMs for easy queries, save 97% cost","IRT-Router: learn LLM ability and query difficulty, then route","Item Response Theory meets LLM routing: interpretable and cost-effective","Route to the right LLM with psychometrics, not just benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2297,"prompt_tokens":924,"completion_tokens":1373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1294}},"tokens_in":540,"tokens_out":1373,"duration_ms":11568,"temperature":1.0,"reasoning_tokens":1294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:52:16.475459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of queries where all 20 candidate LLMs actually respond to every query, compute the rank correlation between IRT-Router's predicted score $S(q_i, M_j)$ and the empirically measured reward of routing $q_i$ to $M_j$; if the correlation is near zero, the logistic IRT interaction is not capturing query-model fit and the routing decisions built on it are miscalibrated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Item Response Theory framework and the monotonicity assumption that the router's interpretability rests on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines multidimensional IRT and the logistic ability-difficulty interaction that MIRT-Router implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NCDM provides the neural interaction and relevance-vector design that NIRT-Router adapts."}],"review_version":1}