Pith. sign in

REVIEW 3 major objections 3 minor 48 references

SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks

T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SCOPE-Router matches each query to a calibration-derived model profile in a shared routing space, and the paper claims it achieves the best accuracy–cost Rank Score on three VLM routing benchmarks while letting new models join without…

desk verdict Solid systems paper with a genuinely useful benchmark and a clean cost-aware loss; the main thing to fix before trusting the numbers is the unexamined GPT-5.5 judge in the correctness pipeline. read the letter →

arxiv 2608.12127 v1 pith:Y4GVGDCV submitted 2026-08-12 cs.CV

Tao Yu , Yifei Qu , Zhiqing Cui , Pengfei Zhou , Zhongtian Luo , Yujia Yang , Shenghua Chai , Haopeng Jin
show 9 more authors
Zhenghao Zhang Xinming Wang Hongzhu Yi Wangbo Zhao Zhenglin Wan Yan Huang Yeshani Jinwen Luo Yang You
This is my paper · ORCID
classification cs.CV
keywords modelroutingvision-languagemodelsopen-setcost-awaretrainingexecution-orientedbenchmarkdual-towermatchingcalibrationprofilesRankScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that model routing for vision-language models can be extended from question answering to tasks that actually execute—code generation, tool-calling agents, and multi-step web search—and that a router trained for this setting can beat existing routers on accuracy per dollar. To do this, the authors build a benchmark with 11 candidate models whose prices span two orders of magnitude, and a dual-tower router that scores each query by matching it against a behavioral profile of each model. The profile is built by running the model once on a 1024-sample calibration set, so a newly released model can join routing without retraining. The paper further claims its cost-aware loss, a per-pair sigmoid objective with a consistency regularizer, improves four different router architectures by 1.25–6.21 Rank Score points. If these claims hold, VLM deployment can trade a small accuracy loss for roughly 85% lower cost compared with always using the strongest model.

What carries the argument

The central object is the query-aware model profile and the shared routing space. Each model gets a profile vector made of a behavioral part (per-sample correctness, normalized cost, a value term, and summary statistics) and a semantic part (aggregated embedding directions for what the model answers correctly, incorrectly, and efficiently). A frozen text-plus-vision encoder embeds the query, a lightweight query MLP and a profile MLP project both into the same 64-dimensional space, and routing is the dot product of query and profile embeddings. Training uses cost-aware relevance matching with independent sigmoid binary cross-entropy, which assigns label 1 to the cheapest correct model and decaying positive labels to more expensive correct models, plus a consistency regularizer that pulls queries with similar routing preferences together in the routing space.

What would settle it

Take a random subset of the benchmark samples, have the same candidate models execute them, and re-verify the outputs with human review instead of the automatic checker and semantic judge. If the human-verification labels disagree with the automatic labels on a pattern that aligns with model provider or cost tier, then the correctness matrix, benchmark rankings, and router comparisons change; observing such a shift would show the labels are not neutral.

Watch

Extended reading notes

Core claim

The central claim is that routing quality and cost can be optimized jointly by treating each candidate model as a query-aware profile rather than a fixed class label, and by training the router with per-pair cost-aware supervision instead of row-wise softmax. On this view, a model is represented by its observed behavior on a small calibration set—correctness, cost, value, and semantic directions—and a query is routed by dot-product matching to these profiles. The paper reports that this yields the best Rank Score on all three benchmarks (80.94, 76.18, and 61.23, first place in each), with a 6.75-point advantage over the runner-up open-set baseline under doubly out-of-distribution evaluation, and that replacing standard losses with its cost-aware objective lifts four existing routers by 1.25–6.21 points. The load-bearing idea is that per-pair independent scoring removes the dilution of multiple correct models that softmax normalization causes, and that cost can be encoded directly into continuous relevance targets.

Load-bearing premise

The rankings depend on the labels in the correctness matrix being unbiased: every candidate model is judged by the same verification rules, and when automatic checks are insufficient, by a semantic judge, and if that judge or the execution scaffolding favors some models, both the benchmark and the router inherit the bias.

Editorial extensions

If this is right

  • If correct, routing evaluation can move beyond VQA to code, agentic tool use, and retrieval, where model strengths complement each other more strongly.
  • A model can join a deployed pool by being run once on the calibration set, without retraining the router, so open-set deployment cost is bounded by one small inference run.
  • The cost-aware objective transfers: existing router implementations can swap in the loss and gain 1.25–6.21 Rank Score points without architectural change.
  • On the new benchmark, routing with this method cuts cost by roughly 85% versus the strongest single model while sacrificing about 5 points of accuracy.
  • The dual-tower profile design keeps the router independent of model identities, so the same trained router can be repopulated as the candidate pool changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the loss is architecture-agnostic, it is a natural candidate for text-only LLM routers and retrieval systems with multiple positive labels; testing it there would show whether the dilution fix generalizes beyond vision-language models.
  • The calibration-set design suggests a general onboarding protocol for dynamic model marketplaces: keep a small canonical probe set, profile new models on it, and serve immediately; the 50/30/20 sampling recipe could be evaluated as a standard.
  • The single-decision limitation means the next natural test is trajectory-level routing inside long agentic tasks, where the correct model may change step by step; the paper lists this as its own future work.
  • The benchmark's Routing Input / Execution Context / Verification Rule schema separates what the router sees from what the model executes, so the same infrastructure could host future routing methods that peek at intermediate tool results.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper addresses cost-aware routing among vision-language models for execution-oriented tasks. It introduces VLM-ExecRouterBench, a benchmark built from 12 datasets across Code, Agentic, and Search domains with 11 candidate models; SCOPE-Router, a dual-tower router that matches frozen query embeddings to per-model behavior profiles constructed from a hybrid-calibrated 1024-sample set; and CRM+RCCR, a cost-aware training objective that replaces softmax with per-pair sigmoid BCE and adds a consistency regularizer. The authors report that SCOPE-Router achieves the best Rank Score on VLM-ExecRouterBench, VL-RouterBench, and MMR-Bench, that open-set routing under a doubly OOD evaluation gains 6.75 Rank Score points over UniRoute-KM, and that CRM+RCCR improves four existing routers by 1.25-6.21 points. The central risk identified in the report is that the correctness labels used to construct the router targets and to evaluate all methods are partly produced by an unvalidated GPT-5.5 semantic judge.

Significance. If the results hold, the paper would make three useful contributions: an execution-oriented routing benchmark that extends beyond VQA, an open-set profile representation that lets new models join without retraining, and a cost-aware loss that transfers across router architectures. The strengths are concrete: the authors release code and data, the calibration ablations and encoder robustness sweeps are systematic, and the loss-transfer experiment is a useful check of generality. However, the empirical claims rest on a correctness matrix whose judge component is not validated. The verification issue is not a circularity concern, since the training targets are derived from external execution outcomes, but it is an unmeasured bias source that could change model-level rankings and therefore every downstream Rank Score. The significance of the claimed 1.84-6.75 point leads is only as strong as the correctness of that matrix.

major comments (3)
  1. [Appendix A, Verification and matrix construction] The correctness matrix Y that drives Eqs. (9)-(11), the oracle upper bound, and the Rank Score (Eq. 14) is built using GPT-5.5 semantic judging whenever rule-based matching is insufficient, yet the paper reports none of the controls needed to rule out judge bias. GPT-5.4 and GPT-5.4 mini are in the candidate pool, so the judge is same-family as two evaluated models. Please report (i) the fraction of sample-model pairs judged by GPT-5.5 per dataset, (ii) agreement between GPT-5.5 and rule-based labels on a held-out subset, and (iii) per-model accuracy conditioned on judge type. Without these, the claimed leads in Table 2 and Table 4 rest on an unverified measurement pipeline.
  2. [§5.1, Table 2 and evaluation protocol] The main results report every router at the λ achieving the best validation Rank Score, and on VL-RouterBench the lead over RouterDC is 76.18±1.44 versus 74.59±1.05, intervals the authors themselves state overlap. Because the comparison is made at each method's validation-optimal λ, the 'first on all three benchmarks' claim is not supported by a significance test on VL-RouterBench. Provide a fixed-λ comparison or paired significance tests across seeds, and state the number of seeds used for the reported means.
  3. [§5.2, Table 4 (open-set)] The doubly OOD open-set claim of +6.75 Rank Score is reported without any variance or significance information and without a precise protocol statement in the main text. Please give the number of random seeds, whether the held-out model profiles are constructed once or repeatedly, and whether the ID and OOD test sets are disjoint from calibration; if the current implementation is a single run, add intervals or paired comparisons before this result is used as headline evidence.
minor comments (3)
  1. [§5.2 and Appendix E] Cross-references between main text and appendix are inconsistent: the OOD paragraph refers to '(Table 1)' for the data split although Table 1 in the main text is the model pool, and the hidden-dimension and freezing ablations refer to 'Table 2' and 'Table 3' that are appendix tables. Renumber or prefix appendix tables to avoid ambiguity.
  2. [Page 9, between Figure 3 and Figure 4] The manuscript contains a long unreadable sequence of '/uni000000...' tokens at the top of page 9; this appears to be a rendering artifact and should be deleted.
  3. [Appendix C, Training paragraph] Appendix C states a default λ=10, while §5.1 says each method reports its best validation λ from {0,10,100,1000,10000,+∞}; please state explicitly which λ values were used for the main table entries and whether the same sweep was applied to the baselines.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the benchmark and CRM+RCCR results are self-contained, with only a non-load-bearing same-author citation.

full rationale

The derivation chain is self-contained. Relevance targets R_i,m (Eq. 9) are defined solely from the externally constructed correctness matrix Y and cost matrix C; the CRM loss (Eq. 10) fits router logits to these fixed targets, and the Rank Score (Eq. 14) is computed from actual test-time routing selections, so no training target is defined by the router's own outputs and no prediction reduces to an input by construction. Open-set profiles (Eqs. 3-5) are built from calibration-set correctness/cost labels only; test queries are not used in profile construction, and held-out models are profiled on the same fixed calibration set only after router training, matching the declared protocol. The only same-author citation found is [6] (Agent-as-a-router), used in the introduction as a general routing motivation; no central claim, uniqueness theorem, or training objective is justified by it, so it is not load-bearing. The Appendix A use of GPT-5.5 semantic judging when rule-based matching is insufficient is a measurement-validity risk (same-family judge for GPT-5.4 candidates, with no reported agreement or per-family bias analysis), but that threatens benchmark validity rather than constituting a circular derivation. The cost-sensitivity coefficient lambda is selected on validation splits, which is standard model selection. Overall, no circular reduction was identified.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

All claims rest on the correctness and cost matrices generated by the authors' execution pipeline and on the representativeness of the 1024-sample calibration set. These are empirical domain assumptions rather than mathematical axioms; the router itself introduces hyperparameters such as lambda, calibration mix, loss weight, and architecture size that are tuned on validation data.

free parameters (6)
  • lambda (cost sensitivity in relevance target) = swept over {0, 10, 100, 1000, 10000, infinity}; best validation selected
    Controls how quickly relevance decays as the cost of a correct model rises; headline results use the validation-best value.
  • Calibration mixture percentages (random/diagnostic/diversity) = 50% / 30% / 20%
    Chosen by the authors; the full combination outperforms individual strategies in the ablation, but the ratio itself is hand-set.
  • Diagnostic sampling weights w_dis, w_cost, and temperature tau_samp = 0.7, 0.3, 0.6
    Parameters of the diagnostic sampling score in Eq. (2); they determine which calibration samples are selected.
  • RCCR loss weight mu = 0.1
    Balances CRM and RCCR in the total loss; only the default is reported.
  • MLP hidden dimension = 128
    Chosen by an ablation over 64 to 4096; Rank Scores span only 0.62 points, so sensitivity is low.
  • Score temperature tau and RCCR temperature tau_s = not stated
    Appear in Eqs. (8) and (11), but their values are not reported, so we cannot tell whether they are tuned.
assumptions (4)
  • domain assumption A 1024-sample calibration set drawn from the training distribution is representative enough to profile unseen models.
    Open-set onboarding and profile discriminability depend on calibration samples exposing model strengths and failure modes; the paper acknowledges this in Appendix F but does not validate representativeness externally.
  • domain assumption The verification pipeline assigns correct binary labels, including GPT-5.5 semantic judging when rule-based matching is insufficient.
    All training targets, relevance values, and evaluation metrics derive from the correctness matrix Y; judge bias would propagate everywhere.
  • domain assumption API token pricing and measured token usage are the right cost model for routing decisions.
    The cost matrix C and the Rank Score depend on this; latency and other operational costs are ignored.
  • domain assumption L2-normalized dot-product compatibility in a shared routing space captures query-model suitability.
    The matching score in Eq. (8) defines what the router learns; other similarity functions could change results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks." pith.science (2026). https://pith.science/paper/Y4GVGDCV

@misc{pith2026260812127,
  author       = {Pith},
  title        = {Pith review of: SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y4GVGDCV}},
  note         = {Machine review of arXiv:2608.12127}
}
read the original abstract

Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost. Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios, and employs training objectives that dilute multi-positive signals via softmax normalization without incorporating cost. We address these limitations with three contributions: (1)VLM-ExecRouterBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains with 11 candidate models spanning nearly two orders of magnitude in pricing; (2)SCOPE-Router, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration (random/diagnostic/diversity sampling), enabling new models to join routing without retraining; (3)CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space. Empirically, SCOPE-Router achieves the best Rank Score on all three benchmarks, surpassing the runner-up by 1.84 points under OOD settings and by 6.75 points under doubly OOD open-set evaluation. When applied to four diverse routers, CRM+RCCR improves Rank Score by 1.25--6.21 points.

Figures

Figures reproduced from arXiv: 2608.12127 by the authors.

Figure 1
Figure 1. Overview of the proposed SCOPE-Router framework. Left: router data preparation constructs unified execution samples, correctness and cost matrices, and query-aware model profiles using the hybrid calibration set. Middle: during router training, the query and model-profile branches are projected into a shared routing space and jointly optimized using Cost-aware Relevance Matching (CRM) and Routing-Consistency Contras… view at source ↗
Figure 2
Figure 2. Data composition of VLM-ExecRouterBench. Candidate Model Pool. # Model Provider Input ($) Output ($) 1 Qwen3-VL-8B-Instruct Qwen / Alibaba 0.08 0.50 2 Gemini 2.5 Flash Lite Google 0.10 0.40 3 Qwen3.5-35B-A3B Qwen / Alibaba 0.14 1.00 4 MiniMax M3 MiniMax 0.30 1.20 5 Gemini 3 Flash Preview Google 0.50 3.00 6 GPT-5.4 mini OpenAI 0.75 4.50 7 Claude Haiku 4.5 Anthropic 1.00 5.00 8 Gemini 3.5 Flash Google 1.50 9.00 9 GPT-… view at source ↗
Figure 3
Figure 3. Transferability of our cost-aware loss across different end-to-end routers on VL-RouterBench and VLM￾ExecRouterBench. Applying our loss consistently improves the overall routing performance across RouterDC, ZOOTER, CosineCls, and VLC [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Accuracy–cost trade-offs on VL-RouterBench. Curves trace each router’s Pareto front across λ; points nearer the upper-left are better. On VLM-ExecRouterBench, SCOPE-Router obtains a Rank Score of 80.94 ± 1.22, followed by CosineCls (79.55 ± 0.50). Compared with the Str…
Figure 1
Figure 1. Figure 1: Text and vision encoder selection across 25 encoder combinations. Rank Scores span less than 0.8 points, demonstrating robustness to encoder choice. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_1.png]
Figure 2
Figure 2. Figure 2: Per-dataset accuracy of each candidate model on VLM-ExecRouterBench. The heatmap exposes complementary model strengths across code, multimodal visual reasoning, and search-style datasets. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_2.png]
Figure 3
Figure 3. Figure 3: Model strength profiles aggregated by dataset group. The radar plot highlights that candidate VLMs specialize differently across code, multimodal visual reasoning, and search-style tasks. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_3.png]
Figure 4
Figure 4. Figure 4: Accuracy–cost distribution of individual candidate VLMs and the oracle reference computed from the final routing matrix. The plot illustrates that higher global accuracy often comes with substantially higher execution cost, motivating cost-aware routing. 25 [PITH_FULL…
Figure 5
Figure 5. Figure 5: Overview of the VLM-ExecRouterBench data-generation pipeline. Heterogeneous Code, Agentic, and Search source tasks are standardized into executor inputs with the appropriate modalities and optional tool schemas. Candidate VLMs produce responses through the shared execu…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages

  1. [1]

    Routerbench: A benchmark for multi-llm routing system, 2024

    Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system, 2024. URL https://arxiv.org/abs/2403.12031

    arXiv 2024

  2. [2]

    Vl-routerbench: A benchmark for vision-language model routing, 2026

    Zhehao Huang, Baijiong Lin, Jingyuan Zhang, Jingying Wang, Yuhang Liu, Ning Lu, Tao Li, and Xiaolin Huang. Vl-routerbench: A benchmark for vision-language model routing, 2026. URL https: //arxiv.org/abs/2512.23562

    work page arXiv 2026

  3. [3]

    Routerarena: An open platform for comprehensive comparison of llm routers, 2025

    Yifan Lu, Rixin Liu, Jiayi Yuan, Xingqi Cui, Shenrun Zhang, Hongyi Liu, and Jiarong Xing. Routerarena: An open platform for comprehensive comparison of llm routers, 2025. URL https://arxiv.org/abs/ 2510.00202

    arXiv 2025

  4. [4]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms with preference data, 2025. URL https: //arxiv.org/abs/2406.18665

    arXiv 2025

  5. [5]

    Carrot: A cost aware rate optimal router, 2025

    Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity. Carrot: A cost aware rate optimal router, 2025. URL https://arxiv.org/abs/2502.03261

    arXiv 2025

  6. [6]

    Agent-as-a-router: Agentic model routing for coding tasks

    Pengfei Zhou, Zhiwei Tang, Yixing Ma, Jiasheng Tang, Yizeng Han, Zhenglin Wan, Fanqing Meng, Wei Wang, Bohan Zhuang, Wangbo Zhao, et al. Agent-as-a-router: Agentic model routing for coding tasks. arXiv preprint arXiv:2606.22902, 2026

    arXiv 2026

  7. [7]

    Kwok, and Yu Zhang

    Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Yu Zhang. Routerdc: Query-based router by dual contrastive learning for assembling large language models, 2024. URL https://arxiv.org/abs/ 2409.19886

    arXiv 2024

  8. [8]

    Universal model routing for efficient llm inference, 2025

    Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Congchao Wang, Zifeng Wang, Alec Go, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal model routing for efficient llm inference, 2025. URL https://arxiv.org/abs/ 2502.08773

    arXiv 2025

Show all 48 references
  1. [9]

    xrouter: Training cost-aware llms orchestration system via reinforcement learning, 2025

    Cheng Qian, Zuxin Liu, Shirley Kokane, Akshara Prabhakar, Jielin Qiu, Haolin Chen, Zhiwei Liu, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. xrouter: Training cost-aware llms orchestration system via reinforcement learning, 2025. URL http...

    2025

  2. [10]

    Mmr-bench: A comprehensive benchmark for multimodal llm routing, 2026

    Haoxuan Ma, Guannan Lai, and Han-Jia Ye. Mmr-bench: A comprehensive benchmark for multimodal llm routing, 2026. URL https://arxiv.org/abs/2601.17814

    2026

  3. [11]

    Icl-router: In-context learned model representations for llm routing, 2025

    Chenxu Wang, Hao Li, Yiqun Zhang, Linyao Chen, Jianhao Chen, Ping Jian, Peng Ye, Qiaosheng Zhang, and Shuyue Hu. Icl-router: In-context learned model representations for llm routing, 2025. URL https://arxiv.org/abs/2510.09719. 11 SCOPE-Router & VLM-ExecRouterBench

    2025

  4. [12]

    Routing to the expert: Efficient reward-guided ensemble of large language models, 2023

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models, 2023. URL https: //arxiv.org/abs/2311.08692

    2023 arXiv

  5. [13]

    Llmrouterbench: A massive benchmark and unified framework for llm routing, 2026

    Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, and Shuyue Hu. Llmrouterbench: A massive benchmark and unified framework for llm routing, 2026. URL https://arxiv.org/abs/2601.07206

    2026

  6. [14]

    Routeprofile: Graph-based profiling for cold-start llm routing, 2026

    Jingjun Xu, Hongji Pu, Tao Feng, Haozhen Zhang, Jiaxuan You, and Ge Liu. Routeprofile: Graph-based profiling for cold-start llm routing, 2026. URL https://arxiv.org/abs/2605.00180

    2026 arXiv

  7. [15]

    Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    2024

  8. [16]

    Dinov2: Learning robust visual features without supervision, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabba...

    2023

  9. [17]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732

    2021 arXiv

  10. [18]

    Big- codebench: Benchmarking code generation with diverse function calls and complex instructions, 2025

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Ya...

    2025 arXiv

  11. [19]

    Measuring coding challenge competence with apps, 2021

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021. URL https://arxiv.org/abs/2105.09938

    2021 arXiv

  12. [20]

    Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974

    2024 arXiv

  13. [21]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255

    2024 arXiv

  14. [22]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL https://arxiv.org/abs/ 2203.10244

    2022 arXiv

  15. [23]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024 arXiv

  16. [24]

    Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12), December 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12), December 2024. ISSN 1869-1919. doi: ...

    2024 doi

  17. [25]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021. URL https://arxiv.org/abs/2007.00398

    2021 arXiv

  18. [26]

    A diagram is worth a dozen images, 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016. URL https://arxiv.org/abs/1603.07396

    2016 arXiv

  19. [27]

    Grok-1.5 vision preview

    xAI. Grok-1.5 vision preview. https://x.ai/news/grok-1.5v, April 2024. Introduces the RealWorldQA benchmark

    2024

  20. [28]

    Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent, 2025

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-plus:...

    2025 arXiv

  21. [29]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junya...

    2025 arXiv

  22. [30]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog? id=qwen3.5

    2026

  23. [31]

    Gemini 2.5 Flash Model Card

    Google DeepMind. Gemini 2.5 Flash Model Card. Model Card, 2025. URL https://storage.googleapis. com/deepmind-media/Model-Cards/Gemini-2-5-Flash-Model-Card.pdf

    2025

  24. [32]

    Gemini 3 Flash Model Card

    Google DeepMind. Gemini 3 Flash Model Card. Model Card, December 2025. URL https://deepmind. google/models/model-cards/gemini-3-flash/

    2025

  25. [33]

    Gemini 3.5 Flash Model Card

    Google DeepMind. Gemini 3.5 Flash Model Card. Model Card, May 2026. URL https://deepmind. google/models/model-cards/gemini-3-5-flash/

    2026

  26. [34]

    Introducing GPT-5.4, March 2026

    OpenAI. Introducing GPT-5.4, March 2026. URL https://openai.com/index/introducing-gpt-5-4/

    2026

  27. [35]

    Introducing GPT-5.4 mini and nano, March 2026

    OpenAI. Introducing GPT-5.4 mini and nano, March 2026. URL https://openai.com/index/ introducing-gpt-5-4-mini-and-nano/

    2026

  28. [36]

    System card: Claude Haiku 4.5

    Anthropic. System card: Claude Haiku 4.5. System Card, October 2025. URL https://www.anthropic. com/claude-haiku-4-5-system-card

    2025

  29. [37]

    System card: Claude Sonnet 4.6

    Anthropic. System card: Claude Sonnet 4.6. System Card, February 2026. URL https://www.anthropic. com/claude-sonnet-4-6-system-card

    2026

  30. [38]

    System card: Claude Opus 4.6

    Anthropic. System card: Claude Opus 4.6. System Card, February 2026. URL https://www.anthropic. com/claude-opus-4-6-system-card

    2026

  31. [39]

    Minimax sparse attention, 2026

    Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou, and Pengyu Zhao. Minimax sparse attention, 2026. URL https://arxiv.org/abs/2606.13392

    2026 arXiv

  32. [40]

    Evelyn Fix and Joseph L. Hodges. Discriminatory analysis—nonparametric discrimination: Consistency properties. Technical Report Project 21-49-004, Report No. 4, USAF School of Aviation Medicine, Randolph Field, Texas, February 1951. 13 SCOPE-Router & VLM-ExecRouterBench

    1951

  33. [41]

    MacQueen

    James B. MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 281–297. University of California Press, 1967

    1967

  34. [42]

    Conditional logit analysis of qualitative choice behavior

    Daniel McFadden. Conditional logit analysis of qualitative choice behavior. In Paul Zarembka, editor, Frontiers in Econometrics, pages 105–142. Academic Press, New York, 1974

    1974

  35. [43]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors.Nature, 323(6088):533–536, 1986. doi: 10.1038/323533a0

    1986 doi

  36. [44]

    Rifkin and Aldebaro Klautau

    Ryan M. Rifkin and Aldebaro Klautau. In defense of one-vs-all classification.J. Mach. Learn. Res., 5: 101–141, 2004

    2004

  37. [45]

    Pp-ocrv6: From 1.5m to 34.5m parameters, surpassing billion-scale vlms on ocr tasks,

    Yubo Zhang, Xueqing Wang, Manhui Lin, Yue Zhang, Penglongyi Deng, Ting Sun, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Changda Zhou, Hongen Liu, Suyin Liang, Cheng Cui, Yi Liu, Dianhai Yu, and Yanjun Ma. Pp-ocrv6: From 1.5m to 34.5m parameters, surpassing billion-scale vlms on ocr tasks,

  38. [46]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    2025 arXiv

  39. [47]

    GPT-5.5 System Card

    OpenAI. GPT-5.5 System Card. https://openai.com/index/gpt-5-5-system-card/, April 2026. Accessed: 2026-07-29. 14 SCOPE-Router & VLM-ExecRouterBench Appendix A Data Generation 15 B Source Datasets 16 B.1 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

    2026

  40. [2026]

    URL https://arxiv.org/abs/2606.13108

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.