{"id":"cf90122e-c2df-4ead-8650-273f2f3bbd3e","arxiv_id":"2608.12127","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"This paper introduces an execution-oriented VLM routing benchmark and SCOPE-Router, a cost-aware profile-matching router that achieves the best Rank Score on three benchmarks and improves four existing routers.","lead":"VLM-ExecRouterBench is a new benchmark for routing vision-language models across code, agentic, and search tasks, and SCOPE-Router is a cost-aware router that matches queries to model behavior profiles, allowing new models to join without retraining. The paper reports the best rank scores on three routing benchmarks, but the evidence is empirical and depends on unverified code and data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All rankings rest on a GPT-5.5 semantic judge applied to the hardest samples, with no agreement or per-family bias analysis; a same-family self-preference bias would corrupt the correctness matrix and every downstream Rank Score.","rationale":"I agree with the reader's weakest assumption and sharpen it to the GPT-5.5 judge because it is the most plausible source of model-family bias: the judge is from OpenAI, and two candidate models are also from OpenAI. The Y matrix is the common substrate for correctness matrices, router supervision, Oracle selection, and Rank Score, so a systematic judge bias propagates to every claim in the paper. Other soft spots exist, such as overlapping confidence intervals on VLM-ExecRouterBench and VL-RouterBench and the omission of ICL-Router from open-set comparisons, but those are either acknowledged or clearly scoped. The verification bias is unaddressed and is load-bearing because the benchmark itself is a core contribution. The proposed check is feasible since the authors commit to releasing the data-generation pipeline and verification code. If the check passes, the central claims stand; if it fails, the benchmark rankings and open-set gains must be revisited. The paper's internal consistency, detailed ablations, and cross-router transfer experiments are real strengths, but they do not remove the need for this external validity check. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":22157,"tokens_out":9558,"duration_ms":87951,"concrete_test":"Using the released data pipeline, take a random sample of 500 model outputs from Agentic and Search samples that received GPT-5.5 semantic judging. Re-verify these outputs with (a) an independent judge from a different model family (e.g., Claude Sonnet 4.6 or Gemini 3.5) and (b) two human annotators. Compute label agreement and the per-model-family change in accuracy. If the independent judge or human labels flip more than, say, 2% of GPT-5.4-family correctness labels relative to other families, the correctness matrix is biased and all main-table comparisons must be recomputed. Also report the fraction of samples per dataset that required GPT-5.5 judging so the exposure is quantified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central results (Table 2, Table 4, and the CRM+RCCR transfer gains) are computed from the correctness matrix Y built in Appendix A. For samples where rule-based matching is insufficient, Appendix A states that GPT-5.5 semantic judging is used. The candidate pool includes GPT-5.4 and GPT-5.4 mini, so the judge is same-family as two of the eleven models. LLM self-preference bias is a documented phenomenon: judges often rate outputs from their own training family higher. If GPT-5.5 favors GPT-5.4 outputs, then Y is inflated for these models, which changes the Oracle upper bound, every router's training targets, and the Rank Score rankings. The paper does not report: (i) the fraction of samples judged by GPT-5.5 per dataset, (ii) any agreement study of GPT-5.5 versus rule-based labels, or (iii) per-model accuracy conditioned on judge type. Without these, the benchmark's validity and the claimed 1.84/6.75-point leads are unverified. This is not an accusation of fraud; it is a missing control in the measurement pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses cost-aware routing among vision-language models for execution-oriented tasks. It introduces VLM-ExecRouterBench, a benchmark built from 12 datasets across Code, Agentic, and Search domains with 11 candidate models; SCOPE-Router, a dual-tower router that matches frozen query embeddings to per-model behavior profiles constructed from a hybrid-calibrated 1024-sample set; and CRM+RCCR, a cost-aware training objective that replaces softmax with per-pair sigmoid BCE and adds a consistency regularizer. The authors report that SCOPE-Router achieves the best Rank Score on VLM-ExecRouterBench, VL-RouterBench, and MMR-Bench, that open-set routing under a doubly OOD evaluation gains 6.75 Rank Score points over UniRoute-KM, and that CRM+RCCR improves four existing routers by 1.25-6.21 points. The central risk identified in the report is that the correctness labels used to construct the router targets and to evaluate all methods are partly produced by an unvalidated GPT-5.5 semantic judge.","tokens_in":22419,"tokens_out":6690,"duration_ms":58949,"significance":"If the results hold, the paper would make three useful contributions: an execution-oriented routing benchmark that extends beyond VQA, an open-set profile representation that lets new models join without retraining, and a cost-aware loss that transfers across router architectures. The strengths are concrete: the authors release code and data, the calibration ablations and encoder robustness sweeps are systematic, and the loss-transfer experiment is a useful check of generality. However, the empirical claims rest on a correctness matrix whose judge component is not validated. The verification issue is not a circularity concern, since the training targets are derived from external execution outcomes, but it is an unmeasured bias source that could change model-level rankings and therefore every downstream Rank Score. The significance of the claimed 1.84-6.75 point leads is only as strong as the correctness of that matrix.","major_comments":[{"comment":"The correctness matrix Y that drives Eqs. (9)-(11), the oracle upper bound, and the Rank Score (Eq. 14) is built using GPT-5.5 semantic judging whenever rule-based matching is insufficient, yet the paper reports none of the controls needed to rule out judge bias. GPT-5.4 and GPT-5.4 mini are in the candidate pool, so the judge is same-family as two evaluated models. Please report (i) the fraction of sample-model pairs judged by GPT-5.5 per dataset, (ii) agreement between GPT-5.5 and rule-based labels on a held-out subset, and (iii) per-model accuracy conditioned on judge type. Without these, the claimed leads in Table 2 and Table 4 rest on an unverified measurement pipeline.","section":"Appendix A, Verification and matrix construction"},{"comment":"The main results report every router at the λ achieving the best validation Rank Score, and on VL-RouterBench the lead over RouterDC is 76.18±1.44 versus 74.59±1.05, intervals the authors themselves state overlap. Because the comparison is made at each method's validation-optimal λ, the 'first on all three benchmarks' claim is not supported by a significance test on VL-RouterBench. Provide a fixed-λ comparison or paired significance tests across seeds, and state the number of seeds used for the reported means.","section":"§5.1, Table 2 and evaluation protocol"},{"comment":"The doubly OOD open-set claim of +6.75 Rank Score is reported without any variance or significance information and without a precise protocol statement in the main text. Please give the number of random seeds, whether the held-out model profiles are constructed once or repeatedly, and whether the ID and OOD test sets are disjoint from calibration; if the current implementation is a single run, add intervals or paired comparisons before this result is used as headline evidence.","section":"§5.2, Table 4 (open-set)"}],"minor_comments":[{"comment":"Cross-references between main text and appendix are inconsistent: the OOD paragraph refers to '(Table 1)' for the data split although Table 1 in the main text is the model pool, and the hidden-dimension and freezing ablations refer to 'Table 2' and 'Table 3' that are appendix tables. Renumber or prefix appendix tables to avoid ambiguity.","section":"§5.2 and Appendix E"},{"comment":"The manuscript contains a long unreadable sequence of '/uni000000...' tokens at the top of page 9; this appears to be a rendering artifact and should be deleted.","section":"Page 9, between Figure 3 and Figure 4"},{"comment":"Appendix C states a default λ=10, while §5.1 says each method reports its best validation λ from {0,10,100,1000,10000,+∞}; please state explicitly which λ values were used for the main table entries and whether the same sweep was applied to the baselines.","section":"Appendix C, Training paragraph"}],"recommendation":"major_revision","confidential_remarks":"The most important issue is the unvalidated GPT-5.5 semantic judge in the correctness matrix, not the router method itself. The missing controls are achievable in revision: reporting the judged fraction, an agreement study, and per-family judge-bias analysis. The benchmark and code releases are real assets, and I see no evidence of circularity or fabrication, so the paper is likely salvageable with a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a legitimate addition to the VLM routing literature. VLM-ExecRouterBench is a serious execution-oriented benchmark with real cost data, and the CRM+RCCR loss is a clean idea with demonstrated transfer gains. The soft spot that matters most is the verification pipeline: GPT-5.5 semantic judging is used when rule-based matching is insufficient, and the paper gives no agreement statistics, no fraction of judge-based labels per dataset, and no per-family bias analysis. Since two of the eleven candidate models are GPT-5.4 family members, a same-family self-preference bias could inflate their correctness entries and shift every downstream Rank Score. That is a missing control, not evidence of fraud, but it is load-bearing.\n\nWhat is actually new: the first execution-oriented VLM routing benchmark, spanning code, agentic, and search with 33,966 samples and 11 models. The unified Routing Input / Execution Context / Verification Rule structure is well designed. The hybrid calibration strategy (random/diagnostic/diversity) is original and the ablations show it beats any single strategy. The open-set mechanism is practical: a new model joins after one calibration run, no retraining. The loss ablation is coherent: CRM alone beats softmax, RCCR only helps when built on CRM, and the objective transfers to four existing routers with gains from +1.25 to +6.21 Rank Score points. The encoder-robustness sweep (25 text/vision combinations, less than 0.8 point spread) is reassuring.\n\nSoft spots, in proportion. First, the judge issue above: report the judge fraction, a rule-based agreement study, and per-model accuracy conditioned on judge type. Second, the open-set comparison is limited to UniRoute-KM and UniRoute-LM; ICL-Router is cited but not evaluated. The modality-agnostic justification is reasonable, but the open-set claim is narrower than it looks. Third, all results use the validation-best lambda for every method; that is standard but adds optimism, and the VL-RouterBench lead over RouterDC is within overlapping uncertainty intervals, which the paper honestly admits. Fourth, the GitHub and HuggingFace links lack a commit hash, so the released pipeline cannot be audited at a fixed version.\n\nThe citation pattern looks fine; the paper engages with the routing literature and is upfront in Appendix F about the calibration-set dependence and the single-stage policy limitation.\n\nFor whom: anyone building VLM routing systems or evaluating router losses. The benchmark and the loss are useful independently. It deserves a serious referee. I would accept it for review, with the verification-pipeline analysis as the main, and addressable, condition.","headline":"Solid systems paper with a genuinely useful benchmark and a clean cost-aware loss; the main thing to fix before trusting the numbers is the unexamined GPT-5.5 judge in the correctness pipeline.","tokens_in":22987,"tokens_out":5859,"would_cite":true,"duration_ms":50224,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCOPE-Router matches each query to a calibration-derived model profile in a shared routing space, and the paper claims it achieves the best accuracy–cost Rank Score on three VLM routing benchmarks while letting new models join without…","keywords":["model routing","vision-language models","open-set routing","cost-aware training","execution-oriented benchmark","dual-tower matching","calibration profiles","Rank Score"],"falsifier":"Take a random subset of the benchmark samples, have the same candidate models execute them, and re-verify the outputs with human review instead of the automatic checker and semantic judge. If the human-verification labels disagree with the automatic labels on a pattern that aligns with model provider or cost tier, then the correctness matrix, benchmark rankings, and router comparisons change; observing such a shift would show the labels are not neutral.","tokens_in":21970,"feed_emoji":"🧭","tokens_out":6033,"duration_ms":55334,"temperature":0.7,"pith_summary":"This paper claims that model routing for vision-language models can be extended from question answering to tasks that actually execute—code generation, tool-calling agents, and multi-step web search—and that a router trained for this setting can beat existing routers on accuracy per dollar. To do this, the authors build a benchmark with 11 candidate models whose prices span two orders of magnitude, and a dual-tower router that scores each query by matching it against a behavioral profile of each model. The profile is built by running the model once on a 1024-sample calibration set, so a newly released model can join routing without retraining. The paper further claims its cost-aware loss, a per-pair sigmoid objective with a consistency regularizer, improves four different router architectures by 1.25–6.21 Rank Score points. If these claims hold, VLM deployment can trade a small accuracy loss for roughly 85% lower cost compared with always using the strongest model.","feed_headline":"Cost-aware router tops every VLM routing benchmark","feed_subtitle":"It joins 11 models for code, agents, and search in one pool, cutting cost by 85% versus always picking the strongest model.","key_machinery":"The central object is the query-aware model profile and the shared routing space. Each model gets a profile vector made of a behavioral part (per-sample correctness, normalized cost, a value term, and summary statistics) and a semantic part (aggregated embedding directions for what the model answers correctly, incorrectly, and efficiently). A frozen text-plus-vision encoder embeds the query, a lightweight query MLP and a profile MLP project both into the same 64-dimensional space, and routing is the dot product of query and profile embeddings. Training uses cost-aware relevance matching with independent sigmoid binary cross-entropy, which assigns label 1 to the cheapest correct model and decaying positive labels to more expensive correct models, plus a consistency regularizer that pulls queries with similar routing preferences together in the routing space.","core_discovery":"The central claim is that routing quality and cost can be optimized jointly by treating each candidate model as a query-aware profile rather than a fixed class label, and by training the router with per-pair cost-aware supervision instead of row-wise softmax. On this view, a model is represented by its observed behavior on a small calibration set—correctness, cost, value, and semantic directions—and a query is routed by dot-product matching to these profiles. The paper reports that this yields the best Rank Score on all three benchmarks (80.94, 76.18, and 61.23, first place in each), with a 6.75-point advantage over the runner-up open-set baseline under doubly out-of-distribution evaluation, and that replacing standard losses with its cost-aware objective lifts four existing routers by 1.25–6.21 points. The load-bearing idea is that per-pair independent scoring removes the dilution of multiple correct models that softmax normalization causes, and that cost can be encoded directly into continuous relevance targets.","pith_inferences":["Because the loss is architecture-agnostic, it is a natural candidate for text-only LLM routers and retrieval systems with multiple positive labels; testing it there would show whether the dilution fix generalizes beyond vision-language models.","The calibration-set design suggests a general onboarding protocol for dynamic model marketplaces: keep a small canonical probe set, profile new models on it, and serve immediately; the 50/30/20 sampling recipe could be evaluated as a standard.","The single-decision limitation means the next natural test is trajectory-level routing inside long agentic tasks, where the correct model may change step by step; the paper lists this as its own future work.","The benchmark's Routing Input / Execution Context / Verification Rule schema separates what the router sees from what the model executes, so the same infrastructure could host future routing methods that peek at intermediate tool results."],"forward_implications":["If correct, routing evaluation can move beyond VQA to code, agentic tool use, and retrieval, where model strengths complement each other more strongly.","A model can join a deployed pool by being run once on the calibration set, without retraining the router, so open-set deployment cost is bounded by one small inference run.","The cost-aware objective transfers: existing router implementations can swap in the loss and gain 1.25–6.21 Rank Score points without architectural change.","On the new benchmark, routing with this method cuts cost by roughly 85% versus the strongest single model while sacrificing about 5 points of accuracy.","The dual-tower profile design keeps the router independent of model identities, so the same trained router can be repopulated as the candidate pool changes."],"supporting_citations":[{"why":"supplies the benchmark and Rank Score metric used throughout the paper.","marker":"[2]"},{"why":"provides the dual-contrastive router baseline whose training objective the paper replaces.","marker":"[7]"},{"why":"provides the open-set router baseline that SCOPE-Router is compared against in open-set evaluation.","marker":"[8]"},{"why":"supplies the third evaluation benchmark for multimodal routing.","marker":"[10]"},{"why":"provides an end-to-end router architecture used to test transferability of the proposed loss.","marker":"[12]"},{"why":"supplies the text encoder used for query and profile embeddings.","marker":"[15]"},{"why":"supplies the vision encoder used for query embeddings.","marker":"[16]"}],"fun_headline_variants":["Cost-aware VLM router wins open-set benchmarks, cuts cost 85%","SCOPE-Router: no retraining to add models, wins all three routing tests","Per-pair cost-aware training boosts four routers by 1.25–6.21 Rank Score","Routing by behavior profiles: SCOPE-Router best on code, agents, search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings depend on the labels in the correctness matrix being unbiased: every candidate model is judged by the same verification rules, and when automatic checks are insufficient, by a semantic judge, and if that judge or the execution scaffolding favors some models, both the benchmark and the router inherit the bias.","fun_headline_variants_meta":{"raw":{"variants":["Cost-aware VLM router wins open-set benchmarks, cuts cost 85%","SCOPE-Router: no retraining to add models, wins all three routing tests","Per-pair cost-aware training boosts four routers by 1.25–6.21 Rank Score","Routing by behavior profiles: SCOPE-Router best on code, agents, search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001014,"raw_usage":{"total_tokens":4314,"prompt_tokens":1008,"completion_tokens":3306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":3214}},"tokens_in":624,"tokens_out":3306,"duration_ms":22376,"temperature":1.0,"reasoning_tokens":3214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:15:20.068894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of the benchmark samples, have the same candidate models execute them, and re-verify the outputs with human review instead of the automatic checker and semantic judge. If the human-verification labels disagree with the automatic labels on a pattern that aligns with model provider or cost tier, then the correctness matrix, benchmark rankings, and router comparisons change; observing such a shift would show the labels are not neutral.","supporting_citations":[{"cited_title":"Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024","cited_arxiv_id":null,"evidence_quote":"supplies the text encoder used for query and profile embeddings."},{"cited_title":"Dinov2: Learning robust visual features without supervision, 2023","cited_arxiv_id":null,"evidence_quote":"supplies the vision encoder used for query embeddings."},{"cited_title":"Vl-routerbench: A benchmark for vision-language model routing, 2026","cited_arxiv_id":null,"evidence_quote":"supplies the benchmark and Rank Score metric used throughout the paper."}],"review_version":1}