{"id":"3c995b80-b2cf-40bb-ad6f-591c38c748e0","arxiv_id":"2607.08940","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A heterogeneous-graph router jointly selects the optimal modality (text, vision, or both) and model per time series query, beating prior routing baselines and generalizing to unseen models and tasks.","lead":"TSRouter is a graph-based system that chooses, for each time series question, both the best input format (text, chart, or both) and the best AI model, improving accuracy by 16–46% over existing routers on four reasoning benchmarks. It matters because it offers a practical way to get higher accuracy from off-the-shelf LLMs and VLMs at controlled cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot transfer to unseen models is asserted from a single pair of new models; the GNN has no correctness supervision for those models, so the plug-and-play claim is under-validated.","rationale":"I agree with the reader's weakest_assumption. The central claim has two components: (1) in-distribution routing superiority on TSRBench, supported by Table 3 with a 16.5% relative gain over Hybrid LLM; (2) zero-shot plug-and-play generalization to unseen models and tasks, supported by Figure 3 and Table 4. Component (2) is the more novel and more fragile. The architectural mechanism for a new model is an inserted node whose only information is a text profile and message-passing from query/modality neighbors; no correctness observation for that model ever enters Eq. (7). Whether this generalizes is an empirical question, and the paper provides only two new models and one aggregate accuracy. This does not invalidate the in-distribution results or the framework, but it means the strong generalization claim requires a stricter validation than a single pair of models. The reader's conditional verdict is appropriate; no adjustment is needed, but the concern should be resolved before strong claims of plug-and-play are made.","tokens_in":19356,"tokens_out":8982,"duration_ms":97248,"concrete_test":"Run leave-one-model-out cross-validation on the six original models. For each held-out model m: remove all correctness labels involving m from the training set, retrain TSRouter on the remaining five models, insert m as an unseen node exactly as in the Figure 3 protocol, and record overall accuracy on TSRBench. Then average over the six folds and compare with (i) 51.33% (no new model) and (ii) an oracle that knows m's correctness. Also report the fraction of queries routed to m in each fold. If the average gain over 51.33% is not positive in most folds, or is concentrated in the largest models, the §3.3 zero-shot claim is not supported; if gains are consistent across folds, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main novelty beyond GraphRouter-style routing is the claim in §2.3 and §3.3 that TSRouter can route to a model it has never observed, using only its text profile and messages from graph neighbors. For a new model node there are no correctness labels in the training objective (Eq. 7), so the GNN must learn a transferable mapping from model descriptions plus query features to per-query correctness. The only evidence is Figure 3: adding Qwen3.5-397B-A17B and Kimi-K2.5 raises overall accuracy from 51.3 to 53.5. That single aggregate number does not rule out the possibility that the two chosen models are easy to rank because their size/price makes them almost always beneficial, or that the gain is an artifact of the particular text profiles used for those two models. If the text-profile-to-capability transfer fails for less distinctive models, the central 'plug-and-play generalization' claim collapses even though the in-distribution routing results in Table 3 may remain valid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TSRouter, a heterogeneous-graph router that jointly selects an input modality (text, vision, or both) and a foundation model (LLM or VLM) for each time-series reasoning query. Node features are initialized from LLM-generated text profiles of tasks, queries, modalities, and models; an HGT backbone performs message passing over five relation types; and candidate scoring is trained with soft targets derived from oracle effectiveness (Eq. 1). Experiments on TSRBench report 51.33% overall accuracy versus 44.07% for the best baseline (Table 3), plus claimed zero-shot generalization to two unseen models (Figure 3) and two held-out tasks (Table 4), with additional ablations, cost-preference sweeps, and efficiency measurements.","tokens_in":19706,"tokens_out":5896,"duration_ms":67926,"significance":"If the empirical claims hold, TSRouter makes a useful contribution: it is the first routing framework I am aware of that jointly optimizes modality and model selection for time-series reasoning, and the graph formulation provides a natural way to inject new models and tasks at test time. The paper is clearly written, the problem formulation is sound, and the authors include code, hyperparameter sensitivity experiments, component ablations, and an efficiency analysis. The central risk is not in the architecture but in the validation: the headline comparisons and the zero-shot generalization claims rest on single runs, an unspecified correctness criterion, and very thin evidence for the unseen-model setting.","major_comments":[{"comment":"No error bars or significance tests are reported for any of the main empirical claims. Table 7 shows that TSRouter itself has standard deviations of 0.2–0.5 accuracy points across 3 runs, so differences such as Perception 67.63 vs. 62.58 or Decision 42.45 vs. 39.62 are not obviously outside run-to-run noise. Please report means±std over multiple runs (or seeds) for the key tables and figures, state the API decoding temperature/sampling, and run an appropriate significance test. Without this, the central claim of 'substantially outperforms' is not quantitatively supported.","section":"§3.1, Table 3, Figure 3, Table 4"},{"comment":"The correctness criterion for TSRBench answers is never defined. Eq. (1) and the soft training targets in Eq. (5) depend on the oracle binary correctness I(q,c) for every candidate, so the router's supervision is only as reliable as this labeling. Specify how responses are scored (exact string match, rubric, multiple-choice extraction, partial credit, handling of abstentions). For the imputation task, which is evaluated with MSE/MAE, also state how correctness is defined if the router is trained on this task, or clarify that the router is applied zero-shot with an objective different from the evaluation metric.","section":"§3.1 Evaluation Metrics"},{"comment":"Baseline implementations and hyperparameter tuning are not described. Adapting methods such as GraphRouter, which was designed for query–model edge prediction, to joint modality–model candidate selection is nontrivial, and treating each (modality, model) pair as an independent candidate may not be the intended use of those baselines. Without reporting search ranges, final hyperparameters, or using the authors' released implementations, the 16–46% relative improvements may partly reflect undertuned baselines. Please provide this information.","section":"§3.1 Baselines"},{"comment":"The zero-shot plug-and-play claim for unseen models is under-validated. Only two unseen models (one LLM, one VLM) are tested, and the only evidence is the aggregate overall accuracy moving from 51.3% to 53.5%. For these new model nodes, Eq. (7) provides no correctness supervision, so the method's success depends on a transferable mapping from text profiles plus graph-neighbor messages to per-query effectiveness. A single aggregate number does not rule out that these two particular models are easy to rank (e.g., because their scale and cost make them almost always beneficial) or that the result depends on their specific text profiles. Please add per-task results for the unseen models, test with additional unseen models of varying distinctiveness, and include a diagnostic such as the rank correlation between predicted and actual per-query effectiveness for the unseen models.","section":"§2.3, Figure 3"}],"minor_comments":[{"comment":"The '16% to 46% relative improvements' are not tied to specific baselines; the 16% figure matches the overall comparison to Hybrid LLM, while 46% matches the comparison to the largest-LLM rule. Please state the reference point.","section":"Abstract"},{"comment":"The notation 'h_q d (h_m + h_l)' contains a stray 'd'; clarify that denotes element-wise product or another defined operator.","section":"Eq. (4)"},{"comment":"The terms 'smallest' and 'largest' model are used without specifying which models they refer to among the candidate set; clarify for each modality.","section":"§2.1, Figure 1(b)"},{"comment":"The paper alternates between 'TSRouter' (figures, Table 5) and 'TSROUTER' (text, tables); use one consistent spelling throughout.","section":"Table 3 and elsewhere"},{"comment":"Cost is reported in total USD per test set; clarify whether this includes routing overhead (e.g., GNN inference, embedding calls) and whether it is averaged over the test set, since per-task costs in the table sum to the overall cost.","section":"Table 3 cost"},{"comment":"This is the only table with standard deviations; consider moving this practice to the main tables or explicitly stating the number of runs for the main experiments.","section":"Appendix A.3, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and likely of interest to the venue. The main risk is empirical: the lack of error bars and an unspecified correctness criterion make the headline numbers hard to trust, and the zero-shot new-model claim rests on very thin evidence. These issues are addressable without changing the method, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Joint modality-model routing is genuinely new: existing routers pick among models under a fixed modality, while TSRouter selects both modality and model together, using a heterogeneous graph with query-query, query-modality, and query-model edges. The in-distribution results are strong: 16-46% relative gains over the best prior router at comparable cost, and the ablations show each component earns its keep. The motivating analysis genuinely supports the complementary-modality narrative. The cost-accuracy Pareto curves are practical. Soft spots: no error bars or significance tests in the main tables; baseline hyperparameter tuning is not described; the correctness criterion for TSRBench answers is unspecified. The zero-shot plug-and-play claim is the weakest part. For the two unseen models, the training objective has no correctness supervision, and the only evidence is one aggregate number: overall accuracy goes from 51.3 to 53.5. Those two models are both large and expensive, so the gain may just be 'bigger is often better,' not a learned text-profile-to-capability transfer. If transfer fails for less distinctive new models, the plug-and-play claim collapses, though the in-distribution routing remains valid. This is worth a serious referee: the core contribution is solid enough for revision, but the authors need to add error bars, clarify the evaluation protocol, and strengthen the zero-shot evidence with more models and a negative control. The reader's take is fair; my main disagreement is that the zero-shot concern, while real, does not sink the paper's central in-distribution result.","headline":"Joint modality-model routing is a real advance and the in-distribution results are solid, but the zero-shot new-model claim rests on a single pair of models and one aggregate number.","tokens_in":839,"tokens_out":1570,"would_cite":true,"duration_ms":37084,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TSROUTER claims that joint modality–model routing on a heterogeneous graph is the right way to answer time-series reasoning queries, reporting 51.33% accuracy on TSRBench.","keywords":["time series reasoning","model routing","modality selection","heterogeneous graph","LLM routing","VLM routing","cost-aware routing","zero-shot generalization"],"falsifier":"Insert a deliberately misdescribed unseen model into the graph—for example, a weak vision model profiled as a strong numerical reasoner—and check whether it receives the same share of queries as an accurately profiled model with identical true capabilities. If routing accuracy collapses when descriptions are corrupted, the result depends on the text prior; if it does not, the graph structure rather than the profiles is doing the work, and a second test (removing all query-query edges) would show whether the claimed generalization survives without query similarity.","tokens_in":19304,"feed_emoji":"📈","tokens_out":5806,"duration_ms":53670,"temperature":0.7,"pith_summary":"TSROUTER is a routing framework that decides, per time-series query, which input modality—textual numbers, visual chart, or both—and which off-the-shelf language or vision-language model should produce the answer. The paper's premise is that no single modality or model dominates: text preserves numerical precision but misses global shape, vision captures patterns but loses fine detail, and smaller and larger models succeed on largely disjoint sets of queries. To exploit this complementarity, TSROUTER builds a heterogeneous graph whose nodes are tasks, queries, modalities, and models, and scores each modality–model candidate against a user-weighted performance–cost objective. On the four-task TSRBench suite, it reports 51.33% overall accuracy versus 44.07% for the best prior router, 16–46% relative gains over all baselines, and zero-shot generalization to unseen models and two held-out reasoning tasks.","feed_headline":"Graph router lifts time-series reasoning accuracy to 51.33%","feed_subtitle":"Per-query modality-model scoring beats best prior router by 7.3 points and generalizes to unseen models and tasks.","key_machinery":"The central object is the heterogeneous graph G=(V,E) with four node types—task, query, modality, and model—and five edge types: task–query, query–modality, modality–model, query–model, and query–query k-nearest-neighbor edges. Node features are text embeddings of natural-language profiles: task descriptions, question text concatenated with time-series statistics, modality property descriptions, and model capability-and-cost profiles. A two-layer Heterogeneous Graph Transformer (a graph network that updates each node's embedding by attending over type-specific neighbors) propagates information across these edges. A candidate's score for a query is the sum of the modality and model embeddings","core_discovery":"On the paper's own terms, the central discovery is that joint modality-model selection for time series reasoning can be framed as candidate scoring over a heterogeneous graph, and that this framing transfers to models and tasks never seen in training. The load-bearing comparison is Table 3: TSROUTER reaches 51.33% overall accuracy on TSRBench while the best competing router reaches 44.07%, at comparable API cost. The authors attribute the gap to the graph's ability to propagate contextual signals across task, query, modality, and model nodes, and to soft-target training over the full ranking of candidates rather than a single best label.","pith_inferences":["Beyond the paper: the same heterogeneous scoring recipe should transfer to other settings where two complementary input views and a zoo of models coexist—for example, code tasks (text vs. execution traces) or medical charts (numeric tables vs. images).","Beyond the paper: because query-query edges are built from embedding similarity, the router's ceiling likely depends on the embedding model; a testable extension is to replace the fixed text embedder with a task-finetuned similarity and measure whether routing accuracy rises, especially at low training-data fractions.","Beyond the paper: the zero-shot new-model result implies the text profile is doing real work; deliberately corrupting a new model's description (e.g., swapping capability claims) would reveal whether the GNN relies on semantic priors or on structural position in the graph.","Beyond the paper: the cost term is a single scalar normalized across candidates; using latency, energy, or a nonlinear cost in the same objective should yield a different Pareto front, which would show whether the framework's cost-accuracy control is robust to the choice of cost definition."],"forward_implications":["If the central claim is right, systems can route each time-series query to the best modality-model pair and obtain 16–46% relative accuracy improvements over existing routers at comparable or lower cost.","Newly released models can be inserted into the router by adding a node and a text profile; the paper reports that adding two unseen models raised overall accuracy from 51.3% to 53.5%, while several baselines degraded.","The router transfers to unseen task types: on correlation prediction it reaches 31.38% accuracy versus 29.59% for the best baseline, and on imputation it reports lower MSE (0.56 vs. 0.60) and MAE (0.43 vs. 0.46).","The explicit cost coefficient α gives operators a tunable accuracy-cost frontier; the paper shows that at the cost budget of the best baseline, TSROUTER still exceeds all baselines in accuracy.","The ablations show the heterogeneous structure, query-query edges, modality-model edges, and the MLP scoring head each contribute; removing any one drops overall accuracy by roughly 2.8 to 5 points."],"fun_headline_variants":["Graph router fuses LLM and VLM strengths for time-series","Per-query model selection lifts time-series accuracy to 51.33%","Time-series reasoning: graph router generalizes to unseen models","Dynamic routing cuts cost while holding accuracy on time-series"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the graph network can estimate a completely unseen model's per-query effectiveness from its text description and graph-neighbor messages alone, without ever observing that model's correctness; if descriptions and message passing cannot transfer to genuinely new models, the zero-shot plug-and-play claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Graph router fuses LLM and VLM strengths for time-series","Per-query model selection lifts time-series accuracy to 51.33%","Time-series reasoning: graph router generalizes to unseen models","Dynamic routing cuts cost while holding accuracy on time-series"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1589,"prompt_tokens":786,"completion_tokens":803,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":731}},"tokens_in":530,"tokens_out":803,"duration_ms":7916,"temperature":1.0,"reasoning_tokens":731,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:46:17.308522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Insert a deliberately misdescribed unseen model into the graph—for example, a weak vision model profiled as a strong numerical reasoner—and check whether it receives the same share of queries as an accurately profiled model with identical true capabilities. If routing accuracy collapses when descriptions are corrupted, the result depends on the text prior; if it does not, the graph structure rather than the profiles is doing the work, and a second test (removing all query-query edges) would show whether the claimed generalization survives without query similarity.","supporting_citations":[],"review_version":2}