{"id":"aa0acee6-7624-48ce-9b46-6342de47b345","arxiv_id":"2412.12201","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Using a frozen LLM as a test-time selector among graph and hypergraph predictions, plus a ranking loss, improves traffic flow forecasts on PEMS03/04/08.","lead":"This paper proposes LEAF, a traffic flow forecasting method that uses a large language model to choose the best prediction from two neural network branches at test time. A smart generalist might care because the method targets a common real-world problem: models that degrade when traffic conditions change after deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control for the selector: the reported gains could come from the LLM's choice itself, not from the predictor or its ranking-loss adaptation.","rationale":"The reader identified the same load-bearing weakness: because the final output is the LLM's pick, the gain could come from the selector alone, and the paper does not test random or heuristic selection as a control. This is indeed the most important threat to the central claim, because the novelty of LEAF is precisely the use of an LLM as a test-time selector. If a trivial selector matches the reported MAE, the method's contribution reduces to the choice-set construction and the ranking loss, which are not new. The missing control is concrete and testable, and it is not answered by the existing ablations: E1-E4 show that adding the selector helps, but they do not distinguish the LLM's reasoning from the mechanical effect of expanding the choice set and then picking one option. E5 and E6 isolate the transformations and the ranking loss, but again only within the LLM-based pipeline. The paper does release code and reports consistent improvements across three datasets, which is genuine supporting evidence, but the absence of a selector baseline is a gap that directly bears on the stated contribution. I therefore do not change the verdict: the paper remains a conditional accept, with the condition being that the LLM selector is actually better than a random or heuristic selector. No ad hominem is intended; the critique is about experimental control, not author conduct.","tokens_in":17343,"tokens_out":3511,"duration_ms":34575,"concrete_test":"On PEMS08, rerun the full LEAF pipeline with the LLM selector replaced by two controls while keeping the choice set, ranking loss, and K=2 identical: (a) uniform random choice from C_i for each vertex, and (b) the fixed heuristic 'choose the hypergraph branch with downward trend'. Repeat with at least three seeds to obtain means and standard deviations. If either control reaches MAE close to LEAF's 24.68 (e.g., within 0.5), the claimed advantage of LLM discriminative selection is not established. Separately, report the MAE of the predictor's own final output (after the ranking-loss update, before LLM selection) to verify that the predictor itself has improved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that LEAF's frozen-LLM selector improves traffic forecasting, with the final output being the LLM's selected candidate from the choice set in Eq. 6. The predictor branches are updated via the ranking loss in Eq. 7-8 using the LLM's picks as positives, but the reported metric is computed on the LLM's selection, not on the predictor's own output. The ablations (E3-E6) show that adding the selector improves over each branch alone, but they never replace the LLM with a random or simple heuristic selector. Without such a control, the observed improvement can be fully explained by the selector picking a transformed version of an existing prediction, rather than by any discriminative reasoning of the LLM or by the pseudo-label supervision improving the predictor. In particular, the prompt in Figure 8 includes strong temporal hints about rush hours, and the visualization in Figure 6 shows the LLM selecting the lowest downward-trend option at the end of rush hour; a fixed heuristic such as 'always choose the hypergraph branch with downward trend' might perform nearly as well. The paper's load-bearing assumption is therefore that LLM selection is better than cheap selection rules, and this assumption is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LEAF is a test-time traffic forecasting framework that combines a dual-branch predictor (a graph branch and a hypergraph branch) with a frozen LLM-based selector. The predictor is pretrained on a small training set; during inference, each branch produces per-vertex forecasts, and five fixed transformations (smoothing, upward/downward trend, overestimate/underestimate) expand these into a choice set. An LLM prompt containing historical data, spatio-temporal information, and the candidate sequences is used to select the most likely candidate per vertex, and the selected candidates are then treated as positives in a ranking loss that updates the predictor. This prediction-selection loop is repeated K times, and the final forecast is the LLM's selected candidates. Experiments on PEMS03, PEMS04, and PEMS08 report consistent improvements over eight baselines, with ablations on PEMS08 showing contributions from both branches, the transformations, and the ranking loss.","tokens_in":17621,"tokens_out":7062,"duration_ms":60264,"significance":"If the reported improvements are attributable to the proposed mechanism, the paper would provide a practical way to exploit the discriminative ability of frozen LLMs for traffic forecasting while keeping the LLM out of the generative loop, and it ships code and uses standard datasets and metrics. The idea of using LLM selections as ranking-loss supervision for a dual-branch predictor is interesting and goes beyond direct generative LLM forecasting. However, I agree with the stress-test concern that this attribution is currently unsupported: the final output is the LLM's pick, not the predictor's own output, and no cheap-selector control is run. The paper's significance would be substantially strengthened by such controls, by a clear validation protocol for M and K, and by an actual distribution-shift evaluation, which is the stated motivation of the work.","major_comments":[{"comment":"The final forecast is the LLM-selected candidate (Algorithm 1, lines 6-9), not the predictor's own output, and the ranking loss in Eqs. 7-8 is supervised by the same selection. The ablations E1-E6 show that adding the selector to a branch improves over that branch alone, but no control replaces the LLM with a random or simple heuristic selector (e.g., always choose the downward-trend hypergraph candidate when the forecast period is the end of rush hour, or choose the candidate whose trend best matches the historical slope). Without these controls, the reported gains could come entirely from the choice set and the prompt's domain hints rather than from the LLM's discriminative reasoning or from the predictor adaptation. Please add a random-selector baseline, two or three heuristic selectors, an oracle upper-bound selector, and an ablation where the final output is the predictor's own output after ranking-loss training.","section":"§3.3 and Algorithm 1; Table 2"},{"comment":"The hyperparameters M and K in Algorithm 1 are selected on PEMS08 (Figure 4), and Table 1 then reports PEMS08 as a test result without stating a validation split for that selection; the Table 2 ablations are also on PEMS08, so the headline PEMS08 improvement is at risk of selection bias. Furthermore, the paper's motivation is adaptation to test-time environmental changes (Abstract and §1), but the experiments only use a 10% training subset and non-overlapping test slices, which is not a controlled distribution-shift evaluation. Please report hyperparameter choices made on a validation split (or tuned on one dataset and transferred), provide error bars over multiple runs, and add at least one experiment with a genuine train/test distribution shift, such as different time periods or different weather conditions.","section":"§4.1, §4.4, Table 1"},{"comment":"The prompt in Figure 8 explicitly instructs the LLM that rush-hour phase is the most important temporal signal and explains how to infer the beginning or end of rush hours from historical changes. The example in Figure 6 shows the LLM selecting a downward-trend candidate because 'the rush hour is likely ending.' This injected domain knowledge is a legitimate part of the method, but it means a simple rule-based selector could reproduce much of its behavior. Please test the selector with prompts that omit these hints, or compare against a heuristic that encodes the same rush-hour rule, to support the claim that the LLM's internal knowledge and reasoning drive the improvements.","section":"§4.2, Figure 8, Figure 6"}],"minor_comments":[{"comment":"The table header uses 'STSGNN' while §4.2 and the reference list use 'STSGCN'; please make the names consistent.","section":"Table 1"},{"comment":"The expression for the normalized adjacency matrix, dAST = D^{-1/2}AST D^{-1/2}, is notationally unclear because the tilde or self-loop term is missing; please define the normalized matrix explicitly.","section":"Eq. (3)"},{"comment":"Both panels lack axis labels; please add the metric names and the hyperparameter values on the x-axes.","section":"Figure 4"},{"comment":"The sentence 'We choose a subset of non-overlapping slices in the test set' should specify the subset size, the slice length, and the random seed for reproducibility.","section":"§4.1"},{"comment":"The prompt contains the instruction 'Note that smoothing does not reduce MAE error'; if this is a deliberate design hint, it should be explained in the main text, otherwise remove it.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the empirical trend is consistent across datasets, but the missing selector control is load-bearing for the paper's central claim, and the validation protocol for M and K is unclear. These issues are addressable with additional experiments and a clearer evaluation setup, so I recommend major revision rather than rejection. I have no concerns about the citation pattern or the journal scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look: LEAF is one of the few traffic-forecasting papers that uses a frozen LLM as a discriminative selector rather than a generative forecaster, and the mechanism is clearly explained. The dual-branch predictor is standard graph/hypergraph material, but the test-time prediction-selection loop with a ranking loss is new relative to the cited literature. Code is released, and the headline numbers beat eight baselines consistently on three PEMS datasets. The ablations are also more informative than what this area usually ships: removing either branch hurts, removing transformations hurts, and removing the ranking loss hurts.\n\nThe main evaluative weakness is exactly what the stress-test note flags: there is no control for the selector. The final output is the LLM's pick, not the predictor's own output. Any selection rule—random, nearest-to-history, or \"always pick the hypergraph branch with downward trend at the end of rush hour\"—could produce gains over the branch outputs alone even if the LLM contributes nothing. The ablations show the selector helps, but they never replace the LLM with a cheap heuristic. Given that the prompt leaks strong temporal cues (rush-hour descriptions) and the visualization shows the LLM choosing the lowest downward-trend option, a heuristic baseline is not a nitpick; it is necessary to support the paper's claim that LLM discriminative ability is what works.\n\nOther issues are real but less severe. There are no error bars, so the 0.5–1.5 MAE gains could be within noise. Hyperparameters M and K are tuned on PEMS08 without a clear validation split. And despite the abstract's test-time environmental-change framing, no actual distribution-shift experiment is run; training on 10% of the data is a low-data regime, not a shift. I agree with the reader that this is not equation-level circularity—the final numbers are measured against external benchmarks—and the authors are honest about scope limitations.\n\nI would send this to peer review with a request for a non-LLM selector control, error bars, and a real shift experiment. The idea deserves referee time; the current evidence does not yet support the strong LLM-specific claim.","headline":"A genuinely new LLM-as-selector mechanism with reproducible code, but the missing non-LLM selector control leaves the paper's central claim unproven.","tokens_in":18102,"tokens_out":2429,"would_cite":true,"duration_ms":23149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen large language model can improve traffic forecasting by choosing among candidate forecasts rather than generating them.","keywords":["traffic flow forecasting","large language models","test-time adaptation","graph neural networks","hypergraph neural networks","ranking loss","distribution shift","spatio-temporal forecasting"],"falsifier":"Replace the LLM selector on PEMS08 with a random pick from the same choice set, or with a fixed heuristic such as always choosing the downward-trend option during evening rush hours; if the resulting MAE matches or closely approaches LEAF's 24.68, the reported gains come from the choice set and ranking loop rather than from the LLM's reasoning.","tokens_in":17154,"feed_emoji":"🚦","tokens_out":3411,"duration_ms":29767,"temperature":0.7,"pith_summary":"The paper proposes LEAF, a traffic flow forecasting framework that uses a frozen large language model as a test-time selector rather than as a direct forecaster. Two learned branches, one graph-based and one hypergraph-based, each produce forecasts; the LLM reads a text prompt with historical data and candidate predictions, then picks the most plausible option for each sensor. The chosen options are fed back through a ranking loss to refine the two branches. On three public traffic datasets with only 10% of data used for training, LEAF reports consistent improvements over eight baselines, including lower MAE on every dataset. The central claim is that the discriminative ability of an LLM, not its generative ability, is what helps forecasting under distribution shift.","feed_headline":"Frozen LLM picks winners for traffic forecasts","feed_subtitle":"A selector that chooses among graph and hypergraph forecasts beats eight baselines on three public datasets.","key_machinery":"The load-bearing mechanism is the prediction-selection loop built from a dual-branch predictor and an LLM-based selector. The graph branch uses graph convolutions over a spatio-temporal graph to model pair-wise relations; the hypergraph branch uses a low-rank learnable incidence matrix to model non-pair-wise relations. At test time, each branch emits a forecast, a small set of transformations (smoothing, upward/downward trend, over/under-estimation) expands these into a choice set per sensor, and a frozen LLM selects one option from a structured prompt. The selected option becomes a pseudo-label for a hinge-style ranking loss that pushes the predictor's output closer to the chosen candidate than to any rejected candidate; this loss is applied for a few iterations before the next prediction-selection round, with K=2 rounds in the reported experiments.","core_discovery":"LEAF claims that a frozen LLM can serve as a reliable test-time selector for traffic flow forecasting, and that this selector-plus-ranking-loss loop improves prediction accuracy beyond either branch alone or any of eight baselines. For example, on PEMS08 the method achieves MAE 24.68 versus 26.42 for the best baseline, and the ablation shows that removing either branch, removing the transformations that expand the choice set, or removing the ranking loss all degrade performance. The paper interprets this as evidence that the LLM uses its internal knowledge of traffic patterns and rush-hour dynamics to pick the most likely candidate, and that supervising the predictor with these picks through a ranking loss yields further gains.","pith_inferences":["A testable extension the paper does not run is to replace the LLM selector with a random pick or a simple heuristic rule (for example, always choose the downward-trend option during the evening rush) over the same choice set; matching LEAF's MAE would indicate the gains come from the choice set and ranking loop rather than from LLM reasoning.","Because the LLM is frozen and the prompt is purely textual, the same selector design should transfer to other spatio-temporal forecasting tasks with contextual text, such as energy load or crowd flow prediction, as long as a prompt can describe the location and time context.","The iterative loop doubles LLM inference cost when K=2, so deployment would likely benefit from prompt caching or batched selection across sensors; the paper does not report total LLM inference cost.","The ranking loss assumes the LLM's pick is at least better than the closest rejected candidate; if the LLM is wrong in a systematic way, the loop could reinforce that bias, which is why a heuristic baseline comparison matters."],"forward_implications":["LLMs do not need to generate numeric forecasts to help traffic prediction; selecting among candidate forecasts is a lower-risk use of their knowledge.","Combining pair-wise graph relations and non-pair-wise hypergraph relations in one predictor is beneficial, since ablations removing either branch degrade performance.","Test-time selection with ranking-loss supervision reduces long-horizon forecasting errors more than short-horizon errors, as shown by per-timestep MAE curves.","The framework works with small training sets (10% of data), suggesting it is useful in settings where labeled traffic data are scarce and distribution shift is expected."],"supporting_citations":[{"why":"Supplies the graph convolution operation used in the graph branch.","marker":"(Kipf and Welling, 2016)"},{"why":"Defines the problem setup and the STSGCN baseline that LEAF compares against.","marker":"(Song et al., 2020)"},{"why":"Provides the dynamic hypergraph structure learning method that the hypergraph branch builds on.","marker":"(Zhao et al., 2023)"},{"why":"The LLM-MPE baseline, representing the generative-LLM approach that LEAF argues is weaker than selection.","marker":"(Liang et al., 2024)"},{"why":"Foundations for the ranking-style loss used to supervise the predictor with the selector's choices.","marker":"(Weinberger and Saul, 2009; Sohn, 2016)"},{"why":"The DCRNN baseline and the evaluation protocol with 12-step history and 12-step prediction.","marker":"(Li et al., 2018)"},{"why":"The ASTGCN baseline, a spatial-temporal graph attention model LEAF outperforms.","marker":"(Guo et al., 2019)"},{"why":"The COOL baseline, a recent spatio-temporal GNN that represents the current graph-based approach.","marker":"(Ju et al., 2024)"},{"why":"The STAEformer baseline, a transformer-based traffic predictor LEAF is compared against.","marker":"(Liu et al., 2023)"}],"fun_headline_variants":["LLM picks best traffic forecast from two branches","Frozen LLM as referee for traffic flow predictions","Traffic forecasting with an LLM judge","LLM selects between graph and hypergraph forecasts","Ranking loss boosts LLM-selected traffic predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the frozen LLM's chosen candidate is a trustworthy pseudo-label for updating the predictor through the ranking loss, because the final output is the LLM's pick rather than the predictor's own forecast.","fun_headline_variants_meta":{"raw":{"variants":["LLM picks best traffic forecast from two branches","Frozen LLM as referee for traffic flow predictions","Traffic forecasting with an LLM judge","LLM selects between graph and hypergraph forecasts","Ranking loss boosts LLM-selected traffic predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":1111,"prompt_tokens":846,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":193}},"tokens_in":462,"tokens_out":265,"duration_ms":2652,"temperature":1.0,"reasoning_tokens":193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:21:59.869459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the LLM selector on PEMS08 with a random pick from the same choice set, or with a fixed heuristic such as always choosing the downward-trend option during evening rush hours; if the resulting MAE matches or closely approaches LEAF's 24.68, the reported gains come from the choice set and ranking loop rather than from the LLM's reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the problem setup and the STSGCN baseline that LEAF compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundations for the ranking-style loss used to supervise the predictor with the selector's choices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ASTGCN baseline, a spatial-temporal graph attention model LEAF outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The STAEformer baseline, a transformer-based traffic predictor LEAF is compared against."}],"review_version":1}