{"id":"bac4d53b-dcf7-4520-a545-099156031aaa","arxiv_id":"2411.11448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Replacing learned adaptive node embeddings with PCA-derived embeddings keeps traffic forecasting models accurate across years and cities without retraining.","lead":"This paper finds that traffic models which learn their own per-sensor embeddings lose accuracy when tested on data from a later year or a different city, and proposes replacing those learned embeddings with PCA-derived ones. The fix needs no retraining and works across several model architectures, offering a cheap way to keep traffic predictions accurate as cities change.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-year gains are confounded: PCA projection P is computed from 5% of the target-year data (Sec. IV), so the 'training-free' comparison to frozen baselines is not apples-to-apples.","rationale":"The reader's weakest_assumption focused on whether the PCA subspace is stable across years/cities; that is a valid statistical concern for cross-city transfer. But the more directly load-bearing issue is that the cross-year experiments are not a clean test of the central claim: P is estimated from 5% of the target-year data, giving the PCA-embedding method access to target distribution statistics that the frozen baselines lack. The paper even labels this a leakage-avoidance step, but using target-year observations to construct the test-time representation is a form of transduction. The method description (same P) contradicts the experimental protocol (P from second year), so the paper is internally inconsistent on a point that determines the validity of its headline results. The strong cross-year improvements in Tables IV and V are the main evidence for the claim that adaptive embeddings cause inflexibility; if they are confounded, the paper's central message is not established. The proposed concrete test, rerunning with source-year-only P, would settle whether the gains are genuine. The reader's CONDITIONAL verdict remains appropriate, but the condition should be made explicit: the authors must either rerun without target-year PCA or include baselines that also receive 5% target-year data (e.g., fine-tuning on that same slice, which Table III partially addresses). If the gains disappear under the source-only projection, the verdict should move to REJECT. This is a good-faith critique of the experimental design, not of the authors' intent; the method may still work, but the current evidence does not support the strong claim as stated.","tokens_in":21206,"tokens_out":4911,"duration_ms":49445,"concrete_test":"Recompute the PEMS03/04/07/08 cross-year results (Tables IV and V) with the PCA projection matrix P estimated exclusively from first-year training and validation data (no second-year observations), then test on the full second-year data. Compare these new numbers against both the reported PCA-emb results and the original baselines. If the PCA-emb gains shrink to near baseline levels, the headline cross-year benefit is an artifact of target-year leakage. As a secondary check, compare P estimated from a random 5% of the second year versus P from the first year only; the difference isolates the leakage contribution.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section III-C defines E_test = Z_val * P and claims 'the same PCA projection matrix, P' is used at test time, implying P is estimated from source-year data. However, Section IV explicitly states: 'To prevent information leakage, we utilized 5% of the data from the second year for PCA dimensionality reduction to generate embeddings, and then tested on the remaining 95%.' Thus, in the cross-year experiments of Tables IV and V, P is estimated from a 5% slice of the target-year distribution. The baselines (AGCRN, STID, GWNet, etc.) use their frozen trained adaptive embeddings and see no target-year data. The dramatic reported gains (e.g., STID on PEMS03: MAE 33.99→9.18) could therefore be driven by the PCA projection having access to the target year's covariance structure, not by a fundamental advantage of PCA embeddings for out-of-distribution generalization. This is transductive adaptation, not the 'training-free' zero-shot adaptation claimed in the abstract and Section III. The internal inconsistency between the method description (same P from training) and the experimental setup (P from second year) is itself a correctness risk. If the cross-year gains vanish when P is computed only from source-year data, the central claim that adaptive embeddings are the main source of inflexibility is unsupported by the paper's primary evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that trainable adaptive node embeddings are a central cause of poor cross-year and cross-city generalization in traffic forecasting models. To support this, it introduces an alternative embedding computed by PCA: E_pca = Z · P, where P is a projection matrix obtained from the input statistics, and reports that replacing learnable embeddings with PCA embeddings preserves in-distribution accuracy while yielding large cross-year gains (e.g., STID on PEMS03 2018→2019, average MAE 33.99→9.18) and usable zero-shot cross-city transfer (Table I). The paper also introduces four new cross-year PEMS benchmarks and compares PCA embeddings against zero-embedding and fine-tuning strategies.","tokens_in":21419,"tokens_out":3090,"duration_ms":36120,"significance":"If the central claims held, the paper would provide a simple, parameter-light remedy for a recognized practical problem: adaptive embeddings that cannot adapt when sensor networks or urban spatial relationships change. The cross-year benchmark itself is a useful contribution, and the idea of deriving node embeddings from input statistics rather than from learned parameters is worth further study. The paper is also careful to include zero-embedding and fine-tuning comparisons in Table III, which helps isolate the role of the embedding. However, the main experimental evidence is currently confounded by the use of target-year data in fitting the PCA projection, and several reporting choices (single-seed results, per-dataset model selection, unexplained MAPE values) prevent the claims from being accepted as stated. The potential significance is real, but the evidence as presented is not yet load-bearing.","major_comments":[{"comment":"The cross-year experiments in Tables IV and V are not a fair test of the claim that PCA embeddings enable training-free adaptation. Section IV states: \"To prevent information leakage, we utilized 5% of the data from the second year for PCA dimensionality reduction to generate embeddings, and then tested on the remaining 95%.\" That means the projection matrix P is fit on a slice of the target-year distribution, whereas the frozen baselines see no target-year data. Eq. (2) and the surrounding text say that \"the same PCA projection matrix, P\" from training is used at test time, which is inconsistent with the experimental setup. The reported gains (e.g., STID on PEMS03: MAE 33.99→9.18) could therefore be driven by transductive access to the target-year covariance structure rather than by a general advantage of PCA embeddings. The authors must rerun Tables IV and V with P computed only from source-year training data (and possibly source-year validation data), and state explicitly whether any target-year data are used in fitting P. If the gains disappear, the central claim about adaptive embeddings being the main source of inflexibility is not supported by the paper's primary evidence.","section":"Section IV, Section III-C, Eqs. (1)-(2)"},{"comment":"Table III selects a different model for each dataset (AGCRN for PEMS03, GWNet for PEMS04, STID for PEMS07, STAEformer for PEMS08) without stating the selection protocol. If the PCA-embedding comparison is meant to show a general effect, the paper should report results for all evaluated models on all four datasets, or at least specify an a priori rule for choosing the displayed model. As it stands, the reader cannot tell whether the reported PCA-vs-baseline and PCA-vs-fine-tune comparisons are representative or cherry-picked. This matters because the fine-tuning comparison in Table III is the cleanest controlled evidence in the paper, and it should not be limited to favorable examples.","section":"Table III"},{"comment":"The MAPE values in Tables IV and V are not credible as reported. For example, PEMS03 2018→2019 baseline MAPEs range from 753% to 1329% while MAEs are in the 20-50 range (e.g., AGCRN average MAE 28.04 and MAPE 820.77%). Such values are impossible under the standard MAPE definition unless the data contain many near-zero ground-truth values and the error is not masked consistently. The paper says zero values are excluded \"as they represent noisy data,\" but the exact masking rule, the proportion of masked samples, and the MAPE formula are not given. The \"IMPROVE\" rows (e.g., 90.5% MAPE improvement on PEMS03) are largely driven by these inflated baseline MAPEs. The authors should report MAPE on a common, clearly defined subset of valid samples and also report error distributions or additional scale-free metrics that are robust to near-zero values.","section":"Tables IV and V, MAPE definition"},{"comment":"No results are averaged over multiple random seeds, and no standard deviations or confidence intervals are reported. The in-distribution comparisons in Figure 5 and the cross-year comparisons in Tables IV-V show differences that are often small relative to the typical run-to-run variance of these models (e.g., many MAE differences below 1.0 in Figure 5). For the central quantitative claims, the paper needs at least three seeds per configuration and a statement of how the reported numbers are selected. This is especially important because Section IV says the authors \"performed multiple rounds of parameter tuning to achieve optimal results,\" which raises the risk of selecting favorable runs.","section":"All experimental tables"}],"minor_comments":[{"comment":"The claim that the method is \"training-free\" is overstated: PCA itself is a fitted projection, and in the current experiments it is fit on 5% of the target year. The abstract and Section I should distinguish between source-only PCA (genuinely zero-shot) and target-adapted PCA (transductive).","section":"Abstract and Section IV"},{"comment":"The notation is inconsistent: Eq. (2) defines E_test using Z_val, but the text says P is the projection from training data. If the intended protocol is to reuse P from training, the test-time equation should show this, and the validation subset should not be introduced as the source of P for the cross-year experiments.","section":"Section III-C, Eq. (2)"},{"comment":"The introduction contains duplicated paragraphs: the paragraph beginning \"In this paper, we investigate the core limitations\" appears twice with only minor rewording. Please remove the redundancy.","section":"Section I and Section IV"},{"comment":"The paper states that the code is \"released in code,\" but no URL is given. Since the new cross-year benchmarks are a central contribution, the datasets and code should be made available with a stable link, and a data card should describe how the PEMS sensor sets were matched across years.","section":"Section IV, dataset description"},{"comment":"For the cross-city zero-shot experiments, please clarify whether the PCA projection matrix is computed from the source city only or from a subset of the target city. The text says \"projection matrices W derived from PEMS03 and PEMS07\" are applied to PEMS04/PEMS08, but the LargeST description is less explicit; a direct statement for each block of Table I would remove ambiguity.","section":"Table I"},{"comment":"The optimal number of principal components is selected on the test set (PEMS03) by grid search. Please state how the number of components and the variance threshold θ are chosen in all other experiments, and whether the selection is made on validation data only, to avoid test-set tuning.","section":"Section IV-F, Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the benchmark is potentially useful, but the main quantitative support for the cross-year claim is compromised by the 5% target-year PCA fitting in Section IV. This is not merely a presentation issue; it changes what the experiments demonstrate. The paper also needs a release of the new benchmark data and code before it can be properly evaluated by the community. If the authors can show that source-only PCA retains most of the reported cross-year gains, the paper would be a solid candidate; as it stands, the evidence is insufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper identifies a real failure mode—learned adaptive node embeddings go stale when the urban environment changes—but its headline cross-year evidence is confounded. In Section IV they fit the PCA on 5% of the target-year data before testing on the remaining 95%. That is transductive adaptation, not the training-free fix promised in the abstract and Section III. The baselines get no target-year information. So the dramatic gains in Tables IV and V (e.g., STID on PEMS03: MAE 33.99 to 9.18) can't be attributed to PCA embeddings being intrinsically more flexible; they may just reflect having seen the target year's covariance.\n\nWhat's genuinely new: the cross-year benchmarks themselves, and the zero-shot city-transfer experiments in Table I. Those are zero-shot in the honest sense: the PCA projection is computed from the source city and applied to the target. The diagnostic that LSTM stays flat while adaptive-embedding models degrade is also worth attention. And the finding that PCA embeddings don't hurt in-distribution performance (Figure 5) across many architectures is useful.\n\nSoft spots beyond the confound:\n\n- No code or data link despite the abstract claiming release. The new cross-year datasets are not available, and there are no error bars or seeds anywhere. You can't check any of the headline numbers.\n- Tables IV and V contain MAPE values above 700% (e.g., 1157.81%). That's implausible as stated and looks like a denominator/normalization artifact. It needs explanation or correction.\n- Table III cherry-picks one model per dataset, which is fine for illustration but not evidence.\n- The paper's framing of the same PCA projection matrix P in Section III-C directly contradicts the Section IV procedure. That's a flag for the authors to fix, not necessarily a sign of bad faith, but it is load-bearing.\n\nOverall: the central diagnostic might survive once the cross-year confound is removed. I'd want to see a source-year-only PCA ablation, released code, and a cleanup of the MAPE values before trusting the big claims.\n\nThis is worth a serious referee. Send it to review, but the reviewers should demand the source-only ablation and the code. I wouldn't cite it yet.","headline":"Cross-year setup is confounded because PCA is fit on target-year data, but the zero-shot city-transfer experiments and the diagnosis of stale adaptive embeddings are real and deserve review.","tokens_in":21987,"tokens_out":2741,"would_cite":false,"duration_ms":29604,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trainable node embeddings are the main source of inflexibility in traffic forecasting; replacing them with PCA-derived embeddings restores cross-year accuracy and enables zero-shot city transfer.","keywords":["traffic forecasting","adaptive embedding","PCA embedding","spatiotemporal graph neural networks","distribution shift","zero-shot transfer","cross-city generalization"],"falsifier":"Measure the fraction of total variance in the target-year or target-city traffic data that is preserved when projected onto the source PCA subspace. If that fraction drops sharply (for instance, below the level at which the top components explain the training data), then the cross-year and zero-shot gains should collapse, and the model should fall back to the degraded performance seen with trained embeddings. A concrete experiment: train on PEMS03, project PEMS04 or PEMS08 data onto the PEMS03 subspace, and correlate the explained-variance ratio with the observed MAE across different target years and cities.","tokens_in":20961,"feed_emoji":"🚗","tokens_out":18673,"duration_ms":163685,"temperature":0.7,"pith_summary":"Current spatiotemporal traffic forecasting models—both graph neural networks and Transformers—degrade sharply when tested a year after training or on a different city. This paper argues that the degradation is caused by the trainable adaptive embedding layer that encodes node identities, not by the temporal architecture: an LSTM with no such embeddings stays stable across years. The proposed remedy replaces trainable embeddings with PCA embeddings computed from the data via the projection matrix $P$, which is learned once and reused at test time to map new inputs into the same low-dimensional space. With this substitution, cross-year error drops substantially (e.g., STID on PEMS03 average MAE from 33.99 to 9.18) and zero-shot cross-city prediction becomes usable without retraining. If correct, this identifies a single swap-able component as the bottleneck and offers a training-free path to adapting traffic models to evolving urban environments.","feed_headline":"Swap trainable node embeddings for PCA to fix traffic forecasts","feed_subtitle":"Train on one year, predict the next: PCA embeddings cut error by 73 percent and enable zero-shot city transfer.","key_machinery":"The PCA embedding is the central object: for each day $d$, the traffic matrix $Z_d \\in \\mathbb{R}^{N \\times T}$ (nodes by daily time slots) is projected onto the principal subspace learned from training data, $E_d = Z_d P$, and the final node embedding is the average over training days, $E_{\\mathrm{pca}} = \\frac{1}{D}\\sum_d Z_d P$. At test time the same projection matrix $P$ maps new data into the same space, allowing the model to build a fresh graph and process node sets of different sizes without retraining. This machinery directly counters the three diagnosed limitations of adaptive embeddings: it gives inductive capacity because the embedding is recomputed from data, balanced distinguishability because PCA truncation retains only major variance directions, and transferability because the same basis serves different cities. The choice of the number of principal components acts as a knob controlling spatial distinctiveness, with experiments showing optimal values around 4–8 components for STID and STAEformer on PEMS03.","core_discovery":"Adaptive embeddings—randomly initialized learnable node representations that are updated during training to distinguish locations with similar historical patterns—are the principal cause of inflexibility in spatiotemporal forecasting under spatial shift. When the urban environment shifts (a later year or a different city), these fixed embeddings carry outdated spatial biases, and models that rely on them collapse, whereas an LSTM without such embeddings maintains consistent error. Replacing adaptive embeddings with PCA embeddings, defined as $E_{\\mathrm{pca}} = ZP$ where $Z$ is the day-sliced traffic feature matrix and $P$ is the PCA projection learned from the training data (including a 5% validation subset), largely fixes the problem: in-distribution accuracy is preserved or improved, cross-year average MAE drops sharply on four PEMS benchmarks, and zero-shot transfer between cities becomes feasible across several architectures. This is interpreted as evidence that the principal subspace of traffic patterns is sufficiently stable across years and cities to serve as a universal coordinate system for node identity.","pith_inferences":["A testable extension: if the source PCA subspace is a stable 'traffic manifold,' then the same projection matrix $P$ could be applied across many cities and years, opening the door to a shared embedding space for large-scale traffic foundation models—something the paper hints at but does not fully develop.","The paper's diagnosis suggests that fine-tuning adaptive embeddings is a stopgap: fine-tuning matches PCA on the tested benchmarks, but because it re-fits embeddings to the target distribution, it should be more sample-hungry than the training-free PCA projection, especially when the target has few labeled samples—an experiment the paper does not run.","A boundary condition for the method: the stability of the principal subspace across domains is assumed rather than proven; one could stress-test it with a city that has a fundamentally new transport mode or a radically different sensor layout, where the source subspace might fail to span the target variance.","The finding may generalize beyond traffic to other spatiotemporal prediction tasks (energy demand, crowd flow, disease spread) wherever node identities shift over time, but the subspace-stability assumption would need to be verified for each domain."],"forward_implications":["If the claim holds, traffic forecasting models can be trained once and then applied to a later year's data with substantially lower error, eliminating the need for yearly retraining.","Zero-shot cross-city deployment becomes practical: a model trained on one city's sensors can forecast traffic in another city without fine-tuning, as demonstrated on PEMS and LargeST pairs.","The adaptive embedding layer, rather than the graph or temporal architecture, is the common bottleneck; therefore, swapping in PCA embeddings should improve any ST-GNN or Transformer that uses trainable node embeddings.","PCA embeddings make the graph structure flexible between training and testing, so sensor additions, removals, or failures no longer require model modification—only a recomputation of the embedding from the new feature matrix.","In-distribution accuracy is not sacrificed for generalization; in several cases the frozen PCA embedding outperforms the trained adaptive embedding because it curbs overfitting and over-distinguishability."],"supporting_citations":[{"why":"Spatial-Temporal Identity (STID) supplies the adaptive-embedding baseline whose cross-year error collapses (MAE 33.99 to 9.18 with PCA on PEMS03), providing the paper's headline evidence.","marker":"[11]"},{"why":"Graph WaveNet introduces the learnable-embedding adaptive adjacency that the paper diagnoses as inflexible, and GWNet is one of the models showing large cross-year gains with PCA.","marker":"[12]"},{"why":"Adaptive Graph Convolutional Recurrent Network (AGCRN) is a core adaptive-embedding model; its performance degradation over time motivates the paper's diagnosis.","marker":"[13]"},{"why":"Spatio-temporal Adaptive Embedding makes a vanilla Transformer SOTA for traffic; its results show the adaptive-embedding bottleneck and the PCA fix extend to Transformer architectures.","marker":"[16]"},{"why":"This benchmarking study supplies the concepts of spatial indistinguishability and excessive distinguishability that the paper uses to explain why PCA embeddings help.","marker":"[17]"},{"why":"The PeMS system is the data source for the four cross-year benchmarks the paper constructs to measure spatial-shift degradation.","marker":"[21]"},{"why":"The standard PEMS benchmark splits define the data alignment that the new cross-year test sets are built from.","marker":"[23]"},{"why":"Principal component analysis is the statistical method underlying the proposed PCA embedding, providing the projection matrix $P$ used at train and test time.","marker":"[27]"},{"why":"LargeST supplies the large-scale cross-city datasets (San Diego, Bay Area, Los Angeles) used to demonstrate zero-shot transfer in a big-data regime.","marker":"[6]"}],"fun_headline_variants":["PCA embeddings beat adaptive embeddings for traffic forecasting","Adaptive embeddings break traffic models; PCA embeddings fix them","Zero-shot city transfer with PCA embeddings for traffic","PCA embeddings enable robust traffic forecasting across years and cities","Adaptive embeddings cause traffic forecast decay; PCA fixes it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire cross-year and zero-shot benefit rests on the assumption that the PCA projection matrix learned from the training year remains a valid coordinate system for later-year or other-city traffic data; if the target data has variance in directions outside the source subspace, the projection discards exactly the information that distinguishes the new sensors.","fun_headline_variants_meta":{"raw":{"variants":["PCA embeddings beat adaptive embeddings for traffic forecasting","Adaptive embeddings break traffic models; PCA embeddings fix them","Zero-shot city transfer with PCA embeddings for traffic","PCA embeddings enable robust traffic forecasting across years and cities","Adaptive embeddings cause traffic forecast decay; PCA fixes it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000354,"raw_usage":{"total_tokens":1933,"prompt_tokens":961,"completion_tokens":972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":897}},"tokens_in":577,"tokens_out":972,"duration_ms":7497,"temperature":1.0,"reasoning_tokens":897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:31:04.075972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the fraction of total variance in the target-year or target-city traffic data that is preserved when projected onto the source PCA subspace. If that fraction drops sharply (for instance, below the level at which the top components explain the training data), then the cross-year and zero-shot gains should collapse, and the model should fall back to the degraded performance seen with trained embeddings. A concrete experiment: train on PEMS03, project PEMS04 or PEMS08 data onto the PEMS03 subspace, and correlate the explained-variance ratio with the observed MAE across different target years and cities.","supporting_citations":[{"cited_title":"Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting,","cited_arxiv_id":null,"evidence_quote":"Spatio-temporal Adaptive Embedding makes a vanilla Transformer SOTA for traffic; its results show the adaptive-embedding bottleneck and the PCA fix extend to Transformer architectures."},{"cited_title":"Freeway performance measurement system: mining loop detector data,","cited_arxiv_id":null,"evidence_quote":"The PeMS system is the data source for the four cross-year benchmarks the paper constructs to measure spatial-shift degradation."},{"cited_title":"Attention based spatial- temporal graph convolutional networks for traffic flow forecasting,","cited_arxiv_id":null,"evidence_quote":"The standard PEMS benchmark splits define the data alignment that the new cross-year test sets are built from."}],"review_version":1}