{"id":"5e7f7087-13cc-4d71-890c-e8e7383b82f9","arxiv_id":"2505.22686","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"KAN achieves R2 up to 0.9998 for daily temperature in Abidjan and Kigali, but missing split details and baselines make the result unverifiable and likely inflated.","lead":"This paper benchmarks neural network weather forecasters for two African cities and reports that Kolmogorov-Arnold Networks predict temperature almost perfectly. The extreme accuracy is suspicious and likely stems from a data-splitting flaw, so the headline result should be treated with caution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KAN temperature MSE and R2 are internally inconsistent: every other model in Table 2 implies test-set variance ~44 C^2, while the KAN pair implies ~1 C^2.","rationale":"The reader's verdict of REJECT is well-founded, but the identified weakest assumption (non-chronological split) is not the most load-bearing issue. A stronger, directly checkable problem is that the headline KAN temperature metrics are arithmetically incompatible with the test set variance inferred from every other model in the same table. This internal inconsistency does not depend on any assumption about how the data were split: reproducing the reported MSE and R2 on the same test set is mathematically impossible given Var~44 C^2. The paper's public code makes the proposed test straightforward. If the inconsistency is confirmed, the central claim cannot be reproduced as stated; if some explanation emerges (e.g., a test variance near 1), the authors must justify it and correct the units. Either way, the paper needs revision before the KAN temperature claim can be accepted.","tokens_in":12809,"tokens_out":9570,"duration_ms":91691,"concrete_test":"Recompute Var = MSE/(1-R2) for every row of Table 2. For Abidjan temperature, the RNN rows cluster at 43.9-44.0 C^2, while the KAN row gives 1.0 C^2; for Kigali, RNNs give ~48.3 and KAN gives 1.5. Since the code is public, rerun the KAN temperature experiment using the exact 72/8/20 split reported and compute the test-set variance plus MSE and R2 on the raw (inverse-transformed) predictions. If the test variance is ~44 C^2, the reported KAN MSE of 0.0014 and R2 of 0.9986 cannot both be correct; one of them is in error. Also verify that the metric function applies the same target scaling and inverse transform for KAN as for the RNN baselines.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline temperature result is arithmetically self-inconsistent. In Table 2, Abidjan KAN reports MSE=0.0014 C^2 and R2=0.9986. Under the standard definition R2=1-MSE/Var, this implies a test-set variance of MSE/(1-R2)=1.0 C^2. Yet every other model in the same table implies Var~44 C^2 (e.g., LSTM: 7.2616/(1-0.8349)=43.97; BiGRU: 6.222/(1-0.8585)=43.98). The same pattern appears for Kigali: KAN implies Var~1.5 C^2, while all RNNs imply ~48.3 C^2. A daily temperature series with variance ~1 C^2 is physically implausible for either city and contradicts the test set used for the other models. Even if R2 were interpreted as squared Pearson correlation, the inequality MSE/Var >= 1-R2 is violated: 0.0014/44=3.2e-5 < 0.0014. Thus no predictor can realize the reported pair on that test set. This is not a split-leakage question; it is an internal numerical inconsistency in the central claim, likely from mixing scaled and original units or miscomputing R2.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks LSTM, GRU, BiLSTM, BiGRU, an ensemble, KAN, TKAN, and two TKAN variants with GeLU and MiSH activations for one-day-ahead forecasting of temperature, precipitation, and surface pressure at two tropical African stations (Abidjan and Kigali), using 14-day windows of ten meteorological variables from NASA POWER data (2010-2024). The central claimed result is that a default KAN achieves near-perfect temperature forecasts (R² = 0.9986 in Abidjan and 0.9998 in Kigali, MSE below 0.0014 °C²), three orders of magnitude better than all recurrent baselines, while TKAN variants improve precipitation forecasts in low-rainfall regimes and classical RNNs remain competitive for pressure. The paper also introduces and tests two TKAN activation variants.","tokens_in":13151,"tokens_out":4463,"duration_ms":51494,"significance":"If the results were reliable, the paper would provide a useful early benchmark of KAN-family models for localized weather prediction in under-studied tropical African cities, with a concrete claim about data efficiency and a falsifiable performance comparison. The authors provide a public code link, which is a strength for reproducibility, and they compare a broad set of architectures on three meteorological variables. However, the headline temperature result is internally inconsistent with the other metrics reported in the same table, and the evaluation protocol is not demonstrably out-of-sample; both issues directly affect the central claim. As reported, the numbers cannot be accepted as evidence for KAN superiority in temperature forecasting.","major_comments":[{"comment":"The temperature metrics for Abidjan are internally inconsistent under the standard definition R² = 1 - MSE/Var(test). The KAN row (MSE = 0.0014, R² = 0.9986) implies a test-set variance of 1.0 °C², whereas the LSTM row (MSE = 7.2616, R² = 0.8349) implies a test-set variance of 43.97 °C² on the same test set. The same discrepancy appears for Kigali, and the TKAN rows also imply a variance near 1 °C². Since a single test split cannot have two different variances, the reported MSE/R² pairs for KAN and TKAN on the one hand and the recurrent models on the other cannot all be computed on the same target scale. This suggests a mixing of scaled and original units or an incorrect R² computation, and it invalidates the abstract's central claim that KAN achieves R² ≈ 0.999 with MSE < 0.0014 °C².","section":"Section 4, Table 2"},{"comment":"The data-split description states only that the dataset was split into 72% training, 8% validation, and 20% testing; it does not state that the split is chronological. Daily meteorological series are strongly autocorrelated, so a random split places near-identical consecutive days in both training and test sets, making the reported near-perfect temperature R² values an artifact of persistence rather than evidence of genuine forecast skill. The authors must specify whether the split respects time order and, if not, rerun all experiments with a strictly chronological split (e.g., training on earlier years and testing on later years) before any comparison between models is meaningful.","section":"Section 3.1.2"},{"comment":"The manuscript reports that KAN was used with 'default parameters' but does not report the number of KAN layers, hidden widths, grid size, spline order, or the number of training epochs. The same applies to the TKAN variants beyond the number of sub-layers. Given that the central claim is a three-orders-of-magnitude improvement over recurrent baselines, the absence of these architectural and training details makes the result non-reproducible even with the provided code link.","section":"Section 4, paragraph 1"},{"comment":"The Kigali precipitation row for TKAN (5 Sub-layers) reports MSE = 23.5482 and R² = 0.1568, which implies a test variance of 27.93 mm², while other models in the same table imply a variance around 37.5 mm² (e.g., LSTM: 29.4953 / (1 - 0.2145) ≈ 37.55). This is another indication of inconsistent metric computation, although it is less central than the temperature discrepancy.","section":"Section 4, Table 1"}],"minor_comments":[{"comment":"The phrase 'the most best performance' should be corrected to 'the best performance.'","section":"Section 4, paragraph 2"},{"comment":"The figures labeled 'predicted vs. actual temperature' appear with small or illegible axis labels and no quantitative scale; please enlarge the axes and add units so the reader can visually verify the claimed near-perfect fit.","section":"Section 4, Table 2 and Figures 4-5"},{"comment":"Several references are incomplete, for example [1] lacks a publication venue and year, and [8] appears to have a formatting error ('2002 2002'); please unify the bibliography style.","section":"References"},{"comment":"The description of the scaling choices is confusing: it says MinMaxScaler [0,1] was used for T2M and PS with recurrent models, and [-1,1] for PREC, but 'for the other models, all variables were scaled to [0,1]'. This makes it unclear which models and which targets are evaluated in original units; please clarify the exact scaling and inverse-scaling procedure for each metric reported.","section":"Section 3.1.2"}],"recommendation":"reject","confidential_remarks":"The paper's main selling point is the KAN temperature result, but that result is arithmetically inconsistent with the RNN rows in the same table, and the split is not specified as temporal. These are load-bearing problems that require a full re-analysis of the experimental protocol, not a local fix. Even if the authors correct the metrics, the near-perfect R² values are likely to vanish under a chronological split. A rejection with a detailed explanation is appropriate; the authors could resubmit after a genuinely out-of-sample benchmarking with corrected metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe headline temperature result in this manuscript is arithmetically impossible, and that's the main thing you need to know. Table 2 reports KAN with MSE=0.0014 and R2=0.9986 in Abidjan, which implies a test-set variance of ~1 C^2. Every RNN baseline in the same table implies ~44 C^2 (LSTM: 7.26/(1-0.835)=44.0). Same pattern in Kigali. No predictor can have both numbers on the same test set, so the result is internally inconsistent, not just surprising. This is a load-bearing flaw.\n\nWhat's legitimately good: the paper uses real station data (2010-2024) for two under-studied African cities, compares six model families, and provides code. The precipitation and pressure results are plausible—RNNs win on pressure, TKAN variants are competitive on low-rainfall precipitation. The GeLU/MiSH activation variants are minor but documented.\n\nSoft spots beyond the arithmetic: the 72/8/20 split is not stated to be chronological, so a random split of daily series with autocorrelation could leak near-twins into the test set. There is no persistence baseline, so even if the KAN numbers were consistent, the improvement claim would be hard to evaluate. And the R2/MSE inconsistency suggests a bug in the evaluation code, probably a scale mismatch.\n\nI agree with the reject verdict. The central claim is unsupported and the numbers contradict each other. But the benchmark structure and data collection are worth acknowledging—this isn't a cynical paper. If the authors fix the evaluation (temporal split, persistence baseline, correct scaling), it could become a useful regional benchmark. As submitted, though, it needs a desk reject.\n\nRecommendation: desk reject; the internal inconsistency alone warrants that.","headline":"KAN temperature numbers are internally inconsistent; as written the main result is unsupported, though the benchmark setup is decent.","tokens_in":13663,"tokens_out":3527,"would_cite":false,"duration_ms":35574,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A default Kolmogorov-Arnold Network predicts next-day temperature in two tropical cities with $R^2$ above 0.998, far ahead of deep recurrent baselines.","keywords":["Kolmogorov-Arnold Networks","weather forecasting","tropical Africa","recurrent neural networks","LSTM","TKAN","temperature prediction","precipitation prediction"],"falsifier":"Re-run the KAN temperature experiment with the final 20% of the daily series held out as the test set and report MSE and $R^2$; if $R^2$ falls from about 0.999 to the 0.75--0.86 range seen for the recurrent baselines, the claimed order-of-magnitude superiority is an artifact of the split.","tokens_in":12670,"feed_emoji":"🌤️","tokens_out":11661,"duration_ms":100737,"temperature":0.7,"pith_summary":"The paper sets out to determine whether spline-based Kolmogorov-Arnold networks can outperform deep recurrent networks for next-day local weather prediction in tropical Africa. Using 14 years of daily station data from Abidjan and Kigali, it benchmarks LSTM, GRU, BiLSTM, BiGRU, an ensemble, KAN, and TKAN (including two new activation variants) on temperature, precipitation, and surface pressure. Its central finding is that the default KAN forecasts 2-meter temperature almost exactly, with $R^2$ above 0.998 and MSE below $0.0014\\,^\\circ\\mathrm{C}^2$ in both cities, while recurrent baselines sit near $R^2 = 0.83$--$0.86$. It also reports that TKAN variants with GeLU and MiSH activations give the best precipitation errors in low-rainfall conditions, and that classical RNNs remain best for pressure. If these results hold out of sample, they point to a lightweight, data-efficient alternative to numerical weather prediction for localized forecasting.","feed_headline":"Kolmogorov-Arnold networks beat deep RNNs on tropical temperature","feed_subtitle":"Default KAN reaches R² above 0.998 for temperature in Abidjan and Kigali, versus about 0.86 for LSTM and GRU baselines.","key_machinery":"The load-bearing object is the Kolmogorov-Arnold Network, which replaces each fixed neuron activation with a learnable univariate B-spline on the edge, so a layer is a composition of spline functions rather than a matrix of weights followed by a fixed nonlinearity. The paper uses the layer form $\\Phi_l = W_b\\, b(x) + W_s\\, \\mathrm{Spline}(x)$ with the SiLU base activation, and for the temporal variant TKAN it couples these KAN sub-layers with LSTM-style forget, input, and output gates to carry memory across time steps. That machinery is what the paper credits for smooth, local, data-efficient approximations; the custom variants swap SiLU for GeLU or MiSH in the base function, and the comparison against deep RNNs is what isolates the contribution of the spline representation.","core_discovery":"On its own terms, the paper's central discovery is that a default-configured Kolmogorov-Arnold Network predicts daily 2-meter temperature from a 14-day window of ten weather variables with near-perfect accuracy: $\\mathrm{MSE}=0.0014\\,^\\circ\\mathrm{C}^2$ and $R^2=0.9986$ in Abidjan, and $\\mathrm{MSE}=0.0003\\,^\\circ\\mathrm{C}^2$ and $R^2=0.9998$ in Kigali, outperforming every recurrent baseline by roughly three orders of magnitude in mean squared error. For precipitation, the temporal KANs---especially the new GeLU and MiSH variants---produce the lowest absolute errors in Kigali, a low-rainfall regime, while KAN itself gives the highest $R^2$ on precipitation in both cities. For surface pressure, the deep RNNs (LSTM, BiLSTM, GRU, BiGRU, ensemble) remain superior, with $R^2 \\approx 0.83$--$0.86$, while KAN drops to about $0.50$. The paper interprets this as evidence that spline-based architectures are data-efficient for smooth targets like temperature but need temporal memory mechanisms to compete on noisier variables.","pith_inferences":["A direct next step is to rerun the temperature benchmark using only the final contiguous 20% of the daily series as the test set and report the KAN-versus-RNN gap in $R^2$; this would remove any ambiguity about out-of-sample evaluation.","The same protocol could be extended to 3- and 7-day horizons; if KAN's edge is purely one-step persistence, its advantage should shrink rapidly as the horizon grows.","For precipitation, reporting categorical skill (e.g., probability of detecting a rain day) alongside continuous errors would connect these benchmarks to early-warning use, since the reported MAPE values are in the hundreds of percent.","The pressure results suggest a hybrid architecture---RNN gates for memory plus spline layers for local approximation---might combine the strengths of both families; the paper's data do not test this directly."],"forward_implications":["A default KAN, with no hyperparameter tuning, can serve as a lightweight next-day temperature forecaster for tropical cities using only daily station observations and a 14-day history.","For precipitation in low-rainfall regimes, TKAN variants using GeLU or MiSH activations are the preferred spline-based option over the standard SiLU TKAN.","Surface pressure forecasting should continue to rely on recurrent architectures; replacing them with plain KAN would degrade $R^2$ from about 0.85 to about 0.50.","The same KAN/TKAN setup transfers across two contrasting climates (coastal Abidjan and highland Kigali) without re-architecting, supporting the idea of location-agnostic, data-driven forecasting."],"supporting_citations":[{"why":"It defines the default Kolmogorov-Arnold Network architecture and layer formula that the paper applies without modification.","marker":"[34]"},{"why":"It introduces TKAN and its gating equations, the temporal baseline that the paper extends with new activations.","marker":"[17]"},{"why":"It states the Kolmogorov-Arnold representation theorem that underlies the KAN layer formula.","marker":"[18]"},{"why":"It supplies the GeLU activation used in the custom GeLU TKAN variant.","marker":"[26]"},{"why":"It supplies the MiSH activation used in the custom MiSH TKAN variant.","marker":"[35]"},{"why":"It provides the prior KAN weather-forecast result (solar radiation and temperature) that motivates the claim that spline architectures beat recurrent baselines.","marker":"[16]"},{"why":"It gives a KAN-based meteorological downscaling and correction study that supports the applicability of spline networks to weather fields.","marker":"[33]"},{"why":"It defines the LSTM baseline that the paper benchmarks against.","marker":"[27]"},{"why":"It defines the GRU baseline that the paper benchmarks against.","marker":"[13]"}],"fun_headline_variants":["KAN near-perfect on tropical temperature, RNNs still win for pressure","GeLU-MiSH tweaks improve temporal KAN for low-rainfall forecasting","KAN beats deep RNNs by 1000x on Abidjan and Kigali temperature","Spline-based KAN excels at heat, RNNs hold pressure edge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's train/validation/test split is described only as 72/8/20 and not as chronological, so the near-perfect KAN temperature scores could be inflated if randomly chosen test days have near-duplicate training days.","fun_headline_variants_meta":{"raw":{"variants":["KAN near-perfect on tropical temperature, RNNs still win for pressure","GeLU-MiSH tweaks improve temporal KAN for low-rainfall forecasting","KAN beats deep RNNs by 1000x on Abidjan and Kigali temperature","Spline-based KAN excels at heat, RNNs hold pressure edge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1743,"prompt_tokens":1164,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":780,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":780,"tokens_out":579,"duration_ms":7222,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:25:49.260397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the KAN temperature experiment with the final 20% of the daily series held out as the test set and report MSE and $R^2$; if $R^2$ falls from about 0.999 to the 0.75--0.86 range seen for the recurrent baselines, the claimed order-of-magnitude superiority is an artifact of the split.","supporting_citations":[{"cited_title":"Long short-term memory.Neural computation, 9:1735–80, 12 1997","cited_arxiv_id":null,"evidence_quote":"It defines the LSTM baseline that the paper benchmarks against."},{"cited_title":"On the properties of neural machine translation: Encoder–decoder approaches","cited_arxiv_id":null,"evidence_quote":"It defines the GRU baseline that the paper benchmarks against."},{"cited_title":"Givental, Boris A","cited_arxiv_id":null,"evidence_quote":"It states the Kolmogorov-Arnold representation theorem that underlies the KAN layer formula."},{"cited_title":"A revolutionary neural network architecture with interpretability and flexibility based on kolmogorov–arnold for solar radiation and temperature forecasting","cited_arxiv_id":null,"evidence_quote":"It provides the prior KAN weather-forecast result (solar radiation and temperature) that motivates the claim that spline architectures beat recurrent baselines."}],"review_version":1}