{"id":"1ec454a2-ec80-49fb-97d2-fad0a64e615d","arxiv_id":"2505.03188","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"On a one-year Stanford sky-image dataset, a frozen pretrained vision transformer predicts solar PV output worse than the CNN baseline (overall RMSE 3.35 vs 2.36), contradicting the paper's own 'almost as well' framing.","lead":"This paper tests whether a vision transformer, a popular deep learning architecture, can predict solar panel power output from sky images as well as an existing convolutional neural network. It finds the transformer clearly underperforms the baseline, with total error about 40 percent higher and sunny-day error about five times worse.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's parity claim is contradicted by the paper's own tables: ViT RMSE is higher on sunny, cloudy, and overall days, and the reported ViT RMSE is not even a single consistent number.","rationale":"I agree with the reader's REJECT verdict and with the emphasis on baseline commensurability, but my primary load-bearing concern is broader: the paper's own reported numbers, taken at face value, contradict the abstract's 'almost as well' claim, and the ViT results are internally inconsistent across Tables 2, 3, 4, and 5. This is not a question of external consensus or a subtle modeling choice; it is an internal contradiction in the evidence presented. The baseline-commensurability issue raised by the reader is real and would matter if the ViT numbers were close to the baseline, but here the gap is large enough that the claim fails even before that question is resolved. I would not credit the single negative data point as statistically supported because no code, seeds, or error bars are provided and the manuscript has unresolved placeholders and a truncated conclusion. The underlying idea, applying a pretrained ViT to sky-image PV nowcasting, is reasonable and the negative result could be a useful observation, but the paper as written does not support its central claim.","tokens_in":4924,"tokens_out":2997,"duration_ms":28139,"concrete_test":"Re-run the original SUNSET CNN in the exact ViT pipeline on the same held-out days: same 256x256 training images, same HuggingFace 224x224 feature-extractor resizing at inference, same power-label normalization, and one unambiguous test split with the specific days listed. If the recomputed baseline RMSE is close to the ViT's 3.35, the gap between Tables 1 and 5 is a pipeline artifact; if it remains near 2.36, the abstract's parity claim is contradicted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Taking Tables 1 and 5 at face value, the ViT is worse in every reported category: sunny RMSE 2.62 vs 0.52, cloudy 4.01 vs 3.38, and overall 3.35 vs 2.36. That is about 42% higher overall error and roughly five times higher error on sunny days, so the abstract's claim that the transformer 'performs almost as well' and the caveat that it is worse only on sunny days are directly contradicted. The paper also reports three different ViT overall RMSE values: 2.61 in the pilot (Table 3), 3.38 in Table 2, and 3.35 in Table 5, while Table 4 still contains XX placeholders. If Table 5 is the intended full-data result, the central claim is false. If Table 3 or 4 is the intended result, that result comes from a 654-pair pilot with no hyperparameter tuning, so it cannot support a parity claim. Independently, the baseline SUNSET numbers are imported from prior work rather than recomputed, and the test set is described as both 20 days and 10 days, so no validated cross-pipeline comparison exists. The paper's own data therefore fail to support the abstract's conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies a pretrained HuggingFace Vision Transformer (ViT) with a fully-connected regression head to the task of short-term solar PV power output nowcasting from sky images, comparing its RMSE with the CNN-based SUNSET baseline of the Stanford EAO lab. The authors report experiments with a 654-pair pilot, a warm-start full-data run, and a hyperparameter-tuned frozen-layer run, and they claim in the abstract that the transformer performs almost as well as the baseline but worse on sunny days.","tokens_in":4972,"tokens_out":3084,"duration_ms":26366,"significance":"If the central claim were supported, the paper would show that a frozen, pretrained ViT is a near-parity substitute for a task-specific CNN in a regression-from-sky-images problem, which would be a useful data point for the solar-forecasting community. The paper also leverages publicly available pretrained weights and a real-world dataset, which is a strength. However, the manuscript does not deliver a validated comparison: the baseline numbers are imported rather than recomputed, the ViT results are internally inconsistent, and the abstract's parity claim is contradicted by the paper's own tables. These problems are load-bearing rather than cosmetic, so the result as presented cannot be accepted.","major_comments":[{"comment":"The abstract's central claim that the transformer 'performs almost as well as the baseline model' and 'performs worse on sunny days' is contradicted by the reported quantitative results. Table 5 gives ViT RMSE values of 2.62 (sunny), 4.01 (cloudy), and 3.35 (overall), while Table 1 gives SUNSET baseline RMSE values of 0.52 (sunny), 3.38 (cloudy), and 2.36 (overall). On these numbers the ViT is worse in every category: about five times worse on sunny days and about 42% worse overall, and it is also worse on cloudy days. The sunny-day-only caveat in the abstract is therefore incomplete and the parity claim is unsupported by the paper's own evidence.","section":"Abstract; Section 5.3, Table 5 vs Section 4.1, Table 1"},{"comment":"The paper reports three different overall RMSE values for the ViT: 2.61 in the pilot (Table 3), 3.38 as the 'best model' (Table 2), and 3.35 for the frozen-layer full-data model (Table 5), while Table 4 lists 'XX' placeholders for the sunny and cloudy categories despite reporting an overall value of 2.61. The manuscript never explains which result is the definitive one, leaving the reader unable to determine the model's actual performance. This internal inconsistency undermines the paper's main empirical claim.","section":"Section 5 (Tables 2, 3, 4, 5)"},{"comment":"The comparison between the ViT and the SUNSET baseline is not a controlled experiment. Section 3 states that the test set consists of 20 days (13,689 pairs, 10 sunny and 10 cloudy), but Section 4.1 describes a test set of 'ten days (five cloudy and five sunny)'. Moreover, the baseline RMSE values in Table 1 are quoted from prior work rather than recomputed in this pipeline: the paper notes only that 'This is the same test dataset used by the baseline SUNSET model.' The ViT pipeline resizes 256x256 images to 224x224 inside the HuggingFace feature extractor and uses a shuffled 95/5 train/validation split, whereas the baseline operated on 64x64 images under an unstated split and preprocessing regime. Without re-running the baseline under the same conditions, the reported RMSE gap could be a pipeline artifact rather than an architecture comparison, and the paper's parity conclusion rests on an unverified commensurability assumption.","section":"Section 3 vs Section 4.1; Table 1 provenance"},{"comment":"The pilot result of RMSE 2.61 is based on a single day of data (654 image-output pairs) with no hyperparameter tuning, and the paper describes the split as '90% − 5%', which is arithmetically incomplete. This pilot cannot support the inference in Section 5 that 'transformers are well-suited to our PV power output regression task,' nor can it substantiate the abstract's parity claim. The subsequent full-data runs produce different and worse overall RMSE values, so the pilot appears to be a small-sample artifact rather than a reliable measure of performance.","section":"Section 5, pilot experiment (Table 3)"}],"minor_comments":[{"comment":"The description of the test set as 'ten days (five cloudy and five sunny)' directly contradicts Section 3's '20 days of image-PV output pairs... 10 of these days are classified as sunny and the other 10 as cloudy'; the paper should reconcile these counts and specify which test set was actually used for each reported result.","section":"Section 4.1"},{"comment":"The sentence 'Using 654 samples... split 90% − 5% into training and validation sets' is incomplete: the two percentages sum to 95%, leaving 5% unattributed; in addition, the split description differs from the 95/5 split stated in Section 3.","section":"Section 5.1"},{"comment":"Table 4 contains 'XX' placeholders for the sunny and cloudy RMSE values; a published table should either report the actual values or the table should be removed, since its presence signals that the experimental results are incomplete.","section":"Table 4"},{"comment":"Several figure references are garbled: Section 5.2 says 'The model performance is presented in 4' without a figure number, Section 5.3 refers to 'Table The model performance' and 'figure 5.3', and the captions for Figures 7, 8, and 9 are not mentioned in the text.","section":"Throughout, Figures 4, 5, and 6"},{"comment":"The Conclusion section is incomplete: it begins with 'Additionally, we may incorporate a CNN component...' and does not actually summarize the findings, restate the main conclusion, or discuss implications; the paper appears to be missing its concluding paragraphs.","section":"Section 6 (Conclusion)"},{"comment":"There are typographical and grammatical errors, including 'Univeristy' twice in the author affiliations, 'great improves' in Section 5.2, and the special character rendering in reference [7] ('Szu˝cs'); these should be corrected in revision.","section":"Title page and Section 5.2"},{"comment":"The phrase 'using a mean-squared error loss on a one-class classification' is not standard terminology for a regression task with a single continuous output; the paper should describe the model as a single-output regression head with MSE loss.","section":"Section 4"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an incomplete project report rather than a finished research paper: the central abstract claim is contradicted by the paper's own tables, multiple tables are inconsistent (including XX placeholders), the baseline is not re-run in the same pipeline, and the conclusion section is missing. The comparison problem is not fixable with minor edits; a valid assessment of transformer-SUNSET parity would require recomputing the baseline under identical preprocessing and resolving the internal numerical inconsistencies, which is beyond the scope of a revision. The paper's novelty claim (that a frozen ViT approaches a task-specific CNN) is, on the reported numbers, false, so I recommend rejection. There is no indication of misconduct; the issues are accuracy and completeness of reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I read the paper myself and I agree with the rejection. The abstract says the transformer \"performs almost as well as the baseline\" and is worse only on sunny days. Table 1 gives SUNSET RMSEs of 0.52 sunny, 3.38 cloudy, 2.36 overall; Table 5 gives ViT 2.62, 4.01, 3.35. That is five times worse on sunny days and 42% worse overall. Cloudy is worse too. That is not \"almost as well\" by any reading.\n\nCredit where due: the paper does one genuinely new empirical thing. It applies a frozen pretrained HuggingFace ViT to the SUNSET Nowcast test set and reports the resulting RMSEs. That specific measurement does not appear in any cited paper, and the authors are transparent enough to put the numbers in tables rather than hiding them. The data collection effort (103k sky-image/power pairs) is real, and using an external baseline rather than inventing their own is the right instinct.\n\nBut the soft spots are load-bearing. The RMSE numbers do not stay consistent: 2.61 in the pilot, 3.38 in Table 2, 2.61 in Table 4, 3.35 in Table 5. Table 4 literally has XX placeholders. The test set is described as 20 days in one place and ten days in another. The conclusion is truncated mid-sentence. No code, data, or seeds are provided. The baseline SUNSET numbers are imported from prior work, not recomputed in the ViT pipeline, so the comparison may be a preprocessing artifact: the ViT goes through a 224x224 feature extractor while the baseline was evaluated at 64x64. That does not explain the internal contradictions, but it means even the fixed numbers would not be commensurable without extra work.\n\nThe citation pattern is thin — no transformer-based solar forecasting work after 2020 — but that is a minor issue compared to the claim/data mismatch.\n\nMy assessment: this is an unfinished project draft, not a paper. The central claim is contradicted by its own evidence, and the completeness problems (XX, truncated text, contradictory test-set counts) would make referee time a waste. I would recommend desk rejection, not because the idea is bad — applying a ViT to sky-image regression is a reasonable thing to try — but because the authors need to first produce a consistent set of numbers and a properly rerun baseline. If they do that, the single negative data point for a frozen pretrained ViT might be worth a short note in a specialist venue. As submitted, it is not ready for peer review.","headline":"The abstract claims the ViT is almost as good as the CNN baseline, but the paper's own tables show it is 42% worse overall and five times worse on sunny days, so the central claim is contradicted by the reported numbers.","tokens_in":5737,"tokens_out":2561,"would_cite":false,"duration_ms":21587,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen vision transformer nearly matches a CNN baseline for short-term solar PV power forecasting, though it lags on sunny days; the paper's headline claim is that a ViT predicts solar output from sky images almost as well as the SUNSET…","keywords":["solar PV forecasting","vision transformer","sky images","nowcasting","transfer learning","regression","renewable energy"],"falsifier":"Re-run the SUNSET CNN baseline on the identical 20-day test set, using the same 224x224 rescaling and the same train/validation split protocol applied to the ViT, and compare RMSEs. If the baseline's overall RMSE rises from 2.36 kW to roughly the ViT's 3.35 kW or higher, the 'almost as well' conclusion is a pipeline artifact; if it stays near 2.36 kW, the ViT is genuinely worse than the CNN on this task.","tokens_in":4501,"feed_emoji":"☀️","tokens_out":2909,"duration_ms":23854,"temperature":0.7,"pith_summary":"This paper asks whether a pure transformer architecture can replace a convolutional neural network in the task of predicting solar PV power output from sky images. Using a frozen, pretrained HuggingFace Vision Transformer with a small trainable head, the authors report RMSE values close to those of the SUNSET CNN baseline on a held-out test set, concluding that the transformer performs almost as well in the PV output prediction task. The significance, if the claim holds, is that a generic image-classification transformer pretrained on ImageNet can serve as a near-parity feature extractor for a specialized renewable-energy regression problem without bespoke architecture design. The paper also notes that the transformer performs worse on sunny days, a caveat that matters because sunny conditions are precisely where solar forecasting is easiest and most reliable.","feed_headline":"Frozen vision transformer near-parity for solar nowcasting","feed_subtitle":"A pretrained ViT predicts PV output from sky images with RMSE 3.35 versus SUNSET CNN's 2.36, but lags badly on sunny days.","key_machinery":"The central object is the HuggingFace Vision Transformer (ViT), a pure transformer architecture that splits an image into 16x16 patches, embeds them linearly, and processes the sequence with self-attention layers. The paper loads weights pretrained on ImageNet, freezes all but the final layer normalization, pooler, and fully connected head, and trains only that head with MSE loss to regress PV power output. The load-bearing mechanism is feature transfer: the attention-based representations learned on natural images are reused, without fine-tuning the body, as the input to a small regression head for solar forecasting.","core_discovery":"A pretrained Vision Transformer (ViT) with its weights mostly frozen, followed by a fully-connected regression head, can predict solar PV power output from a sky image with root mean squared error (RMSE) of 3.35 kW overall on a 20-day test set, compared with the SUNSET CNN baseline's 2.36 kW. On cloudy days the ViT reaches 4.01 kW versus the baseline's 3.38 kW, and on sunny days 2.62 kW versus 0.52 kW. The paper states this means the transformer performs almost as well as the baseline in the PV output prediction task, while acknowledging it performs worse on sunny days.","pith_inferences":["The paper's own tables show the ViT is worse on both sunny and cloudy days, and roughly 42% worse overall, so the phrase 'almost as well' is doing more work than the reported RMSEs support; a fair reader should treat the headline claim as optimistic relative to the numbers.","Because the baseline numbers come from prior work measured at 64x64 resolution while the ViT pipeline rescales to 224x224 inside the HuggingFace feature extractor, the comparison likely bundles preprocessing differences into the architecture comparison; an apples-to-apples rerun could shift both the magnitude and the sign of the gap.","The near-parity on cloudy days, where the ViT error is 4.01 versus 3.38, suggests the transformer captures cloud-structure cues reasonably well even without training the body, which if confirmed would make the attention mechanism a genuinely competitive feature extractor for irradiance-driven tasks.","A testable extension is to evaluate the frozen ViT on the same 64x64 inputs the baseline actually used, by downscaling inside the model or retraining the head on 64x64 patches, to separate the architecture effect from the resolution effect."],"forward_implications":["If the ViT nearly matches the CNN baseline, then future solar nowcasting systems can be built from off-the-shelf pretrained image transformers, saving the effort of designing and training task-specific CNNs from scratch.","The sunny-day degradation, if real, would push hybrid designs that switch to a simpler model or a different head when sky conditions are clear.","The ViT's ability to work as a feature extractor at 224x224 resolution suggests resolution flexibility: the model can ingest lower-resolution sky images and still produce usable forecasts, which is relevant for deployment on constrained hardware.","Since the model is frozen, the result would support a workflow where the expensive transformer body is shared across many solar sites and only the small regression head is retrained per site.","The extension of the same architecture to the SUNSET Forecast problem (multi-step-ahead forecasting) becomes a low-cost next step if the nowcast result is taken at face value."],"supporting_citations":[{"why":"Defines the SUNSET Nowcast CNN baseline whose RMSE values (0.52 sunny, 3.38 cloudy, 2.36 overall) are the comparison target for the ViT.","marker":"[7]"},{"why":"Supplies the ViT architecture and the pretraining-on-large-datasets approach the paper adapts to the regression task.","marker":"[11]"},{"why":"Establishes that the transformer's attention mechanism can substitute for convolutions, motivating the pure-transformer choice.","marker":"[13]"},{"why":"Provides the HuggingFace implementation and pretrained weights the paper freezes and uses as a feature extractor.","marker":"[14]"}],"fun_headline_variants":["ViT nearly matches CNN for solar PV, but sunny days lag","Transformer solar forecast: overall close, sunny days worse","Frozen vision transformer near-parity, weaker on sun","Solar PV: ViT RMSE 3.35 vs CNN 2.36, but sunny gap","Transformer almost matches CNN for solar, not on sun"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on assuming the baseline SUNSET RMSE numbers are commensurable with the ViT numbers, even though the baseline was never re-run in this pipeline and the two models use different image resolutions, preprocessing, and train/validation splits.","fun_headline_variants_meta":{"raw":{"variants":["ViT nearly matches CNN for solar PV, but sunny days lag","Transformer solar forecast: overall close, sunny days worse","Frozen vision transformer near-parity, weaker on sun","Solar PV: ViT RMSE 3.35 vs CNN 2.36, but sunny gap","Transformer almost matches CNN for solar, not on sun"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1271,"prompt_tokens":840,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":456,"tokens_out":431,"duration_ms":5042,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:58:44.627677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SUNSET CNN baseline on the identical 20-day test set, using the same 224x224 rescaling and the same train/validation split protocol applied to the ViT, and compare RMSEs. If the baseline's overall RMSE rises from 2.36 kW to roughly the ViT's 3.35 kW or higher, the 'almost as well' conclusion is a pipeline artifact; if it stays near 2.36 kW, the ViT is genuinely worse than the CNN on this task.","supporting_citations":[{"cited_title":"The use of trans- formers for computer vision tasks is also quite recent [10]","cited_arxiv_id":null,"evidence_quote":"Defines the SUNSET Nowcast CNN baseline whose RMSE values (0.52 sunny, 3.38 cloudy, 2.36 overall) are the comparison target for the ViT."},{"cited_title":"We therefore applied a pure transformer architecture to the solar forecasting problem","cited_arxiv_id":null,"evidence_quote":"Establishes that the transformer's attention mechanism can substitute for convolutions, motivating the pure-transformer choice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HuggingFace implementation and pretrained weights the paper freezes and uses as a feature extractor."}],"review_version":1}