{"id":"116aeb79-577f-461f-9594-09c33174d2ae","arxiv_id":"2504.19066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces EWRA, a two-stage reasoning-aware fine-tuning method for small language models, together with the ExtremeWeatherNews dataset of over 127,000 extreme-weather news sentences, and evaluates both on three weather analytics tasks.","lead":"This paper builds a dataset of extreme weather news articles and uses a large AI model to generate step-by-step reasoning labels, which are then used to fine-tune smaller AI models. The authors report that this reasoning-based fine-tuning improves how small models categorize impacts, topics, and emotions in weather news, with performance varying by task and model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline EWRA advantage is not robust: one seed, no significance tests, and the paper's own tables show EWRA losing in several task/model cells, so the 5.2% Tab. 1 gain may be noise.","rationale":"I read the paper in good faith: the dataset and task formulation are useful, and EWRA's two-stage curriculum is a reasonable way to transfer reasoning traces to small models. The reader's conditional verdict matches my own. However, I locate the load-bearing weakness differently. The reader's weakest_assumption focuses on unvalidated teacher labels; that concern is partly addressed by the human review of explanations described in S.2.1, though not with quantitative agreement metrics. The more direct threat to the central claim is that the headline result is a single unreplicated SRC gap with no uncertainty quantification, and the paper's own tables contradict the uniform-superiority wording in the conclusion. My proposed test is a multi-seed rerun with paired bootstrap inference on the 200-sample gold set. If the gap persists with tight confidence intervals across seeds, the statistical concern is discharged and the remaining issue is external validity of the taxonomy. If it does not persist, the conclusion must be moderated. Because the required evidence is missing rather than contradicted beyond repair, the appropriate verdict remains conditional, matching the reader's verdict; hence no change is recommended.","tokens_in":18455,"tokens_out":7118,"duration_ms":74704,"concrete_test":"Rerun the Qwen2.5-3B Vulnerability/Impact/Emergency comparison from Tab. 1 with 10 independent random seeds (same data and hyperparameters, seed varied), and for each seed compute SRC on the same 200-sentence gold set. Report mean±SD and a paired bootstrap 95% confidence interval for the EWRA minus ReasonExplicit-SFT difference, plus a permutation test on the 200 gold samples. If the confidence interval includes zero or EWRA wins fewer than 9 of 10 seeds, the reported 5.2% gain is not robust and the conclusion should be weakened to 'competitive on some tasks.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—EWRA 'substantially improves' SLM alignment and 'outperforms' both standard SFT and other reasoning-based approaches (§5)—rests on a single unreplicated number: SRC 0.8230 vs 0.7820 for Qwen2.5-3B on Vulnerability/Impact/Emergency (Tab. 1), a 5.2% relative gain over ReasonExplicit-SFT. Training fixes one seed (3407, §4.1), no variance or significance tests are reported, and the gold set has only 200 sentences per task (§4.4). The same tables contain counterexamples: on Emotion analysis, EWRA (0.8708) trails ReasonExplicit-SFT (0.8716) for 3B; on subtopic labeling, zero-shot 3B reaches 0.7727 while EWRA reaches 0.7149; on keyword extraction, ReasonImplicit-SFT 3B (0.3269) beats EWRA (0.2856). The authors themselves acknowledge these shortfalls in §4.5, yet the conclusion states uniform superiority. A 0.041 SRC gap on n≈200 can easily arise from sampling variation; without repeated seeds and paired inference the headline improvement is not distinguishable from noise. The shared-taxonomy evaluation (teacher prompts and gold annotations both follow Tab. S3 definitions) compounds the problem: even a statistically reliable gap may reflect alignment to the authors' category scheme rather than independent domain judgments. The S.2.1 human review of generated explanations mitigates, but does not replace, the need for statistical replication.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ClimaEmpact, a framework combining a new corpus (ExtremeWeatherNews, 127,454 location-filtered sentences from 60 extreme weather events) with an LLM-generated reasoning alignment dataset (ExtremeAlign) and a two-stage curriculum fine-tuning method (EWRA) that first trains small language models on implicit-prompt reasoning and then on explicit-prompt reasoning with category definitions. The method is evaluated on three tasks: vulnerability/impact/emergency assessment, topic/subtopic labeling with keyword extraction, and emotion analysis, using Spearman rank correlation, Jaccard index, and BERTScore against a 600-sentence human gold set. The authors claim that EWRA substantially improves domain alignment of SLMs and outperforms standard SFT and other reasoning-based approaches, especially for Qwen2.5-3B-Instruct.","tokens_in":18761,"tokens_out":2847,"duration_ms":28953,"significance":"If the central claim were fully supported, the contribution would be useful: the paper introduces a new domain corpus, a synthetic reasoning-alignment dataset, a concrete two-stage curriculum, and an online dashboard for extreme weather analytics, and it addresses a real gap in localized extreme weather impact assessment. The dataset release and the explicit taxonomy for vulnerability/impact/emergency analysis are constructive contributions. However, the significance is currently limited by two issues: the reported superiority of EWRA is not consistent across the paper's own tables, and the evaluation pipeline shares the same taxonomy and category definitions between the teacher prompts and the human gold annotation, so the benchmark primarily measures alignment with the authors' category scheme rather than with an independently anchored external standard. These issues are fixable within the manuscript's scope, so the appropriate decision is major revision rather than rejection.","major_comments":[{"comment":"The conclusion that EWRA 'substantially improves' alignment and 'outperforms both standard supervised fine-tuning and other reasoning-based approaches' is contradicted by the paper's own experimental tables. In Tab. 1, EWRA on Qwen2.5-1.5B achieves SRC 0.7140 on Vulnerability/Impact/Emergency, below ReasonExplicit-SFT's 0.7960. In Tab. 3, EWRA on Qwen2.5-1.5B emotion analysis (0.8308) is below Direct-SFT (0.8389), and on Qwen2.5-3B emotion analysis (0.8708) is below ReasonExplicit-SFT (0.8716). In Tab. 2, zero-shot Qwen2.5-3B subtopic labeling (0.7727) exceeds EWRA (0.7149), and ReasonImplicit-SFT on 3B keyword extraction (0.3269) exceeds EWRA (0.2856). The conclusion should be revised to a more qualified claim, such as 'competitive or superior in several settings,' or the authors must show a statistically robust aggregate advantage that justifies the current wording.","section":"§5 and Abstract vs. Tables 1–3"},{"comment":"The load-bearing headline result, SRC 0.8230 vs. 0.7820 for Qwen2.5-3B on Vulnerability/Impact/Emergency (Tab. 1), rests on a single training run with a fixed seed (3407), no repeated seeds, no confidence intervals, and no significance tests. The gold test set contains only 200 sentences per task (§4.4), so a 0.041 SRC difference is well within the range of sampling variation. The same limitation applies to the ReverseEWRA ablation in Tab. 4. The authors should report multiple seeds with mean and variance, or paired significance tests across the same gold set, before claiming that EWRA provides a reliable improvement over ReasonExplicit-SFT.","section":"§4.1, §4.4, and §4.5 (statistical robustness)"},{"comment":"The method assumes, without validation, that Qwen2.5-32B-Instruct is an 'expert-level model in extreme weather reasoning' whose generated reasoning paths and probability assignments can serve as training labels. The one-shot prompts embed the taxonomy and category definitions of Tab. S3, and the human gold set is annotated using the same taxonomy and guidelines (§S.2.2). This design makes the benchmark internally consistent with the teacher, but it does not establish that the resulting SLM behavior corresponds to genuinely correct or externally validated extreme weather analysis. The authors should provide either a human validation study of teacher outputs against independent expert judgments, or an additional gold set built from an independently derived taxonomy, to support the claim that EWRA transfers domain reasoning rather than learning to mimic the teacher's category scheme.","section":"§3.2 and §S.2.2 (teacher validity and evaluation circularity)"},{"comment":"The discussion in §4.5 acknowledges several EWRA shortfalls (e.g., subtopic labeling and keyword extraction), yet the conclusion in §5 states uniform superiority. The authors should reconcile these statements by presenting an aggregate analysis, such as averaged ranks across all tasks and metrics, and by explicitly identifying the settings where EWRA does not win. Without such an analysis, the narrative in the abstract and conclusion overstates the evidence.","section":"§4.5 (discussion inconsistencies)"}],"minor_comments":[{"comment":"The phrase 'Done-shot' appears to be a typo for 'one-shot'; please correct it.","section":"§3.2"},{"comment":"In the Spearman rank correlation formula, the quantity d_i is not defined in the text; please state that d_i is the difference between the ranks of the i-th observation in X and Y.","section":"§4.2"},{"comment":"The table formatting with slashes (e.g., '0.3040/0.528830') is hard to read, and the entry '30.1670' in the ReasonImplicit-SFT row appears to be a typo; please clean up the table formatting and define what the two numbers represent.","section":"Tab. 2 and §4.3"},{"comment":"The text states that EWRA improves Jaccard Index by 2.7% on the 3B Vulnerability/Impact/Emergency task, but the table values 0.3340 vs. 0.3294 correspond to roughly a 1.4% relative improvement; please verify and align the reported percentages with the table.","section":"§4.5"},{"comment":"The description of the task-specific models is confusing: it refers to 'the third task' while the preceding sentence discusses the first and second tasks; please clarify which model is used for which task.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core idea is worth pursuing, and the authors have built a useful dataset and an online system. My main concern is that the headline claim of consistent superiority is not supported by the paper's own tables and is based on unreplicated, statistically fragile comparisons. The shared-taxonomy evaluation is also a validity concern, though it is not a fatal flaw if reframed as evaluating alignment with a specific category scheme. I would accept a revised version that adds multi-seed experiments with significance testing, validates the teacher outputs against independent human judgments, and replaces the overbroad conclusions with claims that match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the useful thing about this paper is the dataset, not the method. ExtremeWeatherNews is 127k sentences across 60 events, location-filtered, with a task taxonomy for vulnerability/impact/emergency, topics/subtopics, and emotion. That plus the 600-sample gold set with Fleiss kappa 0.88 is a real resource for the applied NLP and disaster analytics crowd. The EWRA training recipe is a two-stage curriculum variant of chain-of-thought distillation from an LLM teacher to small models; the paper's own references show that is an established family. The genuinely new pieces are the dataset and the task-specific taxonomy.\n\nThe authors are careful in places: they discuss why zero-shot beats EWRA on subtopic labeling, they ablate curriculum order (ReverseEWRA), and they report Fleiss kappa. So the work is not sloppy.\n\nBut the central claim in the conclusion — that EWRA 'substantially improves' alignment and 'outperforms' both SFT and other reasoning-based approaches — is not backed by their own tables. On the 1.5B model, EWRA is below ReasonExplicit-SFT on Vulnerability/Impact/Emergency (0.714 vs 0.796) and below Direct-SFT on emotion (0.8308 vs 0.8389). On 3B subtopic labeling, zero-shot (0.7727) beats EWRA (0.7149); on keyword extraction, ReasonImplicit-SFT (0.3269) beats EWRA (0.2856). The headline 5.2% SRC gain on 3B VIE is a single seed on a 200-sample gold set with no significance test. That gap could easily be noise.\n\nThe circularity point is moderate but real: the gold set was annotated under the same author-defined taxonomy and category definitions that are embedded in the teacher prompts. The evaluation rewards alignment to the authors' scheme, not independent external judgments. The human review of LLM explanations (S.2.1) and the high annotator agreement help, but they do not remove the concern. Also, the teacher (Qwen2.5-32B) is asserted to be 'expert-level' without validation against human gold.\n\nWho is this for? Researchers who want a ready-made corpus for extreme-weather text analytics, and people benchmarking reasoning-distillation methods in a new domain. The method itself will not surprise anyone who reads Ranaldi & Freitas or the CoT-distillation literature. I would send it to peer review, but with expectations of major revision: multiple seeds and significance tests, comparison to existing CoT distillation baselines, a released dataset and code, and some tempering of the claims.","headline":"The dataset and task taxonomy are the real contribution; the EWRA method is a two-stage CoT distillation variant whose claimed superiority is not supported by the paper's own tables or by any significance testing.","tokens_in":19352,"tokens_out":2284,"would_cite":true,"duration_ms":21584,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage curriculum that teaches a small language model to reason about extreme weather improves impact assessment, reaching a Spearman correlation of 0.8230 on a 3-billion-parameter model.","keywords":["extreme weather analytics","small language models","reasoning-aware alignment","curriculum fine-tuning","LLM-generated reasoning paths","ExtremeWeatherNews dataset","vulnerability impact assessment","emotion analysis"],"falsifier":"Have independent human annotators, using the published taxonomy but without seeing the teacher's outputs, label a fresh sample of ExtremeWeatherNews sentences, then measure agreement between Qwen2.5-32B-Instruct's one-shot probability assignments and those labels using Spearman correlation or calibration metrics. If the teacher's agreement is no better than the students' agreement with the gold set, or if the teacher's failure cases are exactly the cases the EWRA-trained students get wrong, then EWRA is distilling the teacher's bias rather than teaching domain reasoning.","tokens_in":1632,"feed_emoji":"🌩️","tokens_out":3245,"duration_ms":86945,"temperature":0.7,"pith_summary":"This paper tries to show that a small language model can be taught to produce expert-style extreme weather analyses by training it on reasoning paths distilled from a much larger model. The method, EWRA, first fine-tunes the small model on implicit reasoning examples and then on explicit reasoning examples that spell out category definitions. On the most demanding task—categorizing a sentence as vulnerability, impact, or emergency—the 3-billion-parameter model reaches a Spearman rank correlation of 0.8230, above 0.7820 for the strongest comparison method. If the result holds, agencies and researchers with limited compute could deploy cheap, fast models that still give grounded, explainable assessments of flood, heatwave, and storm impacts. The paper also releases two datasets: ExtremeWeatherNews, with 127,454 location-filtered sentences from 60 events, and ExtremeAlign, with 30,000 LLM-generated reasoning annotations.","feed_headline":"Implicit-first training sharpens small models' weather reasoning","feed_subtitle":"On impact assessment, the 3B model's Spearman correlation rises to 0.8230, beating standard fine-tuning.","key_machinery":"The load-bearing mechanism is the two-stage curriculum over LLM-generated reasoning paths. In the first stage, the small model is trained on implicit prompts—sentences plus step-by-step thoughts and probability outputs, but no category definitions—so it internalizes reasoning patterns without relying on prompt scaffolding. In the second stage, it is fine-tuned on explicit prompts that include detailed category definitions and inclusion/exclusion criteria. Training maximizes the log-likelihood of the full reasoning sequence, and attention is adapted with query-key LoRA so the model's stored factual knowledge is left intact. A one-shot teacher (Qwen2.5-32B-Instruct) produced the 30,000 training examples in ExtremeAlign, with distinct <think> and <output> sections and probability scores constrained to sum to one. Evaluation treats each task as a ranking problem, using Spearman rank correlation against a 600-sample human gold set as the primary measure of alignment.","core_discovery":"The paper's central claim is that EWRA—a two-stage curriculum that first trains a small language model on teacher-generated implicit reasoning and then on explicit reasoning with task definitions—produces small language models whose extreme weather analytics are better aligned with human-annotated rankings than standard supervised fine-tuning or single-stage reasoning training. The evidence is strongest on the vulnerability/impact/emergency assessment task with Qwen2.5-3B-Instruct, where EWRA reaches a Spearman rank correlation of 0.8230 versus 0.7820 for ReasonExplicit-SFT, along with a Jaccard Index of 0.3340 versus 0.3294. The paper also reports that reversing the curriculum order (explicit first, then implicit) drops the Spearman score to 0.7070 on that task, which it reads as evidence that the implicit-to-explicit ordering, rather than extra training time, drives the gain. The authors conclude that EWRA substantially improves the alignment of small models with domain-specific reasoning, outperforming both standard supervised fine-tuning and other reasoning-based approaches.","pith_inferences":["My inference: the same implicit-to-explicit curriculum could transfer to other specialized domains where expert annotations are scarce but large models can articulate plausible rationales, such as public health advisories, infrastructure risk, or financial disclosure.","My inference: because the gold set is annotated under the same taxonomy and prompt guidelines given to the teacher, the reported alignment may partly measure fidelity to the teacher's framing; a human baseline showing expert agreement with the gold set would clarify how much genuine domain skill the students have acquired.","My inference: a sharper test would be calibration—whether the probability scores EWRA produces match the empirical frequencies of human labels, not merely their ranks—since the paper reports only ranking correlation and leaves the quality of the confidence values untested."],"forward_implications":["If EWRA's effect is real, small models of 1.5B to 3B parameters can deliver reasoning-grounded extreme weather analytics without the cost of large-model inference, supporting near-real-time dashboards and resource-constrained deployments.","The implicit-first, explicit-second ordering appears load-bearing: reversing the curriculum on the 3B model drops the vulnerability/impact/emergency Spearman correlation from 0.8230 to 0.7070.","The method's advantage is task-dependent: on the 3B model's emotion analysis task, EWRA's Spearman correlation of 0.8708 essentially ties ReasonExplicit-SFT's 0.8716, and the paper notes that Direct-SFT can win on emotion when a few classes dominate.","EWRA improves explanation quality over Direct-SFT on most tasks, with higher Jaccard and BERTScore values, suggesting the fine-tuned models produce outputs that are easier for analysts to inspect and trust.","The released datasets give the community a common benchmark for vulnerability/impact/emergency categorization, topic/subtopic/keyword labeling, and emotion analysis of extreme weather news."],"supporting_citations":[{"why":"Supplies the Qwen2.5 teacher model whose one-shot reasoning paths and probability labels form the ExtremeAlign training data.","marker":"[46]"},{"why":"Provides the ClimaMeter event list used to select the 60 extreme weather events covered by ExtremeWeatherNews.","marker":"[45]"},{"why":"Supplies the query-key LoRA fine-tuning approach that adapts attention while freezing value matrices and MLP layers.","marker":"[47]"},{"why":"Defines Spearman rank correlation, the primary metric for measuring alignment between model predictions and human-annotated rankings.","marker":"[48]"},{"why":"Defines BERTScore, the secondary metric used to compare generated explanations with gold-set references.","marker":"[49]"},{"why":"Motivates the reasoning-alignment paradigm by showing chain-of-thought reasoning can be transferred from large to small models.","marker":"[38]"}],"fun_headline_variants":["Implicit-first curriculum sharpens small weather models","Implicit reasoning first lifts small model weather scores","Training order boosts small models' extreme weather reasoning","Implicit-to-explicit training powers weather SLMs","EWRA: implicit-first alignment for weather analytics"],"cache_read_input_tokens":21376,"weakest_assumption_plain":"The whole pipeline assumes that the teacher model's one-shot reasoning paths and probability assignments are accurate enough to serve as training labels—and the gold set is annotated under the same taxonomy and guidelines given to that teacher, so systematic teacher errors could be inherited rather than corrected.","fun_headline_variants_meta":{"raw":{"variants":["Implicit-first curriculum sharpens small weather models","Implicit reasoning first lifts small model weather scores","Training order boosts small models' extreme weather reasoning","Implicit-to-explicit training powers weather SLMs","EWRA: implicit-first alignment for weather analytics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3171,"prompt_tokens":1036,"completion_tokens":2135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":2063}},"tokens_in":652,"tokens_out":2135,"duration_ms":17101,"temperature":1.0,"reasoning_tokens":2063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:03:06.629167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators, using the published taxonomy but without seeing the teacher's outputs, label a fresh sample of ExtremeWeatherNews sentences, then measure agreement between Qwen2.5-32B-Instruct's one-shot probability assignments and those labels using Spearman correlation or calibration metrics. If the teacher's agreement is no better than the students' agreement with the gold set, or if the teacher's failure cases are exactly the cases the EWRA-trained students get wrong, then EWRA is distilling the teacher's bias rather than teaching domain reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ClimaMeter event list used to select the 60 extreme weather events covered by ExtremeWeatherNews."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the query-key LoRA fine-tuning approach that adapts attention while freezing value matrices and MLP layers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Spearman rank correlation, the primary metric for measuring alignment between model predictions and human-annotated rankings."},{"cited_title":"& Freitas, A","cited_arxiv_id":null,"evidence_quote":"Motivates the reasoning-alignment paradigm by showing chain-of-thought reasoning can be transferred from large to small models."}],"review_version":1}