{"id":"a1410700-f028-4b7a-9825-16674ec2b468","arxiv_id":"2412.00549","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Two-stage sequential fine-tuning of Qwen2.5 7B improved financial misinformation detection (F1 0.8283) and explanation quality (ROUGE-1 0.7253) over single-stage joint training.","lead":"The authors describe SeQwen, a Qwen2.5 7B model fine-tuned in two stages for the COLING 2025 Financial Misinformation Detection challenge: first for claim classification, then for joint classification with explanation generation. It reports higher F1 and ROUGE scores than one-stage fine-tuning on the FIN-FACT dataset, though the gains are not tested for statistical significance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sequential-learning improvement is supported only by single unreplicated runs, and the test-set comparison omits the matched 8-epoch single-phase baseline; the unspecified justification-augmentation could also confound the comparison.","rationale":"The reader's weakest assumption already identifies the lack of error bars, seeds, and precise preprocessing as the main vulnerability, and I agree that this is where the argument is least secure. My pass adds one concrete detail: the test-set comparison in Table 4 omits the matched 8-epoch single-phase baseline that appears in the dev-set Table 3, so the strongest external check is against a baseline with fewer total epochs. I also quantify how close the Micro F1 differences are to the standard error of a single run on 453 samples. This does not make the paper's claim implausible; the dev and test directions are consistent and the sequential design is a reasonable curriculum-style hypothesis. However, the evidence as reported is not sufficient to separate the effect of the sequential schedule from run-to-run variance or from the unspecified augmentation. The appropriate verdict remains CONDITIONAL, as the reader set: accept only if the authors provide multi-seed variance estimates, clarify or ablate the augmentation, and report the matched-epoch baseline on the test set. No verdict change is needed.","tokens_in":5593,"tokens_out":5880,"duration_ms":57439,"concrete_test":"Run SeQwen and the 8-epoch single-phase joint baseline with the same optimizer, prompt formats, and the same 1500/453 split for at least 5 random seeds, and report mean and standard deviation of Micro F1 and ROUGE on the dev set, with a paired bootstrap or permutation test over claims. Also run a third condition, SeQwen with the 'append claims from justification' preprocessing disabled, to quantify how much of the gain comes from preprocessing rather than from the sequential schedule. If the 8-epoch baseline overlaps with SeQwen within error, or if the no-augmentation version loses most of the gain, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that 3 epochs of classification-only LoRA training followed by 5 epochs of joint training (SeQwen) beats single-phase joint training, rests on Tables 3 and 4. The dev table includes a matched 8-epoch single-phase run, but the test table compares SeQwen only to the 5-epoch single-phase model, so the headline 'test set' validation is not against a baseline with the same total number of epochs. More importantly, every configuration appears to be a single run: no seeds, no error bars, and no significance test. For a 453-sample dev set, the standard error of a Micro F1 near 0.83 is roughly 0.018, so the reported dev Micro F1 gap of 0.0132 between SeQwen and the 8-epoch baseline is within one standard error; on the 1304-sample test set the Micro F1 gap of 0.0118 is about one standard error. The ROUGE gains are larger, but ROUGE scores on generated explanations can have high variance and were not assessed for statistical significance. Finally, the preprocessing sentence at the end of Section 3, 'we appended some claims from the justification field,' is underspecified. If this augmentation was applied only to the sequential condition, or only to the joint fine-tuning data, the entire comparison would be confounded. These issues do not prove the claim false, but they mean the evidence is not yet sufficient to establish that sequential learning, rather than run-to-run noise or preprocessing, caused the improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes SeQwen, the authors' submission to the COLING 2025 Financial Misinformation Detection (FMD) challenge. The system uses Qwen2.5 7B fine-tuned with LoRA in a sequential learning scheme: first 3 epochs of classification-only fine-tuning, then 5 epochs of joint classification and explanation generation. The authors evaluate several open-source LLMs on a development split of FIN-FACT, select Qwen2.5 7B as the best classifier, and then compare SeQwen against single-phase joint fine-tuning baselines on both the development and test splits. They report that SeQwen outperforms single-phase training on all reported metrics, e.g., Micro F1 of 0.8366 and ROUGE-1 of 0.7170 on the dev set, and Micro F1 of 0.8283 and ROUGE-1 of 0.7253 on the test set. The central claim is that the two-stage training schedule improves both classification and explanation quality over one-stage joint fine-tuning.","tokens_in":5938,"tokens_out":1903,"duration_ms":19099,"significance":"If the claimed improvement is real, the result is practically relevant for multi-task fine-tuning of LLMs in financial claim verification, showing that a simple staged curriculum can mitigate the classification/explanation tradeoff observed in single-phase joint training. The paper's strengths include an empirical comparison on a shared challenge benchmark, a clear description of the prompting and LoRA setup, and an honest limitations section that acknowledges the absence of human evaluation and the constraints of 4-bit precision. However, the significance is currently limited by the lack of statistical rigor: all results come from single unreplicated runs, the development set is small (453 samples), and the test-set comparison omits a matched 8-epoch single-phase baseline. The underspecified preprocessing augmentation also clouds the attribution of the gains to sequential learning. These issues are fixable but must be addressed before the central claim can be considered established.","major_comments":[{"comment":"The central comparison rests on single runs with no reported seeds, error bars, or significance tests. On the 453-sample dev set, the Micro F1 gap between SeQwen (0.8366) and the 8-epoch single-phase baseline (0.8234) is 0.0132, which is within one standard error of a proportion near 0.83; on the 1304-sample test set, the gap of 0.0118 against the 5-epoch baseline is similarly about one standard error. The ROUGE gains are larger, but ROUGE on generated text is known to be high-variance and was not tested for statistical significance. Without multiple seeds or a significance test, the observed improvements could be run-to-run noise.","section":"Section 4, Tables 3 and 4"},{"comment":"The test-set table compares SeQwen only to the 5-epoch single-phase joint model, omitting the 8-epoch single-phase baseline that appears in Table 3. Since SeQwen uses 8 total epochs of training, the test-set result does not control for total training budget. The dev-set comparison in Table 3 does include the 8-epoch baseline and shows a smaller gap, but the test-set validation of the headline claim would be strengthened by reporting the 8-epoch single-phase test-set numbers.","section":"Table 4"},{"comment":"The preprocessing step described as 'we appended some claims from the justification field, during the fine-tuning phase' is critically underspecified. It is not stated which conditions received this augmentation, how many claims were appended, how they were selected, or whether the augmentation was applied identically to the classification-only, joint, and sequential training runs. If the augmentation is not applied uniformly across all compared conditions, the entire comparison in Tables 3 and 4 is confounded, and the reported gains cannot be attributed to sequential learning.","section":"Section 3, last paragraph of Methodology"},{"comment":"The paper reports 'Overall Score' but never defines how it is computed from Micro F1 and the ROUGE scores. Since the overall score is used in the narrative (e.g., the 7.1% improvement on the test set), the formula must be given, otherwise the reader cannot verify whether the relative improvements are driven primarily by the ROUGE components, which appear to have larger relative gains than Micro F1.","section":"Section 4, Tables 3 and 4"}],"minor_comments":[{"comment":"The term 'Minsinformation' in the Introduction is a typo for 'Misinformation'; please correct it.","section":"Abstract and Introduction"},{"comment":"The dataset section states the test set has 1304 samples, but Table 1 only reports the train and dev distributions; it would be helpful to also report the test class distribution for completeness, as the confusion matrices in the appendix are only for the dev set.","section":"Section 2"},{"comment":"The two prompt templates in Figure 3 are visually dense and the difference between them is easy to miss; a short caption or annotation highlighting the structural difference (label-only versus label-and-explanation response format) would improve clarity.","section":"Figure 3"},{"comment":"The table lists 'Qwen2.5 32B' but the methodology section enumerates only five models (Qwen2.5, Llama3 8B, Mistral 7B, Phi3 medium 4K, and Gemma-2 9B); the 32B variant should be mentioned in the model list for consistency.","section":"Table 2"},{"comment":"The limitations section is candid about computational constraints and the absence of human evaluation, but it does not mention the lack of multiple runs or significance testing; adding this would be an honest reflection of the evidence level.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a challenge system-description paper, and the bar for acceptance should be calibrated accordingly: the community often values a reproducible recipe even with modest statistical rigor. However, the current manuscript's central claim is under-supported in three specific ways: no repeated runs, a missing test-set control for total epochs, and an underspecified augmentation step. These are not fatal to the idea, and the authors can plausibly address them by adding seed variation results, reporting the 8-epoch baseline on the test set, and describing the preprocessing precisely. I therefore recommend major revision rather than rejection. I would also gently note that the phrase 'some claims' in the preprocessing description should be quantified before publication; as written, it invites skepticism about the fairness of the comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The one thing to know: this is a shared-task system description with a single load-bearing empirical claim—two stages of fine-tuning (3 epochs classification-only, then 5 epochs joint classification+explanation) beat single-phase joint fine-tuning for Qwen2.5 7B on FIN-FACT. The dev and test results point the same way, and the ROUGE gains are large (14-20% relative). If you work on financial misinformation detection, this is a cheap trick worth trying.\n\nWhat's genuinely new: the specific comparison, sequential vs single-phase joint training for this benchmark, is not in FMD-LLaMA, which they cite. The model selection across six LLMs is honest work, and the dev table includes a matched 8-epoch single-phase baseline. That's the right control. The limitations section is candid about 4-bit precision, ROUGE's limits, and potential overfitting.\n\nNow the soft spots, in order of weight. The whole case rests on single unreplicated runs: no seeds, no error bars, no significance test. On the 453-sample dev set, the Micro F1 gap between SeQwen and the 8-epoch baseline is 0.0132, roughly one standard error. The test gap of 0.0118 is about the same relative to its sample size. So the F1 headline isn't established. The ROUGE gains are more substantial and internally consistent, but ROUGE on generated text is high-variance and they don't test it. The test table omits the 8-epoch baseline that appears in the dev table; the dev comparison is the fair one, so this isn't fatal, but it makes the test narrative incomplete. The preprocessing sentence at the end of Section 3—'we appended some claims from the justification field'—is underspecified. If that augmentation wasn't identical across conditions, the comparison is confounded. And there's no released code to check the pipeline.\n\nNone of this proves the claim false. The pattern is plausible and matches curriculum-learning results elsewhere. But as it stands, this is suggestive evidence, not a demonstrated effect. The authors seem aware of some of these limits themselves.\n\nWho gets value: practitioners building financial fact-checking or explanation systems, and shared-task readers benchmarking Qwen2.5. It's not a methods paper. I'd send it to a serious referee, but with the expectation that the authors add variance estimates, specify the augmentation, and include the 8-epoch test baseline. Without those, I'd treat the F1 gain as noise and the ROUGE gain as promising but unproven.","headline":"Plausible but under-powered evidence for sequential fine-tuning on FIN-FACT; worth a serious referee if the authors add variance estimates and specify the preprocessing.","tokens_in":6464,"tokens_out":2618,"would_cite":false,"duration_ms":25700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sequential fine-tuning lifts financial misinformation detection on the FIN-FACT benchmark.","keywords":["financial misinformation detection","sequential fine-tuning","explanation generation","FIN-FACT","Qwen2.5","LoRA","claim verification","ROUGE"],"falsifier":"Train the same Qwen2.5 7B model under three conditions—sequential fine-tuning, 8-epoch single-phase joint training, and 5-epoch joint training—with the justification-field preprocessing removed, repeated across five random seeds. If SeQwen's test-set advantages (micro F1 0.8283 vs. 0.8165, ROUGE-1 0.7253 vs. 0.6337) shrink below the run-to-run noise or reverse in this setup, the claimed benefit of sequential learning would not be sustained.","tokens_in":5422,"feed_emoji":"📊","tokens_out":4018,"duration_ms":148238,"temperature":0.7,"pith_summary":"This paper describes a system for detecting financial misinformation and generating explanatory justifications. Its central claim is that training a large language model in two stages—first to classify claims alone, then to classify and explain simultaneously—outperforms standard single-stage joint training. On the FIN-FACT benchmark, the two-stage model, named SeQwen, achieved a micro F1 of 0.8366 on the development set and 0.8283 on the test set, with explanation quality (ROUGE-1) of 0.7170 and 0.7253 respectively. The authors argue that staged training lets the model specialize in verification before learning to articulate reasons, benefiting both tasks. If correct, this offers a simple training-schedule change that improves interpretable financial misinformation detection without larger models.","feed_headline":"Sequential fine-tuning lifts financial misinformation detection","feed_subtitle":"On the FIN-FACT test set, staged training raised micro F1 to 0.828 and ROUGE-1 to 0.725.","key_machinery":"The mechanism is a two-stage fine-tuning schedule on a generative LLM, using LoRA adapters. Stage one trains the model to output only a label given a claim and justification; stage two continues training to output both a label and an explanation in a fixed format. The same prompt template, taken from FMD-LLaMA, is used with the explanation component added in the second stage. The paper hypothesizes that the first stage stabilizes the classification decision, and the second stage learns to articulate that decision, producing better joint performance than training both objectives from the start.","core_discovery":"The paper's core discovery claim is that sequential fine-tuning of Qwen2.5 7B, with 3 epochs of classification-only training followed by 5 epochs of joint classification-and-explanation training, yields higher micro F1 and higher ROUGE scores than single-phase joint fine-tuning for the same total of 8 epochs. On the test set, SeQwen raised micro F1 from 0.8165 to 0.8283 and ROUGE-1 from 0.6337 to 0.7253 over the 5-epoch single-phase model. The authors present this as evidence that staged, task-specific training can mitigate the tradeoff between predictive accuracy and explanation generation in domain-specific claim verification.","pith_inferences":["The ROUGE gains may reflect the model learning the exact output format (Prediction: ... Explanation: ...) more than better reasoning; a human evaluation or semantic-similarity metric would be needed to separate format alignment from explanation quality.","The sequential schedule resembles a curriculum where the easier task (3-way classification) precedes the harder generative task; this suggests testing gradual mixing of the two losses rather than a hard switch.","Because the dev set has only 453 samples and the reported gains are around 0.01 F1, the model ranking and the SeQwen advantage should be checked with multiple seeds and bootstrapped confidence intervals before relying on the exact numbers.","If the preprocessing step (appending claims from the justification field) is what actually drives the improvement, then sequential learning may be confounded with data augmentation; a controlled ablation would separate the two."],"forward_implications":["SeQwen's schedule is directly transferable to other verification-plus-explanation tasks, such as fact-checking with citations, without changing the model architecture.","The reported results suggest that total training budget can be split to favor explanation quality (5 of 8 epochs) after a shorter classification-only warm-up.","Small 7B models fine-tuned with LoRA at 4-bit precision can come close to or exceed larger models on financial misinformation benchmarks, lowering hardware requirements.","The gap between joint-only and sequential training points to an objective-ordering effect: which task is learned first can matter even when the final objective is the same."],"supporting_citations":[{"why":"Supplies the FIN-FACT dataset and the task formulation of financial claim verification with explanation generation.","marker":"(Rangapur et al., 2024)"},{"why":"Provides the prompt design and the joint classification-and-explanation generation formulation used in all fine-tuning stages.","marker":"(Liu et al., 2024)"},{"why":"Supplies the LoRA fine-tuning method used to adapt the base models with low-rank adapters.","marker":"(Hu et al., 2021)"},{"why":"Defines the ROUGE metrics used to evaluate explanation generation quality.","marker":"(Lin, 2004)"},{"why":"Introduces the Qwen2.5 family of foundation models from which the 7B base model is taken.","marker":"(Qwen Team, 2024)"}],"fun_headline_variants":["Staged training lifts financial misinformation detection","Sequential fine-tuning boosts F1 and explanation scores","Two-phase LLM training improves finance claim checks","SeQwen: sequential learning beats joint fine-tuning","Staged fine-tuning raises financial F1 to 0.828"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the 453-sample development set can reliably rank models when differences are as small as 0.01 micro F1, and that the preprocessing step of appending some claims from the justification field affects all compared training conditions equally.","fun_headline_variants_meta":{"raw":{"variants":["Staged training lifts financial misinformation detection","Sequential fine-tuning boosts F1 and explanation scores","Two-phase LLM training improves finance claim checks","SeQwen: sequential learning beats joint fine-tuning","Staged fine-tuning raises financial F1 to 0.828"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1395,"prompt_tokens":827,"completion_tokens":568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":443,"tokens_out":568,"duration_ms":5806,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:13:40.717820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same Qwen2.5 7B model under three conditions—sequential fine-tuning, 8-epoch single-phase joint training, and 5-epoch joint training—with the justification-field preprocessing removed, repeated across five random seeds. If SeQwen's test-set advantages (micro F1 0.8283 vs. 0.8165, ROUGE-1 0.7253 vs. 0.6337) shrink below the run-to-run noise or reverse in this setup, the claimed benefit of sequential learning would not be sustained.","supporting_citations":[{"cited_title":"FMDLlama: Financial Misinformation Detection based on Large Language Models","cited_arxiv_id":"2409.16452","evidence_quote":"Provides the prompt design and the joint classification-and-explanation generation formulation used in all fine-tuning stages."}],"review_version":1}