{"id":"ab251e85-17bd-4c6b-a6ef-909abb62cfea","arxiv_id":"2611.17072","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Label-only LoRA fine-tuning beats teacher reasoning-trace supervision on in-domain calibration for financial headline classification, with traces providing only a small MMLU-Pro lift.","lead":"Researchers fine-tuned an 8-billion-parameter language model on financial headline classification using three LoRA setups: labels only, labels plus reasoning traces from a larger model, and two separate adapters for traces and labels. The label-only setup gave the best in-domain calibration and accuracy gains while general benchmarks stayed near frozen levels; adding reasoning traces did not help and worsened calibration.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Selection criterion in §3.6 conflates retention with the admission rule: a run is kept only if it beats the frozen model on every task, so the claim that response-only LoRA 'leaves general capabilities near those of the frozen model' is partly an artifact of discarding runs that degraded on any…","rationale":"The reader's weakest_assumption focused on teacher-generated labels as ground truth, which is a valid concern but is disclosed and partially mitigated by the paper's framing (the study measures agreement with the teacher, not with human judges). The selection-criterion issue is more directly load-bearing for the central claim about capability retention. The paper states the rule transparently (§3.6) and uses it consistently, but the headline claim ('leaves general capabilities near those of the frozen model') is presented as a finding about the response-only regime, not about the selected subset. The paper does not report how many runs were discarded, nor the retention scores of the discarded runs. This is a concrete, addressable concern that does not require re-running the full experiment—only a summary statistic that the authors likely already have. The verdict remains CONDITIONAL because the concern is addressable and does not invalidate the in-domain comparisons, which are the paper's strongest contribution. The agreement_with_reader is 'partial' because the reader identified a different weakest assumption (teacher labels) that is also valid but less central to the retention claim.","tokens_in":28311,"tokens_out":713,"duration_ms":54187,"concrete_test":"Re-run the analysis without the selection criterion: report the mean and standard deviation of each retention metric (MMLU-Pro, GPQA, IFEval) across all 432 hyperparameter configurations from Stage 1, not just the 5 that beat the frozen model on every task. If the average retention across all runs is significantly below the frozen baseline, the claim that response-only LoRA 'leaves general capabilities near those of the frozen model' would need qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The staged protocol (§3.6) applies a selection criterion that a run is kept only if it outperforms the frozen baseline on all tasks, both in-domain and retention. This means that any run that improved in-domain metrics but caused a small drop on, say, GPQA (which has high variance, std ≈ 4.3% per Table 1) would be excluded from further analysis. The headline claim that response-only LoRA 'leaves general capabilities near those of the frozen model' is therefore partly a consequence of the admission rule rather than an empirical finding about the typical behavior of LoRA fine-tuning. The paper acknowledges this rule and its motivation (practical usefulness), but does not report how many runs were discarded at each stage, nor what the distribution of retention scores looked like before filtering. Without that information, the reader cannot assess whether the retention claim is robust or fragile. The concern is not that the rule is invalid—it is a legitimate design choice—but that the central claim about capability retention is stated as a general property of response-only LoRA when it is actually a property of the subset of configurations that passed a multi-task threshold. This is a load-bearing concern because the paper's main conclusion (response-only LoRA preserves general capabilities) depends on it; if the discarded runs showed systematic degradation, the conclusion would be weaker.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents a systematic study of reasoning supervision in LoRA fine-tuning for a financial classification task, comparing response-only LoRA with two methods of incorporating teacher reasoning traces (single-adapter blended loss and dual-adapter composite training) and also studying the effect of summarizing traces before training. The main findings are that response-only LoRA improves in-domain performance while preserving general capabilities, and that adding teacher reasoning traces does not extend these gains and leaves calibration behind response-only runs.","tokens_in":28700,"tokens_out":1359,"duration_ms":81257,"significance":"If the results hold, the paper provides practical guidance for practitioners fine-tuning LLMs on domain-specific tasks: response-only LoRA is sufficient and adding reasoning traces may not help and can hurt calibration. The staged experimental design with matched hyperparameters is a strength, as is the evaluation on both in-domain and retention benchmarks. However, the significance is limited by the narrow scope (one dataset, one base model, one teacher model) and the reliance on teacher-generated labels without human validation.","major_comments":[{"comment":"The selection criterion (a run is kept only if it outperforms the frozen baseline on all tasks) conflates the retention claim with the admission rule. The headline claim that response-only LoRA 'leaves general capabilities near those of the frozen model' is partly an artifact of discarding runs that degraded on any task. The paper does not report how many runs were discarded at each stage, nor the distribution of retention scores before filtering. Without this information, the reader cannot assess whether the retention claim is robust or fragile. This is a load-bearing concern because the paper's main conclusion depends on it.","section":"§3.6, §4.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Teacher-generated labels are treated as ground truth for the classification task. Section 3.1 states the 200K examples use labels and reasoning traces generated by Qwen3-32B, and all in-domain metrics measure agreement with those labels on synthetic FinHeadlineMix headlines; no human-annotation validation is provided. If the teacher's labels are noisy or systematically different from human event types, the large calibration improvements (NLL from 5.70 to 0.48) mean the model becomes well-calibrated to the teacher's label distribution rather than to true event types, and the ranking of regimes could change. This premise is disclosed as a data-generation choice but is never checked, and the headline claims are only as strong as it.","section":"§3.1, §4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that only two runs in Stage 2 outperform the frozen model on all tasks (bold squares in Figure 3). However, Table 5 shows that many runs have retention metrics that are within one standard deviation of the frozen baseline, yet they are excluded because they do not strictly outperform on every task. The selection criterion is applied before any claim is made, but the central claim about capability retention is stated as a general property of response-only LoRA when it is actually a property of the subset of configurations that passed a multi-task threshold. The paper should report the number of runs discarded and the distribution of retention scores before filtering.","section":"§4.4, Table 5","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The region-balanced loss uses per-token cross-entropy averaged within each region. The paper should clarify whether the loss is averaged over tokens in each region separately (so each region contributes equally regardless of length) or whether it is a weighted average. The text says 'each bar denotes the mean cross-entropy over the tokens in that region', which suggests the former, but this could be stated more explicitly.","section":"§3.3, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 2 shows the distribution of top runs over hyperparameters, but the sample size is only 5 runs. The conclusions about which hyperparameters matter most (e.g., effective LoRA learning rate) are based on very few data points and should be caveated accordingly.","section":"§4.3, Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The realized compression ratios (e.g., 0.426 for target 0.50) deviate substantially from the targets. The paper notes this but does not discuss whether the deviation is systematic (e.g., the teacher model cannot compress as much as requested) or whether it affects the interpretation of the results.","section":"§4.6, Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 4 shows that the rank split does not separate the runs, but the y-axis ranges are very narrow (e.g., MMLU-Pro from 0.622 to 0.638). The paper should note that the differences are small and may not be practically significant.","section":"§4.5, Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The frozen model's GPQA score has a standard deviation of 4.3% across three runs. This high variance should be taken into account when interpreting retention results; many of the reported differences may be within noise.","section":"§4.2, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper notes that λ=1.0 runs perform far worse than response-only training because the training inputs differ (the reasoning text is present). This is an important caveat that should be highlighted earlier, as it affects the interpretation of the blended loss experiments.","section":"§4.4, Table 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The conclusion states that 'Response-only LoRA is the regime that improves in-domain performance while leaving general capabilities near the frozen model.' This claim should be qualified by noting that it applies only to the subset of configurations that passed the selection criterion.","section":"§5, Conclusion","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper is a clean ablation study comparing response-only LoRA with two ways of incorporating teacher reasoning traces (single-adapter blended loss and dual-adapter rank splits) for a financial classification task. The headline finding—that response-only LoRA improves in-domain calibration and argmax accuracy while leaving general capabilities near frozen levels, and that adding traces does not extend these gains—is supported by the tables within the stated scope. The most novel pieces are the region-balanced loss (Eq. 1), the dual-adapter rank splits under a fixed budget, and the trace-summarization curve showing calibration moving toward the response-only regime as traces shorten. These are small but honest extensions of the authors' prior work [10]. The paper is transparent about its staged protocol and reports means and stds over three seeds.\n\nThe main soft spot is the selection criterion in §3.6: a run is kept only if it beats the frozen model on every task. This means the central claim that response-only LoRA 'leaves general capabilities near those of the frozen model' is partly a consequence of the admission rule rather than an empirical finding about typical behavior. The paper acknowledges the rule and its motivation, but does not report how many runs were discarded at each stage or what the distribution of retention scores looked like before filtering. Without that, the reader cannot assess robustness. The stress-test note is correct that this is a load-bearing concern, though the paper's own framing is careful enough that a charitable reader would not call it deceptive—just incomplete.\n\nA second concern: ground truth is teacher-generated with no human validation. The calibration improvements (NLL from 5.70 to 0.48) mean the model becomes well-calibrated to the teacher's label distribution, not necessarily to true event types. The paper discloses this as a data-generation choice but never checks it. This limits the external validity of the headline claims.\n\nMinor issues: no significance tests, no code or trace data released, and the most novel pieces are small extensions of [10].\n\nWho this is for: practitioners doing LoRA fine-tuning on classification tasks who want a clear default (label-only) and a matched-budget negative result on reasoning traces. The paper deserves a serious referee because the measurements are reproducible in principle and the question is practically important. I would accept peer review but recommend asking for a selection-robustness analysis and a human-label sanity check.\n\nVerdict: CONDITIONAL accept with moderate revisions.","headline":"A careful, well-scoped ablation study showing that for this financial classification task, response-only LoRA beats trace-supervised variants on in-domain calibration, and that the retention claims are partly an artifact of the selection rule.","tokens_in":29260,"tokens_out":607,"duration_ms":41001},"pith_extraction":null,"created_at":"2026-11-13T10:45:17.424853+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}