{"id":"01973cef-a975-4ea2-8339-3398a309b1fc","arxiv_id":"2504.16056","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A small language model distilled with a combination of multitask and counterfactual training on critique-revised explanations produced human-rated higher-quality explanations, while multitask training alone gave the best accuracy.","lead":"The authors compared four ways of distilling a large language model into a smaller one, measuring both test accuracy and how humans rated the models' explanations. They found that combining two training methods with a critique-revision data generation step improved the perceived quality of explanations, though it did not improve accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Explainability comparison is confounded: rating only correct answers yields different question subsets per model because accuracies differ, so MT+CF:Revised's advantage may reflect question difficulty rather than explanation quality.","rationale":"The paper's central novel contribution is the explainability claim about MT+CF:Revised, and that claim rests entirely on the human study. The study design filters explanations by model correctness, and because models have different accuracies, the filtered rating sets differ across models. The outcome measures can then reflect item properties rather than explanation quality. This is not a question of statistical test choice; it is a structural confound in the comparison. The performance results are less affected because accuracy is measured on identical test items for all models. The explainability regressions also drop 17 outliers, but that is secondary. If the common-subset reanalysis preserves the effect, the claim survives; if not, the central claim is not established. The authors could also add MT:Revised and CF:Revised to isolate critique-revision, but that is an additional gap. I therefore agree with the reader's weakest assumption and see no reason to move the verdict beyond CONDITIONAL pending the reanalysis.","tokens_in":21282,"tokens_out":4298,"duration_ms":41878,"concrete_test":"Re-analyze the human ratings restricted to a common subset of CQA questions where every student model gives the correct answer, or use random matched question sets per model, and rerun the Dunn tests and the Table III regression for completeness and contrastiveness with question difficulty as a covariate. If MT+CF:Revised no longer significantly exceeds MT:Unrevised and MT+CF:Unrevised on both dimensions, the explainability claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is in the human study (Section IV-B). The authors restrict ratings to 'only correctly answered explanations to focus on explanation quality rather than answer correctness.' But student accuracies differ substantially (Table I: at 220M, MT:Unrevised is +12.66 points over CF:Unrevised; at 770M, MT+CF:Revised is -1.90 vs MT:Unrevised and -2.20 vs MT+CF:Unrevised). Consequently, the set of questions rated for each model is not the same. A lower-accuracy model can be correct only on an easier or systematically different subset, so its explanations may appear more complete and contrastive because the underlying questions are easier or more stereotyped, not because the model explains better. The Kruskal-Wallis, Dunn, and regression analyses in Section V-B do not include question identity or difficulty as a covariate or blocking factor; thus the reported advantage of MT+CF:Revised on completeness and contrastiveness could be an artifact of this selection. A secondary weakness: Dunn's test shows MT+CF:Revised significantly beats all other models on contrastiveness, but on completeness it significantly beats only CF:Unrevised; the claim that it 'significantly outperforms the other student models' on both dimensions is stronger than the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a controlled comparison of knowledge-distillation variants for T5 student models (220M and 770M parameters) trained on CommonsenseQA data generated by LLaMA-2-13B. The four conditions are counterfactual training (CF:Unrevised), multitask training (MT:Unrevised), combined multitask and counterfactual training (MT+CF:Unrevised), and combined training with critique-revision-prompted revised explanations (MT+CF:Revised). Performance is measured by accuracy on the CommonsenseQA test set; explainability is measured through a within-subject human study with 117 participants who rated explanations on five dimensions. The paper reports that multitask training yields the strongest accuracy, while MT+CF:Revised shows the largest perceived completeness, contrastiveness, and overall quality, leading to the conclusion that critique-revision prompting benefits explainability even though it does not improve accuracy.","tokens_in":21521,"tokens_out":9223,"duration_ms":86481,"significance":"This is a useful empirical contribution: it provides a standardized comparison of data-generation and training choices for distillation, and it evaluates explainability with a human-grounded study rather than a proxy metric. The authors' public repository, fixed seeds, attention-checked participant pool, randomization of statement and task order, and use of ordinal-appropriate nonparametric tests in part of the analysis are strengths. If the main claims hold, the finding that critique-revision prompting improves perceived explanation quality without improving accuracy, and that multitask training dominates counterfactual training, is actionable for practitioners building small deployed models. The novelty is incremental rather than conceptual, but the paper fills a real gap in comparative evaluation of distillation methods.","major_comments":[{"comment":"Section IV-B states that the study 'included only correctly answered explanations to focus on explanation quality rather than answer correctness,' but this creates a selection confound because accuracy differs substantially across models (Table I: at 220M, MT:Unrevised is +12.66 over CF:Unrevised; at 770M, MT+CF:Revised is -1.90 versus MT:Unrevised). Each model's rated set is therefore conditioned on that model's correctness, so lower-accuracy models may be rated only on easier or systematically different questions, and Table II's n=1114 (out of a possible 1404 ratings) confirms that explanations were dropped unevenly. The Kruskal-Wallis, Dunn, and regression analyses in Section V-B do not include question identity or difficulty as a covariate or blocking factor. The reported advantage of MT+CF:Revised on completeness and contrastiveness could thus be an artifact of question-subset selection rather than explanation quality. Please re-analyze on the common set of questions answered correctly by all models, or add question fixed effects or difficulty covariates, and report per-model n and set overlap.","section":"Section IV-B, Section V-B"},{"comment":"Section V-A excludes outliers defined as points beyond 1.5 times the IQR, but does not report how many training runs were performed per condition, how many outliers were removed, or the degrees of freedom in the ANOVA and Tukey tests. Since Section IV-A mentions that 'subsequent models with different random seeds were trained,' the number of seeds per model is essential for interpreting Table I and the associated p-values. Please report n per cell, the seed count, the excluded outlier counts, and the results of the pairwise comparisons with and without outlier removal.","section":"Section V-A"},{"comment":"The claim that MT+CF:Revised significantly outperforms the other student models on completeness and contrastiveness is stronger than the reported evidence. The Dunn test supports a significant contrastiveness advantage over all three other models, but for completeness Section V-B states only that MT+CF:Revised 'surpasses CF:Unrevised significantly'; significant pairwise differences against MT:Unrevised and MT+CF:Unrevised are not reported. The abstract and the first contribution bullet should be revised to state the dimension-specific pairwise results, or the full pairwise comparison matrix should be provided.","section":"Section V-B"},{"comment":"The human study is within-subject (each of 117 participants rated 12 explanations, three per student model), but the Kruskal-Wallis and Dunn tests and the OLS regressions in Section V-B appear to treat every rating as independent, without participant random effects or cluster-robust standard errors. If ratings from the same participant are correlated, the reported p-values (e.g., p=0.0004 for contrastiveness) may be anti-conservative. A Friedman test or a mixed-effects model with participant and possibly question random intercepts would be more appropriate; if a repeated-measures method was actually used, please describe it explicitly.","section":"Section V-B"}],"minor_comments":[{"comment":"Section IV-A states 'T5-base with 220M million (220M) parameters'; 'million' is redundant and should be removed.","section":"Section IV-A"},{"comment":"Figure 2 contains the typo 'Few-short Prompting'; this should be 'Few-shot Prompting' in both the figure and its legend table.","section":"Figure 2"},{"comment":"Section III-C, Eq. (6), writes 'Lconterf actual'; this should be 'L_counterfactual'.","section":"Section III-C"},{"comment":"Section V-A contains the typo 'ANOV A approach'; this should be 'ANOVA approach'.","section":"Section V-A"},{"comment":"The comparison with the PaLM 540B teacher from [17] is not a controlled comparison because the teacher model, dataset, and training settings differ; this should be framed as a suggestive cross-paper observation rather than a firm conclusion.","section":"Section V-A"},{"comment":"The sentence 'Initial validation with corrupted samples from the CQA dataset showed that revised explanations add context-relevant information' is vague; please specify the corruption procedure and the validation metric used.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the empirical work is generally solid, but the explainability headline currently rests on a selection confound and on a stronger pairwise claim than the analysis supports. The issues are fixable with additional analyses and reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this paper has a solid performance comparison and a genuinely confounded explainability comparison. The one thing to know before citing it is that the claim that MT+CF:Revised produces better human-rated explanations rests on ratings of only correctly answered questions, and the models differ by up to 12.66 accuracy points, so the rated question sets are not comparable across models.\n\nWhat's actually new: applying critique-revision prompting (the Constitutional AI style) to distillation data generation, plus the direct comparison of multitask, counterfactual, and combined training under a fixed teacher and dataset. The human study is reasonably careful: 117 participants, attention checks, randomized order, ordinal-appropriate tests, and demographic controls. The performance finding—multitask beats counterfactual—is well supported by the Tukey-Kramer comparisons and is a useful practical takeaway. Code and data are released, which counts.\n\nThe main soft spot is the selection confound in Section IV-B. The authors rate only explanations the student got right, ostensibly to focus on explanation quality. But because accuracies differ substantially, a lower-accuracy model is correct on easier or systematically different questions. The Kruskal-Wallis, Dunn, and regression analyses do not include question identity or difficulty as a covariate or blocking factor, so MT+CF:Revised's edge on completeness and contrastiveness could be an artifact of question difficulty, not explanation quality. This is not a minor footnote; it is the load-bearing assumption of the explainability claim. A secondary issue: Dunn's test shows MT+CF:Revised beats all models on contrastiveness but only CF:Unrevised on completeness, so the paper's claim that it \"significantly outperforms the other student models\" on both dimensions is stronger than the evidence. Also, the performance ANOVA excludes 1.5x IQR outliers without reporting how many runs were removed, which matters for reproducibility. The teacher-size conclusion (a 13B teacher comparable to PaLM 540B) is overclaimed since that comparison crosses studies with different settings; better to drop or heavily caveat.\n\nWho this is for: practitioners choosing distillation methods for small QA models, and people doing human-grounded XAI evaluation. The framework is reusable and the performance comparison is worth having. I'd send it to serious review, not desk-reject, but with the expectation of a major revision that addresses the selection confound—either by rating explanations on a matched set of questions all models answer correctly, or by including question difficulty as a covariate. As it stands, the explainability headline is plausible but not proven.","headline":"A workmanlike distillation comparison with a solid performance finding and a genuinely confounded explainability result that needs a matched-question design before it can be the headline.","tokens_in":22023,"tokens_out":2017,"would_cite":true,"duration_ms":20230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distillation can make small language models explain their answers better, not just answer correctly.","keywords":["knowledge distillation","language models","explainability","critique-revision prompting","multitask training","counterfactual training","human-grounded evaluation","CommonsenseQA"],"falsifier":"A reader could rerun the human study but split each model's rated explanations by question difficulty (for instance, by the fraction of models that answered each question correctly) and check whether the MT+CF:Revised advantage on completeness and contrastiveness persists within difficulty-stratified subsets. If the advantage disappears once question difficulty is controlled, the paper's central explainability claim would collapse.","tokens_in":21070,"feed_emoji":"","tokens_out":1398,"duration_ms":14491,"temperature":0.7,"pith_summary":"This paper asks whether the common practice of shrinking large language models into smaller student models can preserve not only accuracy but also the ability to give clear, human-understandable explanations. The authors add two new tools to the distillation toolbox: critique-revision prompting to polish teacher-generated explanations, and a training loss that combines multitask and counterfactual objectives. They compare four student models on the CommonsenseQA benchmark and in a human study with 117 participants. The central result is that the model trained with the combined method on revised explanations produces explanations that humans rate as more complete and more contrastive, even though it does not beat simpler training on raw accuracy.","feed_headline":"Better explanations, not just answers, from distilled language models","feed_subtitle":"A human study finds that adding self-critique data and combined training makes small models' explanations more complete and contrastive.","key_machinery":"The argument rests on two mechanisms: critique-revision prompting and a synthesized training objective. Critique-revision prompting is a three-step data-generation loop in which the teacher model first writes an explanation, then critiques its own explanation, then rewrites it in light of the critique. The combined training method is the unweighted sum of a multitask loss (answer the multiple-choice question and explain the answer) and a counterfactual loss (answer correctly from the question alone, and answer incorrectly when given a question plus an intentionally wrong explanation). The counterfactual component teaches the student to be sensitive to misleading reasoning, while the multitask component teaches it to produce faithful explanations. Together with revised explanations as training data, these mechanisms produce the student that humans rate as best at completeness and contrastiveness.","core_discovery":"The paper's central claim is that the quality of the explanations a distilled student model gives to humans can be improved substantially by improving the training data through critique-revision prompting, and that this improvement shows up specifically in two explainability dimensions: completeness and contrastiveness. On the performance side, multitask training alone yields the strongest student in terms of accuracy, while counterfactual training alone performs worst. The combined multitask-plus-counterfactual student trained on revised explanations (MT+CF:Revised) significantly outperforms the other students on those two explanation-quality dimensions in the human-grounded study, even though its accuracy is not top. The authors therefore argue that the choice of distillation method involves a trade-off between raw task performance and the perceived quality of explanations, and that for applications where explainability matters, the combined method on revised data is the better choice.","pith_inferences":["The critique-revision loop could be generalized beyond data generation: the same teacher could critique and revise the student's own explanations during inference, potentially improving explanation quality without retraining.","Because the combined method adds a counterfactual loss, the observed improvement in contrastiveness may partly reflect the student learning to reason about why other answer choices are wrong, a skill that could transfer to other multiple-choice reasoning tasks.","A natural testable extension is to vary the temperature or number of critique-revision iterations and measure whether explanation quality continues to improve or plateaus, and whether a larger student can exploit longer revised explanations better than a smaller one.","The limited effect sizes (VDA around 0.4) suggest that the practical benefit of the combined method may depend on the stakes of the application; for high-stakes uses, even small gains in completeness and contrastiveness could matter."],"forward_implications":["If the central result holds, practitioners who care about explainability can keep a small student model and a modest teacher (here a 13B model) rather than needing a much larger teacher to get high-quality human-facing explanations.","The performance-versus-explainability trade-off suggests that evaluation of distillation methods should include human-grounded explanation ratings, not just accuracy benchmarks.","The finding that counterfactual training alone hurts accuracy and does not improve explainability calls into question the automatic adoption of counterfactual objectives for reasoning tasks.","The authors' observation that a 13B teacher can produce students comparable to those from a 540B teacher challenges the assumption that bigger teachers are always better for distillation.","The framework of standardizing teacher and training settings makes different distillation methods directly comparable, which is a precondition for building a reliable body of knowledge about what works."],"supporting_citations":[{"why":"Supplies the multitask training method that the paper combines with counterfactual training and uses as a performance baseline.","marker":"[17]"},{"why":"Supplies the counterfactual training method and the prior claim that it improves explanation faithfulness, which the paper tests with human evaluation.","marker":"[34]"},{"why":"Supplies the critique-revision prompting idea from Constitutional AI, adapted here for generating revised explanations.","marker":"[35]"},{"why":"Supplies the CommonsenseQA dataset that the students are trained and evaluated on.","marker":"[62]"},{"why":"Supplies the LLaMA-2-13B teacher model used to generate explanations and critiques.","marker":"[63]"},{"why":"Supplies the T5 student architectures (base and large) that are fine-tuned in the experiments.","marker":"[65]"},{"why":"Supplies the human-grounded evaluation methodology for explanation quality that the study adapts.","marker":"[31]"},{"why":"Supplies the human-grounded evaluation methodology for explanation quality that the study adapts.","marker":"[32]"}],"fun_headline_variants":["Distilled models' explanations improve with critique-revision training","Small language models: combined distillation yields clearer explanations","Distillation trade-off: accuracy vs. explanation quality in small LLMs","Human study: critique-revision data improves explanation completeness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The explainability comparison assumes that judging only explanations for questions the student answered correctly does not distort the comparison, even though the student models have different accuracy levels and therefore effectively evaluate different, potentially easier or harder, subsets of questions.","fun_headline_variants_meta":{"raw":{"variants":["Distilled models' explanations improve with critique-revision training","Small language models: combined distillation yields clearer explanations","Distillation trade-off: accuracy vs. explanation quality in small LLMs","Human study: critique-revision data improves explanation completeness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1590,"prompt_tokens":922,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":601}},"tokens_in":538,"tokens_out":668,"duration_ms":6691,"temperature":1.0,"reasoning_tokens":601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:22.794699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could rerun the human study but split each model's rated explanations by question difficulty (for instance, by the fraction of models that answered each question correctly) and check whether the MT+CF:Revised advantage on completeness and contrastiveness persists within difficulty-stratified subsets. If the advantage disappears once question difficulty is controlled, the paper's central explainability claim would collapse.","supporting_citations":[{"cited_title":"CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge,","cited_arxiv_id":null,"evidence_quote":"Supplies the T5 student architectures (base and large) that are fine-tuned in the experiments."},{"cited_title":"SCOTT: Self-consistent chain-of-thought distillation,","cited_arxiv_id":null,"evidence_quote":"Supplies the counterfactual training method and the prior claim that it improves explanation faithfulness, which the paper tests with human evaluation."},{"cited_title":"The Impact of Imperfect XAI on Human-AI Decision-Making","cited_arxiv_id":"2307.13566","evidence_quote":"Supplies the critique-revision prompting idea from Constitutional AI, adapted here for generating revised explanations."},{"cited_title":"Explainability in AI Based Applications: A Framework for Comparing Different Techniques","cited_arxiv_id":"2410.20873","evidence_quote":"Supplies the human-grounded evaluation methodology for explanation quality that the study adapts."},{"cited_title":"Multimodal explanations: Justifying decisions and pointing to the evidence,","cited_arxiv_id":null,"evidence_quote":"Supplies the human-grounded evaluation methodology for explanation quality that the study adapts."}],"review_version":1}