{"id":"9675bfdf-1a62-4261-abd4-61a0031419f3","arxiv_id":"2505.15480","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KaFT down-weights training examples that conflict with a model's own knowledge, improving medical QA accuracy over vanilla SFT across four open LLMs.","lead":"This paper proposes KaFT, a fine-tuning method that measures where a language model's internal knowledge conflicts with medical QA training data and then reweights each sample so high-conflict examples are learned less aggressively. Across four open LLMs, KaFT outperforms vanilla supervised fine-tuning on medical benchmarks and is claimed to reduce hallucination.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) conflict score may measure difficulty rather than knowledge conflict, and its stability across random permutations and sampling is untested; this is the load-bearing assumption behind KaFT.","rationale":"The reader's weakest_assumption identifies exactly the hinge of the paper: Eq. (1) must be a valid, stable measure of knowledge conflict for the quartile rewards in Eq. (2) to implement 'knowledge-aware' fine-tuning. I agree with that identification. The paper has genuine independent support: Table 2 reports KaFT above vanilla SFT on all four LLMs, Table 4 shows the detected wrong subset is sharply harmful when trained alone, and the OOD and hallucination analyses are consistent with the authors' story. These are real empirical observations and I do not dispute them. But none of them rules out the difficulty confound, and the absence of error bars plus the test-set-tuned hyperparameters (Figure 4) mean the 'consistent and significant' claim is not yet decisively supported. The row-label swap in Table 10 is a concrete reporting error that weakens the Section 2.3 motivation, though after relabeling the direction of the 25%-wrong result survives. Because the central claim is plausible and the proposed reweighting is simple to test, the CONDITIONAL verdict remains appropriate. I would keep the reader's verdict unchanged, with the concrete test above as the natural condition for upgrading to ACCEPT.","tokens_in":19791,"tokens_out":8004,"duration_ms":74024,"concrete_test":"Run a test-retest experiment on MedQA: repeat the Eq. (1) conflict detection procedure R=5 times for the same frozen base model, using fresh random option permutations and sampling seeds each time. Measure how often each training sample changes quartile (right/might-right/might-wrong/wrong); if more than 10-20% of samples move across the might-right/might-wrong boundary, the reward assignment in Eq. (2) is not stable, and the Table 2 gains should be treated as selection-dependent. In the same run, for samples consistently labeled 'wrong,' also record the model's answer distribution over the 100 samples: if the entropy is high and no specific wrong option dominates, low Score_i reflects uncertainty/difficulty rather than a confident internal answer conflicting with the reference. This would settle whether the conflict-score assumption actually lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"KaFT's central mechanism is Score_i in Eq. (1): the fraction of correct answers by the frozen base model across No x Nr = 100 permuted-option samples at temperature 0.7. This score is binned into quartiles in Section 2.3 and converted into rewards (alpha=0.1, beta=0.5, 1, 1) in Eq. (2). The paper interprets low Score_i as knowledge conflict. However, Score_i only measures whether the base model chooses the correct option; it cannot distinguish 'the model holds a specific competing answer' (true conflict) from 'the model does not know the answer' or 'the question is objectively hard.' Under the definitions cited in the paper (Ren et al., 2024; Xu et al., 2024), an unknown answer is not a conflict. If low Score_i mostly marks hard or unfamiliar examples, KaFT's gains would be explained by down-weighting difficult samples, not by conflict-aware reweighting, and the 'knowledge-aware' mechanism would be unsupported. Moreover, Score_i is a Monte Carlo estimate from 100 binary draws; no repeated-detection statistics are reported, and the main gains in Table 2 (+2.40, +1.06, +1.34, +1.08) have no error bars. The Section 2.3 motivation is further weakened by Table 10, where the row labels for Mistral-7B and LLaMA3-3B are swapped; after correction, the benefit of adding 25% wrong data is only +0.17/+0.57 points. The load-bearing premise that Eq. (1) yields a valid, stable conflict signal is therefore not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses knowledge conflict in domain-specific supervised fine-tuning (SFT) of LLMs. It proposes a query diversification strategy that permutes answer options and samples multiple responses from the frozen base model to compute a per-sample conflict score (Eq. 1), splits the training data into four conflict-level subsets, and then fine-tunes with sample-adaptive rewards (Eq. 2), assigning small weights to the most conflicting subsets and full weight to moderate- and low-conflict subsets. The method, KaFT, is evaluated on medical QA benchmarks across four LLMs, with additional experiments on out-of-domain QA, multilingual medical QA, hallucination detection, and other domains. The main claimed results are consistent gains over vanilla SFT and the No-conflict and Self-aligning baselines.","tokens_in":20160,"tokens_out":4367,"duration_ms":38612,"significance":"If the central claim holds, the paper offers a simple and practical way to improve domain-specific QA fine-tuning by reweighting training samples according to a measured conflict level. The strengths of the paper are the breadth of the evaluation (four LLMs, six medical QA benchmarks, OOD generalization, hallucination evaluation, and multilingual evaluation), the use of publicly available benchmarks and the lm-evaluation-harness toolkit, and the honest limitation statement about model scale. The contribution, however, rests on two load-bearing assumptions: that Eq. (1) measures knowledge conflict rather than general difficulty or model uncertainty, and that the reward hyperparameters, which are tuned on the evaluation benchmarks themselves, are not responsible for the reported gains. The paper is therefore potentially useful but not yet fully convincing.","major_comments":[{"comment":"The load-bearing conflict score is the fraction of correct answers over No×Nr samples generated from the frozen base model. This measure conflates knowledge conflict with question difficulty and with pretraining unfamiliarity: under the definitions cited in §1 (Ren et al. 2024; Xu et al. 2024), a model that simply does not know the answer is not in conflict with the training context. Eq. (1) therefore cannot distinguish 'the model holds a specific competing answer' from 'the model is guessing' or 'the question is hard.' Because the reward assignment in Eq. (2) and all downstream gains in Table 2 depend on this score, the paper should provide evidence that low-Score samples exhibit a consistent alternative answer rather than random or uncertain behavior, and should compare KaFT against a difficulty-based reweighting baseline (e.g., down-weighting samples with low base-model accuracy without option permutation). Without such evidence, the 'knowledge-aware' mechanism is not established.","section":"§2.2, Eq. (1)"},{"comment":"The default rewards α=0.1 and β=0.5 are selected from the grid in Fig. 4 and Table 13 as the configuration that maximizes average performance on the same six medical QA benchmarks used for the main results in Table 2. This makes the comparison against Vanilla SFT and the other baselines partly self-referential, because the baselines do not receive equivalent hyperparameter tuning on the evaluation benchmarks. The authors should either tune α, β, and the number of subsets on a held-out validation split, or report a sensitivity analysis showing that the relative ordering over baselines is stable across a range of settings and random seeds. In addition, the main gains in Table 2 (e.g., +2.40 for Mistral-7B, +1.06 for Qwen1.5-7B) are reported without error bars or significance tests; for the smaller gains, the 'consistent and significant' claim is currently unsupported.","section":"§4.2, Fig. 4, Table 2"},{"comment":"The text states that 'might-right performs better among all LLMs,' but Table 9 shows the opposite for LLaMA3-3B: the right subset averages 46.98 while might-right averages 46.05. Since the reward design in Eq. (2) is motivated by the claim that might-right is the most valuable subset, this contradiction needs to be resolved. If the pattern is not consistent across backbones, the motivation for assigning weight 1 to might-right and the broader claim of a universal trend are weakened.","section":"§2.3, Table 9"},{"comment":"The row labels in Table 10 appear to be swapped for Mistral-7B and LLaMA3-3B: the 0% column matches the No-conflict averages in Table 2 (Mistral-7B: 48.22; LLaMA3-3B: 46.98), so what is labeled 'Mistral-7B' is actually LLaMA3-3B, and vice versa. After correcting the labels, introducing 25% wrong data improves the average by only +0.17 (Mistral-7B) and +0.57 (LLaMA3-3B), which is much weaker than the text suggests and also within the range of sampling noise given that Figure 2(c) uses only three seeds. The authors should correct the table and temper the claim that 'appropriately applying conflict data is more beneficial than filtering.'","section":"Fig. 2(c), Table 10"}],"minor_comments":[{"comment":"There are typographical and formatting issues: 'Sumarization' in Table 6 should be 'Summarization', and Table 7 contains 'inbold' as a formatting artifact.","section":"Table 6, Table 7"},{"comment":"The objective is written as E_{(q,o,a,R)∼D}[R log M(a|q,o)], but R is a deterministic function of the reward assignment rather than part of the data distribution; writing the reward outside the expectation or as R_i · log M(a_i|q_i,o_i) would be clearer and would avoid confusion about what is being sampled.","section":"Eq. (2)"},{"comment":"The probing prompt uses three randomly selected few-shot examples, but §2.2 does not state how these examples are chosen or whether the reported Score_i averages over different few-shot selections; since a stated motivation is to reduce sensitivity to few-shot examples, this detail should be specified.","section":"Appendix A.3"},{"comment":"The paper says evaluation uses temperature 0 for reproducibility, while conflict detection uses temperature 0.7; this is fine, but the two decoding settings should be stated together in the setup so readers do not confuse the detection and evaluation protocols.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline: the proposed method is simple and the experimental scope is broad, but the central mechanism is not yet established because Eq. (1) conflates conflict with difficulty, and the reward hyperparameters are selected on the evaluation benchmarks. I recommend major revision rather than rejection, since the empirical gains could be made credible with a proper validation protocol and with evidence that low-Score samples correspond to genuine knowledge conflict. The row-label swap in Table 10 and the overstatement in §2.3 should also be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: KaFT reweights SFT examples by a conflict score built from permuted-option, multi-sample probing. The gains are real, modest (~1-2 points over vanilla SFT across four LLMs), and the paper has more evaluation breadth than most work in this niche — six medical benchmarks, OOD tests, multilingual, and hallucination. The specific combination of diversified probing and sample-adaptive rewards is new relative to Ren et al. and Gekhman et al., which filter or self-align. The ablation showing the detector picks a wrong subset that causes a 27-point drop (vs. 5-point for the ICL baseline) is genuinely informative.\n\nThe soft spots are real but not fatal. The load-bearing assumption is that Score_i in Eq. (1) measures knowledge conflict. It doesn't distinguish a model that holds a specific competing answer from a model that simply doesn't know, and the cited definitions (Ren, Xu) treat unknown as not conflict. If low score mostly marks hard questions, the reweighting is just difficulty down-weighting and the 'knowledge-aware' story is unsupported. The paper doesn't report stability of the score across permutations or sampling, which is testable and needed. Second, alpha and beta are chosen because they maximize the average of the six test benchmarks. That is circular for the headline numbers, though Figure 4 shows most of the 3x3 grid beats vanilla SFT, so the effect isn't purely a selection artifact. Still, no error bars and no significance tests on ~1-point average gains; 'significant' in the abstract is not supported. Third, there are two labeling errors in the supplementary tables: Table 9 contradicts the Section 2.3 claim that might-right is best for all LLMs (LLaMA3-3B right 46.98 > might-right 46.05), and Table 10 swaps the Mistral-7B and LLaMA3-3B blocks. These are easy fixes but should be caught.\n\nOverall: I think this deserves a serious referee. The method is simple, the experiments are broad, and the central question—does reweighting by a conflict-ish score help, and is the mechanism conflict or difficulty—is answerable with a held-out split and a difficulty control. I'd accept it with major revision; as is, it's a conditional accept. Not a paper I'd cite for the conflict mechanism until that control exists, but I'd put it on the reading group list to discuss the difficulty-versus-conflict identification problem.","headline":"KaFT is a plausible sample-reweighting method with modest but consistent gains, yet the paper doesn't establish that its score measures knowledge conflict rather than difficulty.","tokens_in":20664,"tokens_out":3260,"would_cite":false,"duration_ms":29967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that weighting each training sample by how much it conflicts with the model's existing knowledge improves domain-specific QA fine-tuning, with gains up to 2.40% over standard fine-tuning across four large language models.","keywords":["knowledge conflict","supervised fine-tuning","domain-specific question answering","sample reweighting","conflict-aware rewards","knowledge-aware fine-tuning","large language models","medical QA"],"falsifier":"Recompute the conflict score twice on the same training set with independent option shuffles and response samples; if the two score lists are near random orderings of each other (e.g., Spearman $\\rho < 0.5$) or if KaFT trained on the second ordering no longer beats equal-weight fine-tuning, the central mechanism is not stable.","tokens_in":19619,"feed_emoji":"⚖️","tokens_out":17504,"duration_ms":132218,"temperature":0.7,"pith_summary":"Supervised fine-tuning of large language models for domain-specific multiple-choice QA usually treats every training sample equally. This paper argues that equality is the wrong default: samples whose labeled answer conflicts with what the base model already knows can hurt the final model, while samples near the boundary of its knowledge are the most useful. KaFT scores each sample by how often the frozen base model picks the correct answer under shuffled options and repeated sampling, splits the data into four conflict-level groups, and trains with down-weighted losses on the two most conflicted groups. Across four open models and several medical QA benchmarks, it reports consistent gains over standard fine-tuning and over filtering conflict data outright, together with better out-of-domain generalization and less hallucination. The paper's point is that conflict-aware weighting is a cheap, sample-level alternative to data filtering or answer self-alignment.","feed_headline":"Conflict-aware weighting lifts fine-tuned AI model QA up to 2.4%","feed_subtitle":"A quick pre-training probe scores whether each sample matches the model's own knowledge, then down-weights mismatches.","key_machinery":"The load-bearing object is the conflict score in Eq. (1): for each training question, KaFT reorders the multiple-choice options and samples responses from the frozen base model at temperature 0.7, and the score is the fraction of those sampled responses that match the labeled answer. That score defines a quartile partition of the training set, and the reweighted objective in Eq. (2) multiplies each sample's loss by a reward that is 1 for the two low-conflict quartiles, 0.5 for the 'might-wrong' quartile, and 0.1 for the 'wrong' quartile. The reweighting is what lets the model keep learning from high-conflict examples without letting them dominate, and the option shuffling plus response sampling is what makes the score more stable than a single greedy in-context-learning probe.","core_discovery":"The central claim is that the conflict between a model's internal knowledge and a training sample's labeled answer is a reliable guide to how that sample should be weighted during SFT. Using a query-diversified probe—ten shuffled option orders times ten temperature-0.7 samples from the frozen base model—KaFT computes a score for each question, sorts the training set into right, might-right, might-wrong, and wrong quartiles, and trains with rewards 1, 1, 0.5, and 0.1. Training on the wrong quartile alone produces large accuracy drops, yet mixing some of those samples back into the clean data helps, so the paper concludes that conflict samples should be down-weighted rather than deleted. Across Mistral-7B, Qwen1.5-7B, LLaMA3-3B, and LLaMA3-8B, KaFT reports average gains up to +2.40 percentage points over vanilla SFT and +5.73 points over the base model on a six-benchmark medical QA suite, and similar benefits on out-of-domain QA and hallucination evaluation.","pith_inferences":["Editorial extension: because the conflict score is computed once from the frozen base model, one could recompute it after partial fine-tuning and reweight in a second round, testing whether the benefit comes from capturing the pretraining knowledge boundary.","Editorial extension: the hand-set rewards 0.1/0.5/1/1 and the choice of four quartiles are discrete simplifications; a smooth monotone mapping from score to weight, fit on a validation set, would likely remove the need to tune $\\alpha$ and $\\beta$ by hand.","Editorial extension: since the probing step needs only the base model and the training set, KaFT-style scores could be used as a data-selection filter before any SFT, composing naturally with curriculum or deduplication pipelines.","Editorial extension: the 'might-right beats right' pattern suggests a general curriculum principle, so the same score might transfer to instruction tuning or other multiple-choice settings, although the paper does not test those."],"forward_implications":["If KaFT's claim is right, SFT pipelines can reduce the harm of knowledge conflict without discarding data: a one-time probing pass over the training set is enough to assign per-sample weights.","The finding that 'might-right' samples often outperform 'right' samples implies that the most useful supervised data sits at the boundary of what the base model already knows, not far inside it.","Down-weighting conflict data should transfer beyond medical QA, since the paper reports the same recipe helps on history, engineering, and law QA.","Because the method lowers overfitting to conflicting labels, it should reduce hallucination on held-out tasks, which the paper supports with HaluEval results.","Finer-grained conflict partitions (8 subsets instead of 4) improve results further, so the discrete reward grid is not the limit of the approach."],"supporting_citations":[{"why":"Introduces the in-context-learning conflict probe and the self-aligning baseline that KaFT extends and compares against.","marker":"Ren et al. (2024)"},{"why":"Documents that fine-tuning on new knowledge encourages hallucination, the failure mode KaFT is designed to avoid.","marker":"Gekhman et al. (2024)"},{"why":"Supplies the multiple-response self-consistency idea behind the conflict score in Eq. (1).","marker":"Wang et al. (2023)"},{"why":"Provides the MedQA dataset used as the training corpus and the primary in-domain evaluation.","marker":"Jin et al. (2021)"},{"why":"Defines knowledge conflict as the discrepancy between an LLM's internal and contextual knowledge, the construct KaFT measures.","marker":"Xu et al. (2024)"}],"fun_headline_variants":["Conflict-weighted fine-tuning: +2.4% QA accuracy","KaFT: weight samples by conflict, boost QA by 2.4%","Knowledge-aware fine-tuning lifts LLM QA up to 2.4%","Down-weight mismatches, gain 2.4% in QA fine-tuning","Use model's own knowledge to weight SFT: +2.4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the probe of the frozen model's answers on shuffled options reliably tells which training samples will help and which will hurt after fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Conflict-weighted fine-tuning: +2.4% QA accuracy","KaFT: weight samples by conflict, boost QA by 2.4%","Knowledge-aware fine-tuning lifts LLM QA up to 2.4%","Down-weight mismatches, gain 2.4% in QA fine-tuning","Use model's own knowledge to weight SFT: +2.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1481,"prompt_tokens":996,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":612,"tokens_out":485,"duration_ms":4547,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:16:12.026113+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the conflict score twice on the same training set with independent option shuffles and response samples; if the two score lists are near random orderings of each other (e.g., Spearman $\\rho < 0.5$) or if KaFT trained on the second ordering no longer beats equal-weight fine-tuning, the central mechanism is not stable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the in-context-learning conflict probe and the self-aligning baseline that KaFT extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that fine-tuning on new knowledge encourages hallucination, the failure mode KaFT is designed to avoid."}],"review_version":1}