{"id":"9ef327b4-ea8b-451a-a42b-807f4eba9f1e","arxiv_id":"2505.17470","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning only on questions a model answers incorrectly reaches nearly the same performance as full-data fine-tuning while substantially reducing training time.","lead":"This paper proposes SLearnLLM, a self-learning framework that filters a fine-tuning dataset down to the question-answer pairs a language model answers incorrectly, then fine-tunes only on those. In agriculture and medicine tests, it cut training time by about 25 to 45 percent while keeping accuracy within a point of full-data fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparability claim rests on an unvalidated LLM-as-judge loop: the same CoT scoring template is used to filter training data and to evaluate results, with no human-agreement check reported.","rationale":"The reader's weakest assumption was that the target LLM's chain-of-thought self-grading reliably identifies unknown knowledge, with no validation against human judgment or ground truth. My stress test converges on the same load-bearing point, sharpened into a concrete threat: the paper uses an LLM-as-judge not only for filtering but also for the evaluation that produces the headline numbers, and both uses share the same unvalidated prompt template. If this judge is biased, the filter could keep the wrong pairs and the evaluation could simultaneously reward superficial improvements, making the claimed comparability to full fine-tuning an artifact. This concern is internal to the paper's evidence, not merely a disagreement with consensus: the paper explicitly asserts scoring accuracy without providing agreement data, and it evaluates with the same rubric it uses to define 'incorrect.' A human-judgment audit would settle whether the concern lands. The paper's other weaknesses (single runs, no error bars, potential validation-set overlap with training corpora) are real but secondary; the judge-dependence is the most fundamental because it affects the validity of the central comparison itself. Since the reader's verdict was already CONDITIONAL and this concern is the same assumption, I recommend no change to that verdict, but the condition should be made explicit: the authors must supply human-validated evaluation and filter-quality statistics before the efficiency claim is accepted.","tokens_in":7588,"tokens_out":3725,"duration_ms":30844,"concrete_test":"Select a stratified random sample of 200 validation responses per model-domain from Table 2 (base, full, SL) in both agriculture and medicine. Have domain experts independently score each response as correct/incorrect using a written rubric. Compute (a) Cohen's kappa between GPT-4o scores and expert scores, and (b) human-rated accuracy for each model. If the human-rated difference between full and SL models exceeds a pre-specified margin (e.g., 3 points) in any domain, or if GPT-4o agrees with experts less than, say, 80% of the time, the comparability claim is not supported. As a second check, sample 200 filtered-out and 200 kept training pairs from the self-check step and have experts judge whether the target model's original answer was truly incorrect, yielding precision/recall of the self-grader.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SLearnLLM achieves 'comparable performance improvements' to full fine-tuning depends on two premises: (1) the target LLM's self-check correctly identifies QA pairs the model truly gets wrong, and (2) the evaluation metric faithfully measures quality. Section 3.2 asserts that the CoT self-check 'ensures the accuracy and reliability of the scoring results' purely by comparing the answer and the model response, but no validation against human judgment or ground truth is provided. Section 4.3 then evaluates all models using GPT-4o 'with the same prompt of scoring the answers in Methods section.' This creates a shared-judge bias: the filter selects data based on one LLM-judge rubric, and the headline Table 2 scores are produced by a second LLM using the identical rubric. If that rubric systematically rewards superficial stylistic alignment with the reference answer rather than true correctness, then fine-tuning on the filtered set could inflate GPT-4o scores without improving real domain competence. The paper reports no human agreement statistics, no confusion matrix for the self-grader, and no alternative metric (e.g., exact match on the likely true/false and multiple-choice subsets). Consequently, the observed 0.2-0.8 point gaps between SL and full fine-tuning may be an artifact of optimizing a biased judge, not evidence of comparable performance. This is the weakest link in the argument, and it directly undermines both the performance and the efficiency claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SLearnLLM, a four-step pipeline for domain-specific supervised fine-tuning (SFT). The target LLM (Qwen1.5-7B/14B/32B) first answers all QA pairs in a domain SFT set; it then uses a chain-of-thought prompt to grade its own answers against the reference answers, treating itself as an 'expert language teacher'; it filters the pairs it marked incorrect; and it fine-tunes on that subset with LoRA. The authors report experiments on agricultural (57,638 QA pairs) and medical (35,000 QA pairs) sets, with evaluation by GPT-4o on 1,000 manually crafted held-out QA pairs per domain. Table 2 shows that models fine-tuned on the filtered subset score within 0.2-0.8 points of models fine-tuned on the full data while using substantially less GPU time. The paper concludes that the unknown-knowledge filtering improves training efficiency.","tokens_in":7816,"tokens_out":8525,"duration_ms":89009,"significance":"The strength of the paper is its direct, parameter-free framework: no extra trainable parameters, standard LoRA fine-tuning, and direct wall-clock time measurements across six model/domain combinations. The reported consistency of the -SL versus -full gap is a useful empirical signal. The significance is currently capped by three omissions: the grading loop is unvalidated against human judgment, the evaluation judge shares its rubric with the filter, and no random-subset or other data-selection baseline is provided. These are common, fixable requirements, but until they are addressed the headline claim of 'comparable improvements' is not fully supported.","major_comments":[{"comment":"The headline scores are produced by GPT-4o using the same CoT scoring prompt that defines the filtering criterion in Section 3.2. Because the filter retains data that the target model answers 'incorrectly' according to that rubric, and the evaluation rewards the same rubric, the observed 0.2-0.8 point gaps could partly reflect alignment to the judge rather than real domain competence. The paper reports no human agreement statistics, no confusion matrix for the self-grader, and no alternative metric (e.g., exact match for true/false or multiple-choice items). This is load-bearing for the central comparability claim.","section":"Section 4.3/Table 2"},{"comment":"The claim that focusing on unknown knowledge drives the improvements is not supported by any baseline that controls for subset size. All reported SL results train on the incorrect-answered subset, and the paper does not compare against fine-tuning on a random subset of the same size or against uncertainty/self-consistency filters. Without such a control, the comparable performance could be an artifact of any reduced-size fine-tuning, especially since the models' baseline scores and the large gains suggest much of the QA content is learnable. The authors should add at least a random-subset baseline at matched data volume.","section":"Section 4.4/Table 2"},{"comment":"The self-check grader is asserted to be accurate ('This step ensures the accuracy and reliability of the scoring results'), but no evidence is provided. The grader is the target model itself, and for open-ended answers the grading requires domain knowledge the model may lack; a 7B model with a 24.3 medical baseline score is being asked to judge correctness of its own medical answers. The authors should validate the self-grader against human labels or a held-out labeled sample and report precision/recall of the filter. Until then, both the performance and efficiency claims rest on an unmeasured premise.","section":"Section 3.2"},{"comment":"The evaluation description is internally confusing. The text says the score is 'normalized by dividing it by the total number of questions in the incorrectly answered QA set,' but the validation set described in the same section has 1,000 QA pairs per domain. If the denominator is the incorrect-answered training set, then the reported percentages in Table 2 are not computed as described; if it is the validation set, the sentence should be corrected. The paper also reports no standard deviations or multiple seeds for any of the scores or time measurements, so the 0.2-0.8 point differences between -full and -SL cannot be assessed statistically.","section":"Section 4.3"}],"minor_comments":[{"comment":"The rule that SLearnLLM reduces total time when the incorrect-answer rate is below 75% (and 'significantly' below 50%) is stated without derivation or supporting data; the break-even point should depend on the ratio of training time to inference/filtering time, not only on the data fraction.","section":"Section 4.4"},{"comment":"Several references are missing spaces and punctuation (e.g., 'Ferdinan[6]proposed...'), and the text would benefit from a careful proofread.","section":"Section 2"},{"comment":"The column header 'Total Incorrect-answered Time/GPU hours' is confusing; it presumably refers to the time spent on answering and self-scoring, but it should be labeled explicitly.","section":"Table 1"},{"comment":"The validation set construction is described only as 'manually crafted'; the authors should specify how many annotators, whether experts were used, and how independence from the SFT sets was ensured.","section":"Section 4.3"},{"comment":"The full CoT scoring prompt is not provided verbatim; since the evaluation uses 'the same prompt,' the exact prompt should be included in an appendix for reproducibility.","section":"Section 3.2/Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study of a simple idea, but the current evidence is insufficient for the strength of the claims. The requested additions (random-subset baseline, human validation of the grader, error bars, and an alternative metric) are within the scope of a revision; if the authors cannot provide them, the contribution would be too thin for a serious journal. Please also check the originality of the filtering idea against the cited self-learning literature before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you're doing SFT on a budget. The core move is simple: take an existing SFT dataset, have the target model answer the questions, self-grade against the reference answers with a CoT prompt, and fine-tune only on the answers it got wrong. That filtering step is new relative to the self-improvement methods they cite (response consistency, rationale generation, MCTS, knowledge graphs); those all generate new training data rather than filter an existing set. So the novelty is genuine, if modest.\n\nWhat the paper does well: the experiments cover six model-domain combinations and consistently show the filtered model within 0.2–0.8 points of full fine-tuning while cutting GPU time by 25–45%. Table 2 is coherent. They also measure total time including answering and scoring, which is the honest way to count the overhead. The limitations section is candid about hallucination risk and weak models.\n\nThe soft spots are real but not fatal. No error bars, no multiple seeds, and the evaluation is GPT-4o using the same CoT scoring prompt as the filtering step. That creates a legitimate worry: if the rubric rewards superficial matching to the reference answer, both the filter and the evaluation are biased in the same direction, and the 0.2–0.8 point gap could be understated or overstated. The paper offers no human-agreement check for the self-grader, no confusion matrix, no alternative metric like exact match on true/false and MCQ subsets. The validation set is built from the same corpora as the training data, so there's possible overlap they don't address. No code or data released, so the numbers are hard to reproduce.\n\nThat said, the time savings are a directly measured fact, not an artifact of the judge. Even if the performance comparability needs stronger backing, the efficiency gain is there. The central argument—filter by unknown knowledge, fine-tune on that—holds up as a useful practical heuristic.\n\nWho this is for: practitioners adapting models to vertical domains with a fixed SFT dataset and limited GPU budget. It's a production trick, not a field re-organizer.\n\nRecommendation: send it to peer review. It's a simple, plausible idea with a real experimental question worth nailing down. Ask for multiple seeds, human evaluation on a sample, a self-grader accuracy analysis, and ideally released code/data. If the shared-judge concern survives those additions, that's still a publishable efficiency result.","headline":"A simple, plausible data-filtering trick for SFT with a genuine novelty and consistent efficiency numbers, but the shared judge between filtering and evaluation makes the comparability claim fragile.","tokens_in":8393,"tokens_out":2422,"would_cite":false,"duration_ms":25172,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a domain model only on the questions it answers incorrectly—as graded by its own chain-of-thought self-check—matches full-dataset fine-tuning in accuracy while cutting total training time substantially.","keywords":["large language models","self-learning","domain-specific fine-tuning","supervised fine-tuning","chain-of-thought","knowledge filtering","efficient adaptation","low-rank adaptation"],"falsifier":"Have human domain experts grade a random sample of the model responses (e.g., 500 QA pairs per domain) and compare their correct/incorrect labels with the model's self-check labels; if agreement is far below the roughly 40-60% incorrect-answer rates the self-check reports, the filter is not isolating unknown knowledge. A cheaper check: fine-tune on a random subset of the SFT dataset matched in size to the incorrect-answer set—if its validation score matches the SL score, the gains are explained by data quantity, not by the selection of wrong answers.","tokens_in":7360,"feed_emoji":"🎯","tokens_out":9804,"duration_ms":68502,"temperature":0.7,"pith_summary":"The paper claims that a target LLM can be adapted to a specialized domain more efficiently by fine-tuning it only on the questions it answers incorrectly, as judged by its own chain-of-thought self-check, instead of on the entire supervised fine-tuning dataset. In experiments with Qwen1.5 models of 7B, 14B, and 32B parameters over agricultural and medical datasets, models fine-tuned on the filtered incorrect-answer set scored within 0.2-0.8 points of full-data fine-tuning while cutting total GPU time by roughly 30-45%. The motivating idea is that much of a typical SFT dataset overlaps with knowledge the model already has, so training on the whole dataset spends compute on what is already learned. If the self-grading filter reliably isolates genuine gaps, the result is a cheap, iterative way to target a model's unknown knowledge.","feed_headline":"Wrong-answer-only fine-tuning matches full training","feed_subtitle":"Filtering a domain dataset to questions the model misses yields near-identical scores at roughly half the GPU time.","key_machinery":"The framework's central object is the incorrect-answered QA set, produced by a two-stage Chain-of-Thought self-check prompt in which the target LLM takes the role of 'an expert language teacher'. The model scores its own response against the reference answer for consistency and accuracy, then marks incorrect responses with a 'Yes' flag; this filtered set becomes the training data for LoRA fine-tuning. The self-check prompt carries the whole argument, because it is the mechanism that separates unknown from known knowledge, and the claimed efficiency gain depends on that separation being accurate. LoRA (low-rank adaptation) is the parameter-efficient fine-tuning method used for all training runs.","core_discovery":"The central discovery is that the incorrect-answered QA set—the subset of a domain SFT dataset that the model itself flags as wrong—carries nearly all of the training signal of the full dataset. Across all six model-domain combinations, validation accuracy after fine-tuning on this filtered set stayed within 0.2-0.8 points of full-data fine-tuning (e.g., 99.1 vs 99.3 for Qwen1.5-32B-Chat in agriculture), while total time fell from 71.8 to 40.2 GPU-hours at the largest scale. The authors present this as evidence that their self-check chain-of-thought scoring reliably identifies the knowledge the model lacks, and that concentrating fine-tuning on that knowledge is sufficient for domain adaptation.","pith_inferences":["Beyond the paper: if the self-check grader is reliable, the same filter can be applied iteratively—fine-tune on one incorrect-answer set, re-run the self-check, and fine-tune again—so SLearnLLM naturally extends to a curriculum that may converge with less data than one-shot full fine-tuning.","Beyond the paper: the framework makes a falsifiable comparison: fine-tuning on a random subset of the SFT dataset of the same size should underperform fine-tuning on the self-filtered incorrect set; if it does not, the gains come from data quantity rather than from identifying unknown knowledge.","Beyond the paper: the observed incorrect-answer rate could serve as a cheap proxy for a model's domain competence, which might be reused for data selection or model routing decisions outside this fine-tuning pipeline.","Beyond the paper: the validation scores themselves are produced by GPT-4o grading, so a natural next test is to measure how well that LLM grader agrees with human experts before treating the reported single-point differences as meaningful."],"forward_implications":["Fine-tuning only on self-identified incorrect answers loses less than one validation point while cutting total GPU time by roughly a third to under a half across the tested scales and domains.","The time savings grow with model size: for Qwen1.5-32B-Chat in agriculture, total time drops from 71.8 to 40.2 GPU-hours, so the method becomes more attractive for larger models.","When fewer than 75% of QA pairs are answered incorrectly, the framework reduces overall time; below 50% the savings are substantial, giving a simple rule of thumb for when to apply it.","Because the method uses only the target model and the SFT dataset, it needs no external knowledge retrieval or extra annotation, and any hallucination-mitigation technique developed for ordinary SFT can be layered on top."],"supporting_citations":[{"why":"supplies the chain-of-thought prompting mechanism that powers the self-check scoring step.","marker":"[20]"},{"why":"provides the low-rank adaptation (LoRA) method used for all fine-tuning runs.","marker":"[8]"},{"why":"defines supervised fine-tuning, the baseline that full-dataset fine-tuning instantiates.","marker":"[12]"},{"why":"references the GPT-4 model family used to generate agricultural QA pairs and to grade validation responses.","marker":"[1]"},{"why":"supplies the zero-shot prompting strategy used to generate agricultural QA pairs from domain corpora.","marker":"[10]"}],"fun_headline_variants":["Self-check filters SFT data to wrong answers only","LLM self-grading cuts fine-tuning time in half","Train on mistakes: efficient domain adaptation","Wrong-answer subset suffices for fine-tuning","Self-filtered misses match full-data SFT gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's own step-by-step self-scoring reliably identifies which answers are genuinely wrong; the paper does not validate this grader against human judgments, so if the self-check is noisy the filtered set stops being the unknown knowledge and both the performance and efficiency claims collapse.","fun_headline_variants_meta":{"raw":{"variants":["Self-check filters SFT data to wrong answers only","LLM self-grading cuts fine-tuning time in half","Train on mistakes: efficient domain adaptation","Wrong-answer subset suffices for fine-tuning","Self-filtered misses match full-data SFT gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000499,"raw_usage":{"total_tokens":2435,"prompt_tokens":928,"completion_tokens":1507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1434}},"tokens_in":544,"tokens_out":1507,"duration_ms":9562,"temperature":1.0,"reasoning_tokens":1434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:45:50.212515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human domain experts grade a random sample of the model responses (e.g., 500 QA pairs per domain) and compare their correct/incorrect labels with the model's self-check labels; if agreement is far below the roughly 40-60% incorrect-answer rates the self-check reports, the filter is not isolating unknown knowledge. A cheaper check: fine-tune on a random subset of the SFT dataset matched in size to the incorrect-answer set—if its validation score matches the SL score, the gains are explained by data quantity, not by the selection of wrong answers.","supporting_citations":[],"review_version":1}