{"id":"b0164fe3-2009-43ec-9675-735355f817aa","arxiv_id":"2506.03483","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"APT trains a model on its own wrong answers plus retrieved similar answers using iterative DPO with SFT loss, improving math, code, and instruction-following while keeping general benchmarks about flat.","lead":"This paper introduces APT, a fine-tuning method that makes an LLM generate answers, keeps the wrong ones as training examples, retrieves similar examples, and repeatedly trains the model to prefer correct answers. If it works as reported, it offers a practical recipe for specializing models on math, code, or instructions without clearly losing general performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core of APT is the Prometheus 2-7B score: Section 3.2 gates every 'bad case' on score<4, and the paper's own Limitations concede that APT is constrained by scoring quality; Table 2 and Figure 6 do not establish that the scores are correct, so the reported gains may be scorer artifacts.","rationale":"The paper presents a coherent method with thorough ablations, and the domain metrics (GSM8K exact match, HumanEval pass@10, AlpacaEval) are external to the training loop, which is real evidence that APT does something beyond optimizing the scorer. However, the single point of failure is the quality of the 'weakness case' selection signal, exactly as the reader identified and as the authors themselves concede in the Limitations section. If Prometheus's score<4 labels are not accurate measures of genuine errors, then the entire D_error construction in Section 3.2 is unreliable, the retrieved similar cases are retrieved around unreliable centroids, and the preference pairs in Eq. 7 encode the wrong direction. The paper's robustness evidence is also self-referential: Figure 6 counts 'erroneous samples' using the same Prometheus <4 criterion used to select training data, so a decline in that count could reflect the model learning to satisfy Prometheus's rubric rather than a real reduction in errors. The alternative check I propose would replace the scorer with an independent judge or human labels and retrain; if the gains persist, the concern is resolved, and if they vanish, the central claim fails. I therefore agree with the reader's weakest assumption and recommend keeping the CONDITIONAL verdict: the method is plausible and well-ablated, but the scorer-dependence and test-set-tuned choices need to be settled before the headline claim can be accepted.","tokens_in":22064,"tokens_out":8978,"duration_ms":106140,"concrete_test":"Sample 500 score<4 and 500 score>=4 instances from the GSM8K, CodeAlpaca, and Dolly pools; obtain independent human (or second-judge, e.g., GPT-4o or Llama-3-70B-Instruct) correctness labels and compute agreement with Prometheus. Then retrain the Llama-2 GSM configuration using only cases that the independent judge flags as wrong, keeping all hyperparameters fixed, and compare Table 1 GSM8K, HumanEval, and AlpacaEval numbers; if the gains drop by more than about 1 point, the headline result is scorer-dependent. Also re-run Figure 6 using the independent judge instead of Prometheus to break the circularity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 makes Prometheus 2-7B's 1-5 score the sole gate for constructing D_error: a prediction is treated as a weakness case iff s_i < 4. The Limitations section concedes that 'the performance of our model is primarily constrained by the quality of the scoring information provided.' That concession is the load-bearing point: if the scorer is biased or noisy, the selected cases, the retrieved neighbors, and the preference signal in Eq. 7 are all corrupted. The paper's only direct defense is Table 2, which shows Prometheus beats self-scoring on final benchmarks; this does not validate the score labels as ground truth, it only shows one judge outperforms another. Section 4.6/Figure 6 additionally measures 'erroneous samples' with the same Prometheus <4 criterion before and after training, so the robustness evidence is circular. The threshold (<4) and retrieval scale (1x) are also selected by comparing variants on the same test benchmarks (Figure 3, Table 4), so the reported gains may be optimistic due to selection on the evaluation data. If Prometheus systematically scores a stylistic property such as length or formatting rather than correctness, APT could appear to improve AlpacaEval and even exact-match metrics without actually fixing the underlying errors, and the gains would not replicate with a different scorer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APT, a framework for improving domain-specific LLM performance by iteratively (1) using a frozen assessment model (Prometheus 2-7B) to score the model's own predictions against ground truth, keeping only low-scoring 'bad cases' (score < 4) as dis-preferred data, (2) retrieving a matched number of similar cases from a large instruction pool via tag-based embedding similarity, and (3) optimizing a DPO objective regularized by an SFT loss. Experiments on Llama-2-7B and Mistral-7B-V0.3 report gains over mixed training and DMT baselines on GSM8K, HumanEval, and AlpacaEval while keeping the average of six general benchmarks roughly flat. Ablations examine the score threshold, retrieval method and scale, number of iterations, and the optimization objective, and scaling experiments on larger and math-specialized models are included.","tokens_in":22345,"tokens_out":2731,"duration_ms":31353,"significance":"If the reported gains are reproducible, APT is a useful, relatively simple addition to the specialist-finetuning toolbox: it turns a small amount of self-generated negative data plus retrieved neighbors into consistent domain improvements without degrading general benchmarks. The paper's ablations are reasonably complete: it isolates the contribution of bad-case filtering, similar-case retrieval, the tag-based retrieval granularity, the SFT regularizer, and the iterative loop, and it evaluates on two base models plus several stronger backbones. The use of independent domain benchmarks for the headline results (GSM8K exact match, HumanEval pass@10, AlpacaEval) means the central improvement claim is not vacuous. However, the paper's reliance on a single learned scorer for all data selection, and its selection of two key hyperparameters (the score threshold and retrieval scale) on the same test benchmarks used for the headline numbers, leave the central claim needing additional validation before it can be taken as established.","major_comments":[{"comment":"The score<4 criterion from Prometheus 2-7B is the sole gate for constructing D_error, and the paper's own Limitations admit that 'the performance of our model is primarily constrained by the quality of the scoring information provided.' Table 2 only shows that Prometheus outperforms self-scoring on the final domain benchmarks; it does not establish that the score<4 labels correspond to genuinely erroneous predictions. Because the selected training data and the iterative reward signal both depend on this scorer, a systematic scorer bias (e.g., favoring longer or better-formatted answers) could produce the observed gains without actually fixing reasoning errors. I would like to see a validation of the scorer against a small human-annotated set or against exact match for GSM8K and unit tests for HumanEval, or a version of APT run with a task-specific correctness signal (exact match, unit tests) to confirm the mechanism is not scorer-artifact-driven.","section":"§3.2, Limitations"},{"comment":"The bad-case threshold (score<4), the retrieval scale (1x), and the regularization weight alpha=0.5 are all selected by comparing variants on the same test benchmarks that appear in the headline Table 1 (Figure 3, Table 4, and Figure 8 in Appendix A.5). No held-out validation split is used for these choices, and no error bars or significance tests are reported. This selection-on-the-evaluation-data pattern makes the reported gains (e.g., +5.9 on Dolly for Llama-2) potentially optimistic. Please report multiple seeds or bootstrap confidence intervals, and either fix the hyperparameters a priori or tune them on a validation split disjoint from the test sets.","section":"§4.3, §4.4, §A.5"},{"comment":"Figure 6 measures the reduction in 'erroneous samples' using the same Prometheus <4 criterion that defined the bad-case training set. This makes the robustness analysis partly self-referential: the model may be learning to please the scorer rather than to be correct. The reduction in scorer-detected errors should be corroborated with the independent metrics already used elsewhere in the paper (exact match for GSM8K, pass@10 for HumanEval, and AlpacaEval win rate), and the two sources of evidence should be reported side by side.","section":"§4.6, Figure 6"}],"minor_comments":[{"comment":"The model name 'LLama-2' should be 'Llama-2' consistently.","section":"Abstract and throughout"},{"comment":"In several rows of the 'Ours' entries, numbers are run together (e.g., '70.959.4'); please fix the spacing in the Avg column.","section":"Table 7"},{"comment":"The caption says the retrieval size is 'justified in the ablation study (Figure 3)', but Figure 3 concerns the score threshold, not retrieval scale; the justification is in Table 4 itself. Please correct the cross-reference.","section":"§4.4, Table 4 caption"},{"comment":"The function ℓ in Eq. (7) is never defined; please state explicitly that it is the logistic loss, since the DPO derivation in Rafailov et al. uses the logistic function.","section":"Eq. (7)"},{"comment":"The claim that TAG processing requires 'only an additional 5% of the total time' is vague: please specify the denominator (e.g., 5% of the bad-case-generation plus training time) and provide timing measurements rather than a single percentage.","section":"§3.5"},{"comment":"The statement that code cannot be publicly linked weakens reproducibility; beyond contacting the authors, please consider releasing code and data in a non-company repository or providing a more detailed implementation appendix.","section":"Ethics Statement"}],"recommendation":"major_revision","confidential_remarks":"The reliability of the central claim hinges on whether the Prometheus-based bad-case selection is actually selecting errors. The paper's own limitation statement acknowledges this, and the current evidence (Table 2, Figure 6) is weaker than needed: Table 2 compares two judges, and Figure 6 uses the same judge that selects the data. I also note that the contributions claim 'first' is not fully persuasive given prior work on learning from mistakes (e.g., Zhao et al. 2024a), though this is not central to the technical assessment. The paper is within scope for an NLP/ML venue; the request should emphasize the need for independent scorer validation and for de-selecting hyperparameters on the test benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on APT. The genuinely new thing is the full loop: use an external evaluator to pick the model's own low-scoring outputs, retrieve a matched set of similar examples via tag-aware retrieval, then iterate DPO with an SFT term. Each component is known, but the assembly is clean and the ablations actually support the design choices. Two base models, three domains, and the general-capability checks give the headline claim—targeted error training improves domain tasks without degrading general ability—reasonable support. The case studies show concrete corrections, which helps.\n\nThe soft spots are real but not fatal. Prometheus 2-7B is load-bearing: its score<4 gate selects every training example, and the Limitations concede that the whole method inherits the scorer's quality. The robustness evidence in Figure 6 is partly circular—it counts 'erroneous samples' using the same scorer that selected them. That said, the main results on GSM and HumanEval are objective, and AlpacaEval uses a different judge, so the reported gains are not simply a Prometheus self-consistency artifact. More bothersome is that several hyperparameters (threshold, retrieval scale, alpha) are chosen by comparing variants on the same benchmarks that produce the headline numbers, and there are no error bars or significance tests. Code is also not public, which makes the empirical claims harder to check.\n\nI'd still send this to review. The recipe is practically useful, the problem is relevant, and the core claim is credible enough that a referee should engage. The key asks would be: validate the scorer against human judgments on a sample, move hyperparameter selection to a held-out set, and report variance across seeds. If those hold, this becomes a solid applied contribution.","headline":"A practical, well-ablated recipe for specializing LLMs on weak cases; the central claim holds up, but the load-bearing scorer and test-set-tuned hyperparameters need scrutiny in review.","tokens_in":22915,"tokens_out":2223,"would_cite":true,"duration_ms":24585,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"APT claims that a specialist LLM can be improved by iteratively preference-training it on only its own low-scored answers plus a matching set of retrieved similar cases, without eroding general ability.","keywords":["preference optimization","DPO","iterative training","bad case generation","similar case retrieval","domain-specific LLM","catastrophic forgetting","specialist models"],"falsifier":"Check the scorer against verifiable ground truth: on GSM8K, take predictions the scorer rates below 4 and compare them to exact-match correctness. If a substantial share of those 'bad cases' are actually correct answers, the filter is not selecting true errors; then rerunning APT with only verified-wrong cases should change the gains, and rerunning it with random labels should erase them.","tokens_in":21855,"feed_emoji":"🎯","tokens_out":8140,"duration_ms":83377,"temperature":0.7,"pith_summary":"The paper tries to show that a specialist LLM can be sharpened on a target domain without dulling its general abilities by training only on the model's own mistakes. The proposed APT method has the model answer its training questions, uses an external scorer to flag the weakest answers, retrieves similar correct examples from a general instruction pool, and then runs iterative preference training on those pairs. On Llama-2 and Mistral-7B, this raises math, coding, and instruction-following scores by up to about six points over mixed-data training while leaving the six-benchmark general average flat or slightly higher. If true, it gives a cheap, task-agnostic recipe for upgrading a model where it is weak instead of re-training it on more positive data.","feed_headline":"Train LLMs on their own bad answers to gain up to 6 points","feed_subtitle":"APT beats mixed-data fine-tuning on math, code, and instruction following while keeping general benchmarks flat","key_machinery":"The engine is a three-part loop. First, bad-case generation: the current model predicts answers for the domain SFT set, and an assessment model (Prometheus 2-7B) scores each prediction against the reference answer; predictions scoring below 4 become the dispreferred half of a preference pair, with the reference as the preferred half. Second, similar-case retrieval: each bad case is tagged (InsTag), the bad cases are averaged per tag into an embedding, and the most cosine-similar pool samples, one per bad case, are pulled in to form $\\mathcal{D}_{\\text{retrieval}}$. Third, iterative preference training: the model is tuned with the DPO objective plus an SFT-loss constraint, $\\mathcal{L} = \\mathcal{L}_{\\text{DPO}} + \\alpha \\mathcal{L}_{\\text{SFT}}$, and the new model re-enters the loop.","core_discovery":"On the paper's own terms, the central discovery is that negative self-generated examples are a sufficient training signal: the model's own low-scored outputs, used as dispreferred responses in DPO-style preference optimization, teach it to fix exactly the cases it gets wrong, and retrieving one similar correct example per bad case prevents overfitting and general-capacity loss. The method deliberately discards all high-scoring self-generated data and all unselected training data, training only on $\\mathcal{D}_{\\text{error}} \\cup \\mathcal{D}_{\\text{retrieval}}$. Iterating the generate-score-retrieve-train loop further reduces the number of low-scored outputs and improves downstream scores, with the largest gain appearing in the first iteration.","pith_inferences":["Because the bad-case filter is the only thing separating 'teach' from 'skip,' APT's ceiling is the scorer's judgment; a calibrated or verifiable scorer (e.g., exact-match on math) should raise the ceiling further, while a biased one would silently convert scoring errors into training errors.","The tag-based retrieval is a form of fine-grained hard-negative mining for preference data; this suggests that any cheap grouping signal, domain labels, clustering, or learned retrieval, could replace the tagger, and the paper's own ablation predicts finer grouping helps.","A direct stress test would use random selection of the same data volume: if random 'bad cases' perform nearly as well, the reported gains come from extra training data rather than from weakness targeting; the paper's comparison to non-selected 'Pred.' data already suggests selection matters, but random selection is the cleaner control.","APT should combine naturally with verifiable-answer domains such as math and code execution, where a pass/fail signal can replace the learned scorer and remove the weakest assumption."],"forward_implications":["Domain gains up to +5.9 on Llama-2-7B and +6.0 on Mistral-7B-V0.3 can be obtained by training only on error-derived preference data, beating continued SFT and data-mixing baselines.","General capability, measured as the average over MMLU, BBH, ARC, BoolQ, OpenBookQA, and WinoGrande, stays flat or rises slightly, so the selective training does not induce measurable forgetting.","Harder errors matter more: selecting only score-1 or score-below-4 predictions trains better and more efficiently than using all self-generated data.","Retrieving an equal number of similar cases (1x) is optimal; retrieving 2x or 3x reduces performance.","Iterating the loop improves the target domain further, and the method also lifts stronger models (13B and math-specialized checkpoints), suggesting the mechanism is not limited to weak models."],"supporting_citations":[{"why":"Prometheus 2-7B supplies the 1-5 scores that identify which self-generated predictions become bad cases; the whole selection step depends on it.","marker":"Kim et al. (2024)"},{"why":"DPO provides the preference-optimization loss that APT adapts by adding an SFT term.","marker":"Rafailov et al. (2023)"},{"why":"InsTag is the tagging model that partitions error cases and the retrieval pool into fine-grained semantic groups for tag-based retrieval.","marker":"Lu et al. (2024)"},{"why":"Sentence-BERT embeddings are the vectors used to compute average error embeddings and cosine similarity for retrieval.","marker":"Reimers and Gurevych (2019)"},{"why":"DMT, the data-mixing baseline, is the main prior method APT compares against and outperforms on domain tasks.","marker":"Dong et al. (2024)"},{"why":"Defines the composition of the retrieval pool of instruction datasets that supplies similar cases.","marker":"Wang et al. (2023a)"},{"why":"GSM8K is one of the three domain datasets; it generates the math bad cases and the math evaluation benchmark.","marker":"Cobbe et al. (2021)"},{"why":"Dolly-v2 is one of the domain datasets used for instruction-following training and evaluation.","marker":"Conover et al. (2023)"}],"fun_headline_variants":["LLMs learn from their own errors, gain 6 points","Error-only fine-tuning improves LLMs, no general loss","Self-generated mistakes as training signal for LLMs","Iterative error training: domain skill up, general flat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's load-bearing premise is that the external scorer's 1-to-5 score reliably tells genuine mistakes from acceptable answers; if the scorer is noisy or biased, the 'bad cases' and the preference signal built from them are wrong, and the promised domain gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["LLMs learn from their own errors, gain 6 points","Error-only fine-tuning improves LLMs, no general loss","Self-generated mistakes as training signal for LLMs","Iterative error training: domain skill up, general flat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000297,"raw_usage":{"total_tokens":1679,"prompt_tokens":857,"completion_tokens":822,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":756}},"tokens_in":473,"tokens_out":822,"duration_ms":9053,"temperature":1.0,"reasoning_tokens":756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:02:06.328187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the scorer against verifiable ground truth: on GSM8K, take predictions the scorer rates below 4 and compare them to exact-match correctness. If a substantial share of those 'bad cases' are actually correct answers, the filter is not selecting true errors; then rerunning APT with only verified-wrong cases should change the gains, and rerunning it with random labels should erase them.","supporting_citations":[{"cited_title":"Manning, Stefano Ermon, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"DPO provides the preference-optimization loss that APT adapts by adding an SFT term."}],"review_version":1}