{"id":"38cbd07f-d140-45f6-8aae-a319cb104a2b","arxiv_id":"2505.20771","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A curriculum-learning fine-tuning method that uses self-distilled data as an easy first stage and an adaptive scheduler to shift to real data improves LLM-based recommendation accuracy.","lead":"This paper proposes SOFT, a curriculum-learning fine-tuning method for LLM-based recommenders that first trains on the model's own generated outputs and then gradually shifts to real data. It reports an average relative accuracy gain of 37.59% over standard fine-tuning across three Amazon datasets, a drop-in recipe with no architecture changes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gain of 37.59% is not statistically supported: no error bars or significance tests are reported, and some improvements are just 0.001–0.004 absolute (1–4 hits in a 1,000-sequence test set), so the central empirical claim needs variance reporting before the mechanism can be evaluated.","rationale":"The reader's verdict is CONDITIONAL with MODERATE confidence, and their rationale already notes the lack of error bars and significance tests as a limitation. However, their stated weakest_assumption focuses on the transfer of category-level knowledge from self-distilled data. I agree that this mechanism assumption is unproven, but I find the statistical robustness of the headline empirical claim to be the more immediate load-bearing concern. If the differences in Table 2 are not statistically significant—which is plausible given absolute differences as small as 0.001 on a 1,000-sequence test set—then the central claim that SOFT significantly enhances recommendation accuracy fails outright, regardless of whether the self-distillation mechanism is sound. The reader's verdict of CONDITIONAL already requires strengthened evidence, including error bars and significance testing, so my stress-test does not change the verdict; it refines the priority: first establish that the effect is real, then investigate the mechanism. The concrete test I propose would settle the statistical concern directly, and if it passes, the paper's conditional acceptance would rest on firmer ground. My agreement is 'partial' because the reader identified a valid secondary concern, but the load-bearing concern is the unverified statistical claim.","tokens_in":11537,"tokens_out":6182,"duration_ms":72587,"concrete_test":"Run the provided code with 10 random seeds for SOFT, SFT, and SOFT(w/o SA) on BIGRec with the Game dataset, selecting α on the validation set only (not the test set). Report per-seed H@5, H@20, NG@5, and NG@20 as mean ± standard deviation, and compute a paired bootstrap 95% confidence interval (or one-sided paired t-test) for SOFT minus SFT on each metric. If the confidence intervals include zero on any headline metric, the 'significant enhancement' claim is unsupported; if all intervals exclude zero, the statistical concern is resolved and the mechanism question can then be addressed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SOFT 'significantly enhances the recommendation accuracy (37.59% on average)' over SFT baselines. The support for this claim is Table 2, which reports only averages over five runs with no standard deviations, confidence intervals, or significance tests. Several individual comparisons are extremely small in absolute terms; for example, LLaRA on Movie H@20 is 0.0200 for SFT versus 0.0210 for SOFT, a difference of 0.001 (one hit in the 1,000-sequence test set), and LLaRA on Game H@5 is 0.0110 versus 0.0230, a difference of 0.012 (twelve hits). With stochastic LoRA fine-tuning, sampling of training sequences, and no reported variance, these differences may easily be within run-to-run noise. The paper also tunes the curriculum hyperparameter α per dataset and backbone within {0.1, 1, 10, 100}, but does not state whether selection is on a validation set; if test performance guided α selection, the reported gains are further inflated by selection. The absence of statistical validation is more load-bearing than the reader's mechanism-level concern: if the observed differences are noise, the proposed method has no demonstrated effect, and the question of whether self-distilled 'easy' data transfers knowledge becomes moot. The paper's supporting evidence for the mechanism—lower first-epoch loss on SD data and 44.6% category-level accuracy—does not rescue the empirical claim, since lower loss on the model's own outputs is expected and category-level accuracy is not shown to drive item-level ranking gains. A statistically grounded replication is therefore the single check that would settle whether the headline claim holds.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Self-Optimized Fine-Tuning (SOFT), a curriculum-learning strategy for LLM-based recommender systems. SOFT first constructs an auxiliary dataset by using an SFT-finetuned LLM to generate its own output sequences (self-distillation), then fine-tunes the LLM on a weighted mixture of this self-distilled data and the real recommendation data, with a self-adaptive scheduler that adjusts the mixing weight τ based on the embedding distance between the model's current outputs and the target items. Experiments on three Amazon datasets (Game, Movie, Kindle) with two LLM backbones (BIGRec, LLaRA) report that SOFT improves over SFT and SDPO baselines by an average of 37.59% across metrics, and the paper includes an ablation of the scheduler and a hyperparameter sensitivity study.","tokens_in":11810,"tokens_out":4605,"duration_ms":46493,"significance":"If the empirical claim is validated, the idea of using a model's own outputs as an easy-to-learn curriculum before exposing it to real recommendation data is a simple and potentially useful contribution to LLM-based recommendation, and the paper provides a clear loss formulation and an open-source implementation. The paper also offers a plausible mechanistic story (lower first-epoch loss on self-distilled data, category-level knowledge) and an ablation showing that the scheduler contributes to the gains. However, the central claim of \"significant\" improvement currently rests on averages over five runs with no variance reporting, no significance tests, and only 1,000 test sequences, so the magnitude of the reported gains is not yet established. The mechanism is also not isolated from confounds such as the total amount of training data or the effect of training on the model's own outputs as a regularizer.","major_comments":[{"comment":"The claim that SOFT \"significantly enhances the recommendation accuracy (37.59% on average)\" is not statistically supported. Table 2 reports only averages over five runs without standard deviations, confidence intervals, or significance tests. Several individual improvements are very small in absolute terms: for example, LLaRA on Movie H@20 is 0.0200 (SFT) versus 0.0210 (SOFT), a difference of 0.001, which is one hit in the 1,000-sequence test set, and LLaRA on Movie H@5 is 0.0170 versus 0.0200, a difference of three hits. Given the stochasticity of LoRA fine-tuning and the modest training set size (4,096 sequences), these differences may be within run-to-run noise. Please report per-seed results or standard deviations, and provide paired significance tests (e.g., bootstrap or Wilcoxon signed-rank) over the five runs for each metric and dataset.","section":"§4.1, Table 2"},{"comment":"The hyperparameter α is tuned over {0.1, 1, 10, 100} for each dataset and backbone, but the paper does not state whether this selection is performed on the validation set or on the test set. Figure 5 shows that performance varies substantially with α, so if test performance was used to select α, the reported gains are inflated by selection bias. Please clarify the model selection protocol: state which split was used to choose α, report the chosen α values for each setting, and, if validation-based selection was used, give the corresponding validation metric.","section":"§4.1, §4.4, Figure 5"},{"comment":"The mechanism claim—that self-distilled data provides an easy-to-learn but meaningful curriculum that improves later learning on real data—is not isolated by the experiments. The lower first-epoch loss on the SD dataset (Figure 2a) is expected when a model is trained to imitate its own outputs, and the 44.6% category-level accuracy (Figure 2b) does not by itself show that this knowledge transfers to better real-data performance. The ablation \"w/o SA\" trains only on SD data, which does not control for the total number of gradient steps, the amount of data seen per step, or the effect of training on the model's own outputs as a regularizer. Please add control conditions that separate these factors, such as (i) a version of SOFT that mixes real data with a matched amount of randomly selected or synthetic auxiliary data, (ii) a curriculum using SD data generated by a different model or from an earlier checkpoint, and (iii) a fixed-τ (non-adaptive) curriculum with the same data mixture, to attribute the gains to the proposed SD curriculum rather than to the scheduler alone or to the additional training signal.","section":"§3.1, §4.3, Figure 2"},{"comment":"The evaluation uses only 1,000 test sequences and 4,096 training sequences per dataset. With the all-ranking protocol, the variance of H@5 and H@20 on 1,000 sequences is substantial; for example, a difference of 0.001 corresponds to a single hit. The paper should report confidence intervals (e.g., bootstrap over test sequences) and should either justify the sample sizes or increase them. This is particularly important given that the reported absolute gains are often only a few hits, which may not be practically meaningful even if statistically significant.","section":"§4.1"}],"minor_comments":[{"comment":"The caption states that \"Gain\" is the improvement over \"other fine-tuning strategies,\" but it does not specify that the baseline is the better of SFT and SDPO, not SFT alone. Please make this explicit, as the percentage gains are relative to the best fine-tuning baseline.","section":"Table 2 caption"},{"comment":"The paper says the self-distilled dataset is generated by \"LLMs after SFT\" but does not explicitly state that the SD dataset is fixed before the curriculum training begins and is not regenerated during SOFT training. Please state this clearly, as it affects the interpretation of the self-adaptive scheduler.","section":"§3.1"},{"comment":"The definition of the distance d_t uses token embeddings z_y, but the paper does not explain how the generated output y_t is grounded to an item name or description. Please specify the grounding procedure, since the distance calculation and the SD data construction both depend on it.","section":"§2.2 and §3.2"},{"comment":"There are several typos: \"validing\" should be \"validation\" in the dataset description, \"searving\" should be \"serving\" in Section 4.5, and \"incorperates\" should be \"incorporates\" in the baseline description.","section":"§4.1"},{"comment":"In the sentence \"This process has been demonstrated to be equivalent to unsupervised pre-training, which allows the training to to start in better basins of attraction,\" there is a duplicated \"to\". Please fix this typo.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is a reasonable curriculum-learning adaptation for LLM recommenders, and the code release is a positive feature. However, the primary empirical claim currently lacks statistical support, and the mechanism is not cleanly identified. The required changes—adding variance reporting and significance tests, clarifying the hyperparameter selection protocol, and adding control experiments—are substantial but feasible within the scope of the manuscript. I do not see any indication of misconduct; the issues are about rigor and experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tang et al. propose SOFT, a curriculum fine-tuning strategy for LLM-based recommenders: first train on a self-distilled dataset generated by the SFT model itself, then progressively shift to real data using a scheduler whose mixing weight decays based on the embedding distance between the model's current predictions and the target items. The idea is clean: use the model's own outputs as 'easy' data because they match the model's distribution, and use distance to the target as a difficulty signal. That combination is genuinely new in the LLM-rec subfield, and the method is described precisely enough to re-implement. The motivating observation—LLMs after SFT have roughly 44.6% category-level accuracy but under 1% item-level HR—is a nice empirical hook. The ablation without the scheduler (w/o SA) shows the scheduler contributes, and the code is linked.\n\nThe main soft spot is the empirical support for the headline '37.59% average improvement.' Table 2 reports only averages over five runs: no standard deviations, confidence intervals, or significance tests. Several gains are tiny in absolute terms, e.g., LLaRA on Movie H@20 goes from 0.0200 to 0.0210, one extra hit in a 1,000-sequence test set. With LoRA fine-tuning and 4,096 training sequences, that could easily be run-to-run noise. The paper also tunes alpha per dataset and backbone over {0.1, 1, 10, 100} but does not state whether that selection is on a validation set; if test performance guided alpha, the reported gains are inflated. A second gap: the closest prior self-distillation method for LM fine-tuning (Yang et al. 2024b) is cited but never used as a baseline, so we cannot see whether the specific curriculum scheduler is what helps over plain self-distillation. The self-referential data generation is a mild circularity, but it is not fatal since the final evaluation is on real interactions.\n\nOverall, this is a plausible and potentially useful recipe, not a breakthrough. The mechanism story is coherent and the paper is honest about its limitations (LoRA only, sequential rec only). Researchers working on fine-tuning LLMs for recommendation will get value from this, especially the curriculum formulation. A serious referee should engage, but the authors need to add error bars, run significance tests, clarify the validation protocol for alpha, and compare against the direct self-distillation baseline before the 'significant enhancement' claim is accepted.","headline":"Plausible and clearly-described curriculum recipe for LLM recommenders, but the headline 37.59% gain lacks statistical support and the paper needs variance reporting and a sharper baseline comparison before the claim is credible.","tokens_in":12427,"tokens_out":2552,"would_cite":true,"duration_ms":24627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 'Guidance+Tuning' curriculum that has an LLM recommender learn from its own self-distilled, easy-to-learn outputs before facing real user data improves recommendation accuracy by 37.59% on average.","keywords":["LLM-based recommender systems","self-distillation","curriculum learning","supervised fine-tuning","sequential recommendation","self-adaptive curriculum scheduler","knowledge gap","recommendation accuracy"],"falsifier":"Run SOFT against a control in which each self-distilled item name is replaced by a random item from the same category; if the control reproduces the gains, the easy distribution, not the semantic content, is what transfers. Separately, give standard SFT on real data enough additional epochs to match SOFT's total gradient updates and check whether the reported 37.59% gap closes to zero.","tokens_in":11289,"feed_emoji":"🎓","tokens_out":9780,"duration_ms":90595,"temperature":0.7,"pith_summary":"Large language models know a great deal about language, but their knowledge space is misaligned with the structure of recommendation data, and the paper argues this gap is why plain prompting and plain supervised fine-tuning both fall short. To bridge the gap, the paper proposes Self-Optimized Fine-Tuning (SOFT), a 'Guidance+Tuning' strategy that trains an LLM in two stages: first on an auxiliary dataset built from the model's own predictions after fine-tuning (easy, self-distilled data), then progressively on the real recommendation data (hard data) under an adaptive curriculum schedule. Across three Amazon datasets and two LLM backbones, SOFT reports an average accuracy gain of 37.59% over the corresponding supervised fine-tuning baselines. The paper confines this claim to sequential recommendation over named items with LoRA-based fine-tuning, as stated in its Limitations section.","feed_headline":"Self-distilled curriculum lifts LLM recommendation accuracy 37.59%","feed_subtitle":"Models first learn from their own easy outputs, then shift to real user data, beating SFT across three Amazon datasets.","key_machinery":"The load-bearing objects are the self-distilled auxiliary dataset $\\hat{\\mathcal{D}}$ and the self-adaptive curriculum scheduler $\\tau$. Self-distillation constructs $\\hat{\\mathcal{D}}$ by pairing each prompt with the output of the already fine-tuned LLM, producing data that lies within the model's own distribution and carries the coarse category-level recommendation knowledge the model has managed to learn. The scheduler $\\tau=e^{\\alpha(d_t/d_0-1)}$ balances the distilled-data loss against the real-data loss, with $d_t$ the average L2 distance in token-embedding space between generated outputs and true target items; as training proceeds and $d_t$ falls, weight shifts from easy self-generated data to hard real data. Together these two components turn supervised fine-tuning into a two-stage curriculum: first exploit the model's own easy knowledge space, then tune it into the external hard knowledge space.","core_discovery":"The paper's central claim is that the knowledge gap between an LLM and a recommendation task is bridged by a curriculum built from the model's own outputs. After supervised fine-tuning, the LLM generates a predicted next item for each training prompt; these predictions are seldom the exact target item (Hit Ratio@1 below 1%), but they land in the correct item category about 44.6% of the time on average, and the first-epoch loss on the self-distilled data is only 47% of the loss on the real data. SOFT therefore treats the self-distilled set as the easy stage and mixes it with the real dataset through $\\mathcal{L}_{SOFT}=(1-\\tau)\\mathcal{L}_{SFT}+\\tau\\mathcal{L}_{SDFT}$, where the scheduler $\\tau=e^{\\alpha(d_t/d_0-1)}$ drops automatically as the embedding-space distance between the model's generated outputs and the target items shrinks. The reported result is that this easy-to-hard progression improves both BIGRec and LLaRA on Video Games, Movies and TV, and Kindle Store, with a 37.59% average improvement over SFT, and that the full method beats both training on the self-distilled data alone and training on the real data alone.","pith_inferences":["Beyond the paper: if the 44.6% category-level accuracy is the real carrier of the curriculum, SOFT should improve category-level hit rates earlier and more strongly than item-level hit rates; the reported metrics do not separate these, so this is a testable prediction rather than a paper claim.","Beyond the paper: the same loss-mixing recipe could transfer to other LLM fine-tuning tasks with a large distribution gap, such as structured or domain-specific generation, where a model's own coarse outputs could serve as an easy scaffold; the paper does not claim this.","Beyond the paper: running SOFT with a control self-distilled dataset whose predicted item names are replaced by random same-category names would isolate whether the transfer comes from the easy-to-learn distribution or from the semantic category content, since the paper's evidence correlates but does not separate these two factors.","Beyond the paper: the paper's observation that SDPO underperforms SFT in this all-ranking setting suggests the curriculum may also be regularizing token distributions; combining SOFT with preference optimization is an unstated extension."],"forward_implications":["Because SOFT acts only at the loss level, any SFT-trained LLM recommender can adopt it without changing prompts, item vocabularies, or the grounding step.","Training on the self-distilled dataset alone already beats SFT on most settings, indicating that the model's own outputs filter out the hardest real-data knowledge and provide a useful warm start.","The full SOFT combination outperforms both guidance-only and tuning-only ablations, locating the benefit in the curriculum's mixing rather than in either loss term by itself.","Since LLM training converges within a few epochs, the exact functional form of the scheduler decay matters little; the advantage comes from starting the descent in a better region of parameter space, not from the shape of the $\\tau$ curve.","The gains replicate across two backbones and three datasets, suggesting the loss-level curriculum transfers better than adding ID-based item embeddings to the input."],"supporting_citations":[{"why":"Supplies the BIGRec backbone and the item-grounding step that maps LLM-generated names to real items via L2 embedding distance, which SOFT reuses to measure learning progress.","marker":"(Bao et al., 2023)"},{"why":"Supplies the LLaRA backbone and hybrid prompting baseline, plus the dataset and training protocol SOFT is evaluated against.","marker":"(Liao et al., 2024)"},{"why":"Establishes curriculum learning as a training-order principle, the conceptual basis for progressing from self-distilled easy data to real hard data.","marker":"(Bengio et al., 2009)"},{"why":"Shows self-distillation can bridge the distribution gap in language-model fine-tuning, the direct precedent for using model-generated outputs as auxiliary training data.","marker":"(Yang et al., 2024b)"},{"why":"Provides the Llama3.2-3B backbone used in all LLM-based experiments, so the reported gains are tied to this model family.","marker":"(Grattafiori et al., 2024)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method whose frozen-base setup is the setting in which SOFT is tested.","marker":"(Hu et al., 2021)"}],"fun_headline_variants":["Curriculum from self-distilled data boosts LLM recs 37.59%","LLM recs learn easy first, then hard: +37.59% accuracy","Self-distilled curriculum lifts LLM recommender accuracy","SOFT: self-optimized fine-tuning boosts LLM recs 37.59%","Easy-to-hard self-training sharpens LLM recommendation accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the self-distilled data, which consists of the fine-tuned model's own often-wrong predicted item names, is easy to learn and carries the category-level recommendation knowledge that transfers to later learning on real data, rather than merely reinforcing hallucinations or adding extra training iterations.","fun_headline_variants_meta":{"raw":{"variants":["Curriculum from self-distilled data boosts LLM recs 37.59%","LLM recs learn easy first, then hard: +37.59% accuracy","Self-distilled curriculum lifts LLM recommender accuracy","SOFT: self-optimized fine-tuning boosts LLM recs 37.59%","Easy-to-hard self-training sharpens LLM recommendation accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3608,"prompt_tokens":1083,"completion_tokens":2525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":2426}},"tokens_in":699,"tokens_out":2525,"duration_ms":18514,"temperature":1.0,"reasoning_tokens":2426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:47:43.278224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SOFT against a control in which each self-distilled item name is replaced by a random item from the same category; if the control reproduces the gains, the easy distribution, not the semantic content, is what transfers. Separately, give standard SFT on real data enough additional epochs to match SOFT's total gradient updates and check whether the reported 37.59% gap closes to zero.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLaRA backbone and hybrid prompting baseline, plus the dataset and training protocol SOFT is evaluated against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes curriculum learning as a training-order principle, the conceptual basis for progressing from self-distilled easy data to real hard data."}],"review_version":1}