{"id":"93fb3eed-6d4f-4723-8f0b-f274e4b0e1a4","arxiv_id":"2502.15685","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selecting about 500 sessions with an active-learning policy lets a small recommender learn from an LLM teacher better than prior distillation baselines on two datasets.","lead":"This paper presents ALKDRec, a method that calls an LLM as a recommendation teacher on only a few hundred carefully chosen sessions, then transfers that knowledge into a small session-based recommender. The practical payoff is LLM-quality ranking at a fraction of the compute and cost, which matters for on-device recommendation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Max-min guarantee relies on true effective/similar/incorrect counts, but experiments grid-search these as a fixed 1:5:4 ratio; if true counts differ, Theorem 3 gives no lower bound for ALKDRec.","rationale":"The reader's weakest assumption already identifies the count-dependence, and I agree that it is the load-bearing point because it attacks the exact mechanism claimed. Equation (3) is solved over Δ(kef, ksi, kin), and the proof of Theorem 3 uses both lower and upper bounds to conclude that the max-min value equals Γ(k*) only for that Δ. The implementation replaces true counts with a validation-tuned ratio, so the theoretical result is conditional on an unverified property of each new dataset. This matters more than the fixable proof typos or the absence of error bars, because if the 1:5:4 ratio is wrong on a real deployment, the method reduces to a heuristic that may be no better than random selection. The empirical ablation in RQ3 uses the same tuned-ratio policy for all variants, so it cannot rule out this possibility. A direct measurement of counts on an unseen dataset, followed by a comparison between the true-count and fixed-ratio policies, would settle the issue. If the fixed-ratio policy is robust, the paper's claim can stand; otherwise the central proposition needs qualification. Since this is a fixable empirical-validation gap rather than a demonstrated failure, the current CONDITIONAL verdict remains appropriate.","tokens_in":20004,"tokens_out":11800,"duration_ms":116010,"concrete_test":"On a held-out dataset not used for tuning, compute the true counts kef, ksi, kin by querying GPT-4-turbo on all training sessions (or on a random subset of at least 1,000 sessions) and classifying each prediction as effective, similar, or incorrect under the paper's own definitions (ground-truth rank and agreement with the student and teacher recommenders). Then run three variants of ALKDRec: (a) the published fixed 1:5:4 policy, (b) the policy from Equation (3) using the measured counts, and (c) random sampling, with the same τ and backbone. If (a) and (b) perform similarly and both beat random, the count assumption is not the source of the gain; if (b) substantially outperforms (a), or if (a) is no better than random on the new data, then the reported advantage depends on the 1:5:4 grid-search and the theoretical guarantee is not what delivers the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (3) and Theorem 3 define the active policy p* as the max-min solution over the set Δ(kef, ksi, kin) of all label assignments with the true totals kef, ksi, kin. The lower-bound proof (Theorem 1) and the equilibrium argument (Theorem 3) both require that these totals describe the actual data. In Implementation Details, the authors do not measure these totals; they state 'We set the number of effective/similar/incorrect instances as 1:5:4 for all datasets by the grid search.' Thus p* is computed for a different Δ than the one containing the true assignment. If the actual ratio on a test set is not 1:5:4, Γ(k*) is not a lower bound on the realized expected gain, and the claim that ALKDRec 'selects as informative as possible instances while avoiding incorrect or similar predictions' has no theoretical support for that deployment. The empirical comparison in RQ3 is not a substitute: Random, Easiest, Hardest, and RAD-BC are evaluated with the same 1:5:4 count assumption, so the comparison cannot detect whether the source of the gain is the max-min principle or the validation-set tuning of the three counts. These counts are load-bearing because they set k*, the support of p*, and every probability mass in Definition 1. This is an internal-validity concern, not a disagreement with a baseline choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ALKDRec, an active knowledge distillation method for session-based recommendation. A conventional teacher recommender is first trained on all sessions; its predictions are summarized by an LLM, which is then prompted to produce ranking lists for candidate items. To limit LLM inference cost, only a subset of sessions is selected for distillation: the selection policy is obtained by maximizing the minimum expected gain, where gains depend on an instance's difficulty and on whether the LLM prediction is effective, similar, or incorrect. The authors prove (Theorems 1-3) a max-min characterization of the optimal policy, and they report experiments on Hetrec2011-ML and Amazon-Games with FPMC, STAMP, and AttMix backbones, showing that ALKDRec improves over KD baselines in most configurations and is much cheaper than running the LLM on all sessions.","tokens_in":20301,"tokens_out":7621,"duration_ms":66480,"significance":"If the theoretical guarantee were intact, the paper would make a useful contribution: it extends robust active distillation from binary classification to ranking-oriented session-based recommendation with three outcome types, and it shows a practical path for a lightweight student to absorb LLM knowledge from a few hundred sessions. The empirical package is a genuine strength: real-world datasets, three backbones, statistical significance tests, efficiency and cost numbers, and ablations against Random, Easiest, Hardest, and RAD-BC. The main reservation is that the max-min guarantee is invoked under preconditions that the implementation does not verify, and the proof sketches contain enough index and notation errors that the claimed result cannot be checked as written.","major_comments":[{"comment":"The max-min problem in Eq. (3) and Theorems 1-3 require the true totals kef, ksi, and kin of effective, similar, and incorrect instances. The text states that only these totals are assumed known, but the implementation does not measure them: 'We set the number of effective/similar/incorrect instances as 1:5:4 for all datasets by the grid search.' The resulting policy p* is therefore optimal for a different assignment set than the one containing the actual data, and Theorem 3 gives no lower bound on the realized expected gain when the true ratio differs from 1:5:4. Since this is the load-bearing justification for the claim that ALKDRec 'can select effective instances,' the authors should either measure these counts on validation data and report their stability, formulate the theorem over a family of possible counts, or explicitly reposition the max-min solution as a heuristic whose value is empirical.","section":"Active learning for LLM-based distillation, Eq. (3); Implementation Details"},{"comment":"The proof sketches contain errors that prevent verification of the central claim. In Definition 1, the second branch of H_k sums from s=1 to k instead of from k_si+k_in+1 to k, in both the main text and Appendix B. In Theorem 1's proof, the minimization step says we should set c^ef_s=1 for s >= k* because p*_s=0, but p*_s=0 only for s > k*. In Definition 2, the denominators for hat c^ef and hat c^si appear swapped relative to the subsequent algebra in the proof of Theorem 2: the first branch of hat c^si is defined with g^ef_s+g^in_s, yet the proof substitutes (Γ(k*)+g^in_s)/(g^si_s+g^in_s) for that branch. These are not merely typographical: they obscure whether the lower-bound and upper-bound arguments are valid. The manuscript needs a complete, consistently labeled proof (or a correctly typeset appendix) for the max-min theorem before the theoretical contribution can be accepted.","section":"Definition 1, Definition 2, and Proofs of Theorems 1-2"},{"comment":"The gain values used in practice are set by hand: g^ef_s = 1/[rank(d^f_s)]^mu with mu=10, g^si_s = g^in_s = g^ef_s/2, and the three counts are grid-searched to 1:5:4 on the same validation sets used to select hyperparameters. Because the theory optimizes the expected value with respect to these user-supplied gains, the theorem does not by itself establish that the selected subset maximizes actual knowledge-distillation utility; it only establishes robustness relative to the chosen gain model. The empirical comparison in RQ3 cannot fully separate the benefit of the max-min principle from the benefit of this validation-set tuning, since the Random, Easiest, Hardest, and RAD-BC baselines are all evaluated under the same hand-set gains. I would like to see a sensitivity analysis over the gain ratio and the count ratio (beyond Figure 3) and, ideally, a calibration check showing that the chosen ratios approximate the true counts on a held-out part of the data.","section":"Implementation Details; Figure 3; Ablation Study (RQ3)"}],"minor_comments":[{"comment":"The sentence ending '...with k* shown in Theorem 2, the detailed .' is incomplete; it should refer to the appendix or be finished.","section":"Before Algorithm 1"},{"comment":"In Definition 1, the notation H_k in the second branch should make the summation range explicit; the current 'Xk_{s=1}' is ambiguous and is repeated in Algorithm 1.","section":"Definition 1 and Algorithm 1"},{"comment":"The reported 'Improve' values for AttMix on Hetrec2011-ML include one negative value (recall@5, -6.25%) but the text says ALKDRec 'significantly outperforms ... in most cases'; this is acceptable, but the negative case should be acknowledged in the discussion.","section":"Table 1"},{"comment":"The efficiency claim ('around 44 minutes and 8.6 USD' vs '1782 minutes and 347.0 USD') is useful but the exact inference setup, including the number of LLM calls for summarization versus ranking, should be reported in a table for reproducibility.","section":"Experiments and Analysis"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable: the empirical claim is real, and the max-min framework is a reasonable extension of Baykal et al. if the counts and gains are treated as calibrated parameters rather than as known quantities. The main risk is that reviewers will not be able to verify the theorems as written; the proof appendix needs a full rewrite. If the authors cannot obtain or estimate the true counts, they should soften the theoretical claim and market the method as an empirically motivated selection heuristic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a genuinely new idea: it extends robust active distillation (Baykal et al. 2023) from binary classification to ranking for LLM-based session recommendation, modeling three types of LLM predictions (effective, similar, incorrect) and selecting a small subset of sessions to query the LLM teacher. The experiments are real, on two datasets and three backbones, and show consistent gains over existing KD baselines with only ~500 LLM calls. If that holds up, it's a useful contribution for on-device SBR.\n\nThe soft spot is the theory. The max-min guarantee in Theorem 3 depends on knowing the true counts kef, ksi, kin of effective/similar/incorrect instances. The implementation doesn't measure them; it grid-searches the ratio 1:5:4 on the validation set. So the p* being used is not the p* for which the theorem gives a lower bound. The empirical comparison with Random/Easiest/Hardest/RAD-BC uses the same fitted counts, so it can't distinguish the max-min principle from the validation tuning. That's an internal-validity issue, and it should be fixed before publication.\n\nThere are other issues: the gain mapping is ad hoc (mu=10, gsi=gin=gef/2), no code or data are provided, no error bars (they say five runs but don't report variance), and the claim that ALKDRec 'outperforms the teacher recommender with a 10X model size' is overstated—on Amazon-Games it's worse than the teacher for every backbone. The proofs also have typos and one incomplete sentence; that's minor but should be cleaned up.\n\nWho is this for? Anyone working on LLM-based knowledge distillation or efficient session-based recommendation. It deserves serious referee time, but the authors need to release code/data, correct the derivation or clearly weaken the theoretical claim, and report variance. If the empirical gains survive with a properly specified selection policy, this will be a solid contribution.\n\nMy vote: accept for peer review, expect major revision.","headline":"New three-way active distillation idea for LLM-based SBR, but the theoretical guarantee doesn't match the implemented policy.","tokens_in":20850,"tokens_out":2982,"would_cite":true,"duration_ms":25813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"500 hand-picked sessions can beat full LLM distillation","keywords":["session-based recommendation","knowledge distillation","large language models","active learning","max-min optimization","ranking distillation","student-teacher learning","sustainable AI"],"falsifier":"On a new dataset with ground-truth session labels, label each LLM prediction as effective, similar, or incorrect, compute $p^*$ from the true counts, and compare it against the grid-searched 1:5:4 policy; if the true-count policy does not outperform the grid-searched one, or if the grid-searched policy falls below random selection as the true mix diverges from 1:5:4, the central claim is weakened.","tokens_in":19798,"feed_emoji":"🎯","tokens_out":9761,"duration_ms":79509,"temperature":0.7,"pith_summary":"Session-based recommenders that distill knowledge from large language models normally ask the LLM to score every training session, which is expensive and noisy. The paper claims that a small student model can instead learn from a few hundred sessions picked by an active-learning rule that maximizes the minimal expected gain. The rule sorts sessions by three possible outcomes of an LLM prediction—effective, similar, or incorrect—and weights the sampling so that even the worst-case mix of these outcomes yields a guaranteed positive distillation gain. On two real-world datasets the method improves over prior knowledge-distillation baselines across three backbone recommenders while querying the LLM only about 500 times. If true, this makes LLM-based distillation practical for on-device session-based recommendation at a fraction of the API cost.","feed_headline":"500 hand-picked sessions can beat full LLM distillation","feed_subtitle":"Trained on only 500 sessions, the small student beats prior KD baselines and a 10x larger teacher.","key_machinery":"The load-bearing object is the max-min equilibrium distribution $p^*$: each session's sampling probability is proportional to $1/(g^{ef}_s + g^{in}_s)$ or $1/(g^{si}_s + g^{in}_s)$ depending on its rank, with a cutoff $k^*$ that maximizes the closed-form value $\\Gamma(k^*)$. Difficulty is measured by the negative consistency between the session embedding and item embeddings in a conventional teacher, and the gains for effective, similar, and incorrect predictions are derived from that difficulty. This machinery converts an intractable optimization over unknown instance labels into a solvable max-min problem, which is what carries the argument.","core_discovery":"The paper's central claim is that the active selection policy $p^*$ obtained by maximizing the minimal expected gain selects instances that are as informative as possible while avoiding incorrect or similar LLM predictions. The paper models distillation gain as a max-min game: the selector chooses a sampling distribution over sessions, and an adversary decides which sessions are effective, similar, or incorrect subject only to overall counts; Theorems 1--3 locate the equilibrium and give the closed-form value $\\Gamma(k^*)$. It then shows empirically that this policy, implemented as ALKDRec with a GPT-4-turbo teacher and only 500--750 predicted sessions, beats all tested KD baselines on Hetrec2011-ML and Amazon-Games, and in several configurations outperforms a teacher recommender with ten times the latent dimension.","pith_inferences":["An extension the paper leaves implicit: the 1:5:4 ratio is a hyperparameter, so a production system could estimate the three counts on a small probe set and recompute $p^*$, which would preserve the theoretical guarantee on new data.","The same three-outcome max-min decomposition should transfer to distilling from any expensive teacher in ranking tasks, such as vision-language models or ensembles, not only LLM recommenders.","A testable prediction of the theory is that the optimal subset size grows with the fraction of effective instances and shrinks as noise rises; the paper's single-dataset study of subset size (peak near 500) is consistent with but does not prove that trade-off.","The paper's stated future work—embedding-level distillation—suggests the active policy could also select sessions for hidden-state alignment; selecting for both ranking and embedding gains might close more of the gap to the LLM teacher."],"forward_implications":["Querying the LLM teacher for about 500 sessions instead of all sessions cuts the reported API cost from roughly 347 USD to under 9 USD and wall-clock time from about 30 hours to 44 minutes on Amazon-Games.","A student with latent dimension 10 can match or beat a teacher with latent dimension 100 after distillation, which brings LLM-level ranking quality to devices with small memory budgets.","Because the policy samples from the whole gain-ranked list rather than picking only the hardest or easiest sessions, it avoids both redundant and misleading training signals that degrade existing KD baselines.","The active-learning module is backbone-agnostic and improves FPMC, STAMP, and AttMix students, so the recipe can be composed with other session-based recommenders.","The max-min formulation needs only aggregate counts of the three prediction types, not per-instance labels, so the method does not have to know which LLM predictions are correct in advance."],"supporting_citations":[{"why":"Supplies the max-min robust active distillation formulation and the RAD-BC baseline that the paper generalizes from binary classification to ranking.","marker":"Baykal et al. 2023"},{"why":"DLLM2Rec is the strongest LLM-based KD baseline; the paper's comparison motivates the active selection module.","marker":"Cui et al. 2024"},{"why":"DE establishes the knowledge-distillation framework and teacher/student settings that the experiments follow.","marker":"Kang et al. 2020"},{"why":"FTD and HTD are topology-distillation baselines that ALKDRec must beat for the central claim.","marker":"Kang et al. 2021"},{"why":"Ranking distillation is the base technique that LLM-based KD extends to a language-model teacher.","marker":"Tang and Wang 2018"},{"why":"On-device self-supervised distillation motivates the efficiency requirement that ALKDRec targets.","marker":"Xia et al. 2022"}],"fun_headline_variants":["Active learning picks 500 sessions that beat full LLM distillation","Max-min game: select 500 sessions to outperform 10x teacher","500 smartly chosen sessions outperform larger teachers in SBR","Active distillation: fewer, smarter sessions win in recommendation","Pick 500 sessions strategically: beat full distillation and 10x teacher"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee holds only if the total numbers of effective, similar, and incorrect LLM predictions are known in advance; the experiments do not measure these counts but fix a 1:5:4 ratio by grid search, so a dataset with a different mix could weaken the guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Active learning picks 500 sessions that beat full LLM distillation","Max-min game: select 500 sessions to outperform 10x teacher","500 smartly chosen sessions outperform larger teachers in SBR","Active distillation: fewer, smarter sessions win in recommendation","Pick 500 sessions strategically: beat full distillation and 10x teacher"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3327,"prompt_tokens":978,"completion_tokens":2349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2262}},"tokens_in":594,"tokens_out":2349,"duration_ms":14563,"temperature":1.0,"reasoning_tokens":2262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:18:15.596181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a new dataset with ground-truth session labels, label each LLM prediction as effective, similar, or incorrect, compute $p^*$ from the true counts, and compare it against the grid-searched 1:5:4 policy; if the true-count policy does not outperform the grid-searched one, or if the grid-searched policy falls below random selection as the true mix diverges from 1:5:4, the central claim is weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the max-min robust active distillation formulation and the RAD-BC baseline that the paper generalizes from binary classification to ranking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DE establishes the knowledge-distillation framework and teacher/student settings that the experiments follow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FTD and HTD are topology-distillation baselines that ALKDRec must beat for the central claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ranking distillation is the base technique that LLM-based KD extends to a language-model teacher."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"On-device self-supervised distillation motivates the efficiency requirement that ALKDRec targets."}],"review_version":1}