{"id":"bb03470e-a3a4-4df7-8551-dba379bd6550","arxiv_id":"2608.07055","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A full-attention teacher distills knowledge to token-merging students, letting a recommender use 20K-token behavior sequences at near-baseline serving cost.","lead":"ByteDance reports a teacher-student design for ad recommendation that trains once on full 20,000-token user behavior sequences, then serves lighter students that merge tokens to cut compute. The claimed payoff is a +1.036% advertiser score gain with only +5.6% serving latency, making longer history affordable in industrial ranking.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '~85% recovery' claim rests on two tiny AUC deltas (+0.26% vs +0.22%) measured without confidence intervals and under different teacher/student batch sizes; this ratio is not statistically grounded.","rationale":"Good-faith reading: this is an industrial systems paper whose main evidence is a deployed A/B test with a claimed +1.036% ADSS gain, plus extensive offline ablations. I do not see an internal contradiction that would justify rejection. The attention analysis in Sec. 3.3 is indeed thin (m=5 queries, first head only), but the merge strategies are tested directly in Table 3; if the motivation were badly unrepresentative, the ablations would probably show larger degradation than the reported 0.02-0.06% per strategy. The most load-bearing quantity is the recovery ratio, because it is the paper's quantitative reason to believe the student is practical: 0.22/0.26. That ratio is fragile for the numerical reasons above. The reader's designated weakest assumption was the attention sample; I partially agree but think the bigger soft spot is the uncharacterized uncertainty and the teacher/student training mismatch. Absence of code/data also prevents independent verification, but I do not treat that alone as a technical flaw. The appropriate disposition remains CONDITIONAL: the central claim is plausible, but the '~85% recovery' phrasing should be conditioned on confidence intervals or a matched teacher-training run.","tokens_in":16802,"tokens_out":13462,"duration_ms":121085,"concrete_test":"Re-analyze the offline evaluation data behind Table 2 by bootstrapping over evaluation dates (or users) to obtain 95% confidence intervals for the teacher's ΔAUC, the student's ΔAUC, and their ratio. If the ratio's CI does not contain roughly 85%, or if the student's CI overlaps the teacher's CI, the paper should soften the recovery claim. As a complementary check, retrain the teacher with batch size 320 via gradient accumulation on the same data schedule and compare ΔAUC; if the teacher gain shifts materially, the 85% denominator is configuration-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim that the student recovers ~85% of the teacher's 20K gain (Sec. 5.2, Table 2) is computed as 0.22/0.26 from ΔAUC values relative to a 5K baseline. Three issues make this ratio load-bearing rather than decorative. First, the two deltas differ by only 0.04 percentage points, so without confidence intervals or repeated-seed variance the ratio could easily move from ~85% to near parity or above 100% under plausible run-to-run noise. Second, the teacher is trained with a batch size of 96 while the student uses 320 (Sec. 5.1.3), and the teacher is described as 'heavily trained'; the denominator is therefore not an apples-to-apples estimate of the maximum gain achievable under the student's optimization protocol, so the recovery fraction is not a clean measure of what compression preserves. Third, although the online A/B section asserts statistical significance, no test details are given; the only quantitative support for the 'student nearly matches teacher' narrative is the uncharacterized offline ratio. The paper's attention-statistics motivation (Sec. 3.3) is a secondary concern, because Table 3 directly tests the merge strategies and shows modest AUC losses; the unreported variance on those very deltas is what actually controls the headline recovery figure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TM20K, a two-stage knowledge-distillation framework for ultra-long e-commerce behavior sequences (up to 20K raw tokens) in a deployed advertising recommender system. The teacher model is a full-attention transformer trained once on uncompressed 20K sequences; the student model applies three attention-motivated token merge strategies (LITM, PATM, LPTM) to compress sequences for efficient training and serving, and is distilled from the teacher's cached logits. Offline experiments on a proprietary industrial dataset compare TM20K with several long-sequence baselines, report throughput/memory numbers, and include ablations of each merge strategy and of distillation weights. An online A/B test shows ADSS +1.036% and ADVV +0.780% with serving latency +5.6% relative to a deployed 5K baseline. The paper's headline quantitative claim is that the distilled student recovers about 85% of the teacher's AUC gain, computed as +0.22%/+0.26%.","tokens_in":17047,"tokens_out":8712,"duration_ms":73293,"significance":"If the reported results hold, the paper provides a practically valuable industrial solution: it demonstrates that a one-time full-attention teacher can transfer long-sequence knowledge to a token-merging student, making 20K-scale sequence modeling feasible online with modest overhead. The three merge strategies are simple, interpretable, and directly tied to observed attention statistics, and the paper includes transparent efficiency measurements, a broad ablation study, and real deployment evidence. The main weakness is statistical: the central recovery ratio and most offline deltas are single runs with no confidence intervals or significance tests, and the teacher and student are trained under different optimization budgets, so the quantitative strength of the claims is currently overstated. The online A/B gains are encouraging but lack test details. Overall, this is a useful applied contribution with a defensible architecture, but the evidence base needs tightening before the central claims can be accepted at face value.","major_comments":[{"comment":"The paper's headline 'recovering around 85% of the teacher's total performance improvement' is computed as 0.22/0.26 from single-run ΔAUC values that differ by only 0.04 percentage points, yet no confidence intervals, standard errors, or repeated-seed results are reported for any offline metric in Tables 2-6. Given that several ablation deltas in Tables 3-6 are of the same 0.01-0.06% magnitude, run-to-run variance could easily move the ratio from 85% to near parity or above 100%. Please report multiple seeds or a significance test for the key comparisons, especially the 5K baseline, teacher, and student-with-KD rows of Table 2.","section":"Table 2 / Sec. 5.2"},{"comment":"The teacher is trained with batch size 96 while the student uses batch size 320, and the abstract describes the teacher as 'heavily trained,' but the paper does not specify training steps, epochs, or data repetition for either model. The teacher's +0.26% AUC gain is therefore not a clean measure of what is preserved by full tokens versus token merge; part of the gap may reflect a larger training budget. Please report the exact optimization budget for both models, or include a control where the teacher is trained with the student's batch size and protocol.","section":"Sec. 5.1.3 / Sec. 5.2"},{"comment":"The online A/B section states that 'TM20K yields statistically significant positive gains across all business metrics' but gives no test details, confidence intervals, or evaluation window. Since the ADSS +1.036% and ADVV +0.780% numbers carry the paper's practical claim, please specify the statistical test, number of days, sample sizes, and confidence intervals for the reported deltas.","section":"Sec. 5.4"},{"comment":"The attention analysis that motivates the three token merge strategies uses an unstated number of training instances, m=5 query tokens, and only the first attention head. The paper should state the instance count and, ideally, verify that the observed patterns (same-ID attention closeness, recency concentration, layer-wise entropy differences) hold across heads and layers; alternatively, temper the 'well-motivated' characterization, since Table 3 is the direct empirical test of the merge strategies.","section":"Sec. 3.3"},{"comment":"All deltas are relative to the deployed 5K baseline, which itself uses LITM0 and UTM2 compression and has an average sequence length of 1.2K. Table 3 shows that a full 5K sequence outperforms this compressed baseline by 0.05% AUC, so the teacher's +0.26% conflates the benefit of extending to 20K with the benefit of removing the 5K compression. To support the interpretation that TM20K extends the sequence length usefully, report deltas relative to a full 5K model or explicitly decompose the length and compression effects.","section":"Sec. 5.2 / Table 2"}],"minor_comments":[{"comment":"Equation (10) uses q_T both for the teacher's logit and for its sigmoid-transformed probability; please distinguish the logit (e.g., z_T) from the probability to avoid notational collision.","section":"Eq. (10)"},{"comment":"The caption 'AUC gains relative to the TM20K-S model with different distillation weights' is ambiguous; clarify whether the reference is the student without KD or the best lambda configuration.","section":"Table 5 caption"},{"comment":"The row 'w/o KD in Late Period' does not define what 'Late Period' means; specify the training stage at which the distillation loss is dropped.","section":"Sec. 5.3.4"},{"comment":"State the position-ordering convention once (smaller index = more recent) and keep it consistent in the figure, the text, and the algorithm descriptions.","section":"Fig. 3(b) / Sec. 4.2.2"},{"comment":"The abstract's phrase 'nearly the same' training cost should acknowledge the one-time teacher training cost; the main text does this, but the abstract could be more precise.","section":"Abstract"},{"comment":"Equation (9) merges adjacent token pairs via reshape to (L_n/2, 2, d), which requires L_n to be even; specify how odd lengths are handled, especially after LITM/PATM produce variable-length sequences.","section":"Sec. 4.2.3 / Eq. (9)"},{"comment":"The exclusion of DIN and TWIN results is justified qualitatively; consider reporting a single performance number to substantiate the claim of 'obvious performance drops.'","section":"Sec. 5.1.2"},{"comment":"Reference [14] lists 'Ads Recommendation' as the author, which appears to be a venue or organization name rather than a person; please check the bibliographic entry.","section":"Reference [14]"}],"recommendation":"major_revision","confidential_remarks":"This is a solid industrial systems contribution with a real deployment, and the online A/B results are potentially valuable if properly characterized. My main reservation is statistical: the headline '85% recovery' figure rests on single-run AUC deltas without confidence intervals, and the teacher/student training budgets differ, making the recovery denominator difficult to interpret. The attention-motivation analysis in Sec. 3.3 uses a very small sample (m=5, one head, unstated instance count), though the ablation study in Table 3 mitigates this. The paper cites several works from the same group (LONGER, Rec-Distill, RankMixer) as baselines or components; this is not a problem per se, but the novelty relative to that line of work should be articulated more sharply. Given the proprietary data and absence of reproducibility artifacts, the paper is better suited to an applied venue (e.g., RecSys or KDD applied track) than to a general IR journal; the current form would benefit from a revision that adds statistical rigor before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: this is a well-built industrial sequencing paper, not a deep research contribution, but it delivers a deployable recipe—three simple token-merge rules plus a full-attention teacher with cached-logit KD—that extends e-commerce sequence length from 5K to 20K at roughly 5% serving latency cost. The online A/B gains (+1.036% ADSS) are the real headline and carry more weight than any offline number.\n\nThe genuinely new pieces are the three merge strategies: LITM merges by product ID in a local window, PATM applies position-dependent compression factors, and LPTM halves the sequence every few layers. LONGER does uniform merge; these are adaptive variants with a clear motivation. The teacher-student split, where the teacher is trained once on uncompressed tokens and only its logits are cached, is a sensible practical application of knowledge distillation. The paper does a lot right: it shows the FA-vs-TA comparison, gives ablations for each merge strategy, reports hyperparameter sensitivity, and ends with an honest limitations paragraph admitting the rules are hand-tuned and need retuning per scene. That transparency earns credit.\n\nThe soft spots are real but mostly statistical, not design-level. The 'recovers ~85% of the teacher's gain' line is computed from two ΔAUC deltas, +0.26% and +0.22%, with no confidence intervals or seed variance. A 0.04pp difference does not support a precise recovery fraction. Worse, the teacher is trained with batch size 96 versus the student's 320, so the denominator is not the maximum gain achievable under the student's optimization protocol. The stress-test note is right: that ratio is load-bearing but statistically ungrounded. The attention-analysis motivation in Sec. 3.3 is weaker—5 query tokens, first head only—but Table 3 tests the merges directly, so the motivation is not what the punchline rests on. All offline AUC deltas are small, and without interval estimates you can't tell which differences are real. The online test is asserted to be statistically significant, but no test details are given.\n\nThis paper is for practitioners in industrial recommender systems and for researchers curious about what works in production. It deserves a serious referee: the method is coherent, the evaluation is extensive even if under-reported, and the deployment evidence matters. I'd send it to review rather than desk-reject, and I'd ask the authors to supply variance estimates for the offline deltas and a description of the online significance test. It's an accept-with-revisions candidate.","headline":"Well-engineered industrial sequence modeling worth refereeing, but the offline recovery ratio is statistically fragile.","tokens_in":17664,"tokens_out":2255,"would_cite":true,"duration_ms":19730,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TM20K shows that extending ad sequences to 20K tokens is practical: a once-trained teacher supervises a token-merging student, lifting an online advertiser metric by 1.036% with only 5.6% higher serving latency.","keywords":["ultra-long sequence modeling","recommender systems","knowledge distillation","token merge","full attention","e-commerce advertising","CVR prediction","sequence compression"],"falsifier":"A concrete check: compute the same attention statistics on the fully trained teacher using all query positions and all heads, then rebuild the student's merge masks from that full distribution. If the resulting student closes most of the remaining 15% gap to the teacher, the paper's rule-based merges are not preserving all the signal they claim; if the performance is comparable, the small-sample statistics were representative.","tokens_in":16548,"feed_emoji":"📈","tokens_out":9664,"duration_ms":77434,"temperature":0.7,"pith_summary":"The paper claims that full-attention transformers over user behavior sequences can be scaled to 20,000 tokens in an industrial ad recommender without wrecking training or serving cost. It does this by training a teacher model once on the complete, uncompressed sequence and storing its prediction logits, then letting a student model serve online after compressing the sequence with three simple token-merge rules. In offline tests the student recovers about 85% of the teacher's AUC gain, and in an online A/B test the system raises advertiser score by 1.036% while serving latency grows only 5.6%. The authors argue this makes ultra-long sequences practical for online serving, where previous approaches either compressed away fine-grained information or used lightweight attention that extracts less signal.","feed_headline":"20K user actions at near-5K cost: advertiser score up 1.036%","feed_subtitle":"A once-trained teacher supervises a student that compresses 20K tokens to ~1.8K, keeping most of the gain.","key_machinery":"The mechanisms are (1) full attention (FA) as the sequence encoder, which the paper shows beats target-only attention by up to 0.25% AUC at 20K and uses sequence-to-sequence interactions that are nearly as informative as target-to-sequence ones; (2) three hierarchical token-merge rules derived from attention statistics: LITM merges consecutive same-product-ID tokens, PATM applies stronger compression to older positions, and LPTM halves the sequence length every two transformer layers; and (3) a two-stage distillation setup where a teacher trained once on full 20K tokens caches its logits, and the student optimizes a main prediction head plus a distillation head with loss $\\lambda\\ell_{\\mathrm{kd}}$ matched in scale to the cross-entropy loss. Together they move the heavy computation into offline training while keeping the online model's input short.","core_discovery":"The central discovery is that the conflict between effectiveness and efficiency for ultra-long sequences can be separated into a one-time training problem and an online serving problem. A teacher transformer trained with full attention on all 20K tokens (with causal masking) achieves the best prediction quality, improving CVR AUC by 0.26% over the 5K baseline. Three rule-based merge strategies—merging repeated product-ID interactions in a short window (LITM), compressing older tokens more aggressively than recent ones (PATM), and halving sequence length every few transformer layers (LPTM)—shrink the student's average sequence from 8.8K to 1.8K tokens. The student alone gains +0.15% AUC, and after distillation from the teacher's cached logits it rises to +0.22%, roughly 85% of the teacher's gain, at a training throughput of 83K compared with the baseline's 88K and a serving latency increase of only 5.6%.","pith_inferences":["If the same attention statistics are re-measured with all heads and full query coverage, the merge rules might change; this would be a direct test of whether the paper's small-sample analysis is representative.","The framework could transfer to non-advertisement domains such as feed or short-video recommendation, but the recency and same-ID priors may differ, so the Sec. 3.3 statistics would need to be recomputed there.","Because the teacher is trained once and its logits are cached, one teacher could serve as the distillation source for many students with varying sequence budgets, turning the one-time cost into an even smaller amortized fraction of deployment.","A learned token-merge policy that imitates the teacher's attention would likely close the remaining 15% performance gap, at the cost of the extra engineering the paper explicitly avoids."],"forward_implications":["Other advertising and recommendation systems can extend behavior sequences to tens of thousands of tokens without expensive online serving changes, if they re-estimate the attention priors on their own data.","A one-time teacher can be reused across several student deployments with different latency budgets, since the teacher's logits are cached and the merge hyperparameters are tuned per student.","Full attention should be preferred over target-only attention for long sequences in this setting, contradicting the implicit assumption in some earlier sequence-compression work that target interactions capture most of the signal.","The 85% recovery means the majority of the long-sequence benefit can be obtained without serving a full 20K transformer, leaving a small residual gap that future learned-merge methods might close."],"supporting_citations":[{"why":"PATM and LPTM generalize this prior uniform token-merge operation, used here as the UTM2 baseline.","marker":"[2]"},{"why":"FlashAttention makes full-attention computation at 20K tokens feasible for the one-time teacher.","marker":"[5]"},{"why":"QK-norm stabilizes student training; removing it causes late-stage divergence.","marker":"[6]"},{"why":"Rec-Distill provides the industrial distillation pipeline that the two-stage teacher–student setup follows.","marker":"[7]"},{"why":"EST is the prior work claiming target attention suffices; the paper's full-attention comparison directly addresses its finding.","marker":"[18]"},{"why":"ExFM inspired the external-teacher / amortized-student framework used for ultra-long sequence scaling.","marker":"[23]"},{"why":"M-Falcon serving imposes the causal-mask requirement and is the serving engine whose latency is compared.","marker":"[35]"},{"why":"RankMixer is the interaction backbone that fuses sequence and non-sequence features into the final prediction head.","marker":"[39]"}],"fun_headline_variants":["Teacher keeps all 20K tokens, student merges to 1.8K: +1.036% ADSS at 5.6% latency","Full attention teacher distills to merging student: 20K tokens, 1.8K effective, +1.036%","20K user actions, teacher full attention, student merges: near-zero overhead, +1.036%","One-time teacher, always-on student: 20K tokens merged to 1.8K, ADSS +1.036%","Full-attention teacher, token-merge student: 20K sequences, 1.8K cost, +0.22% AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The three token-merge rules are based on attention statistics gathered from only five query tokens and the first attention head of a single model; if those statistics do not represent the full multi-head attention distribution, the merges could be discarding signal that the ablations would not reveal.","fun_headline_variants_meta":{"raw":{"variants":["Teacher keeps all 20K tokens, student merges to 1.8K: +1.036% ADSS at 5.6% latency","Full attention teacher distills to merging student: 20K tokens, 1.8K effective, +1.036%","20K user actions, teacher full attention, student merges: near-zero overhead, +1.036%","One-time teacher, always-on student: 20K tokens merged to 1.8K, ADSS +1.036%","Full-attention teacher, token-merge student: 20K sequences, 1.8K cost, +0.22% AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001383,"raw_usage":{"total_tokens":5635,"prompt_tokens":1013,"completion_tokens":4622,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":4461}},"tokens_in":629,"tokens_out":4622,"duration_ms":25901,"temperature":1.0,"reasoning_tokens":4461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:36:30.162408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: compute the same attention statistics on the fully trained teacher using all query positions and all heads, then rebuild the student's merge masks from that full distribution. If the resulting student closes most of the remaining 15% gap to the teacher, the paper's rule-based merges are not preserving all the signal they claim; if the performance is comparable, the small-sample statistics were representative.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QK-norm stabilizes student training; removing it causes late-stage divergence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RankMixer is the interaction backbone that fuses sequence and non-sequence features into the final prediction head."}],"review_version":1}