{"id":"1e18fbf1-562c-4e00-945e-2df64033feb2","arxiv_id":"2608.10709","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SQuaT projects teacher features onto the student's quantization lattice to eliminate the unattainable-target lower bound in label-free QAT distillation.","lead":"A new training method quantizes the teacher network's internal features using the student's own quantization parameters, so a quantized student learns from targets it can actually represent. This makes label-free quantization-aware training more stable and accurate, with the largest gains at 1- and 2-bit precision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is a projection tautology; the real load-bearing assumption is that the student's learned clipping range preserves teacher information, which is untested and possibly violated when teacher features lie outside that range.","rationale":"The Reader's verdict is already CONDITIONAL, and my analysis agrees with the main condition: the student-aware projection must preserve teacher information. I go further by noting that Theorem 1 is a projection tautology and cannot supply the missing guarantee, and that joint optimization of φ_S permits a collapse where the feature loss is trivially satisfied. The experiments are extensive and mostly consistent; Table 7 provides real evidence that Student-aware projection helps relative to Teacher-aware and Non-aware schemes. However, because those comparisons all include logit KD and do not isolate the feature term, the mechanism is not pinned down. A zero-target ablation plus clipping-coverage logging would settle the concern cheaply. I also note Table 11 shows a negative result on W3A3, so 'consistently' overstates the empirical record, but this is secondary. No verdict change is needed; the existing CONDITIONAL should be retained with the proposed ablation as an explicit condition.","tokens_in":1111,"tokens_out":1121,"duration_ms":164837,"concrete_test":"Re-run the W1A1 ResNet-20/CIFAR-10 configuration of Table 1 with three feature-target settings, keeping the logit loss and all hyperparameters fixed except as noted: (A) SQuaT's projected teacher features; (B) a constant zero feature target; (C) no feature term at all, λ_feat = 0. If setting (B) or (C) achieves accuracy within 0.1 percentage points of SQuaT's reported 86.97, then the student-aware projection is not the source of the gains and the central mechanism is unsupported. If (A) beats both (B) and (C) by at least 0.3 percentage points, the projected teacher features carry usable signal and the information-loss concern is weakened. During this run, record the fraction of teacher-feature values clipped by Π_{φ_S} at each distillation layer, at initialization and at convergence; a clipped fraction above 20% confirms that Eq. (6) operates in the discarding regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that SQuaT eliminates an irreducible loss lower bound, but Theorem 1 (Section 3.3) is a projection tautology. For the component-wise uniform quantizer in Eqs. (1)-(3), the nearest point in G = Im(Π_{φ_S}) to f_T is Π_{φ_S}(f_T); this is the definition of a projection. Part (b) holds because the target is chosen to be that projection, so achievability is trivial: any element of G, including a constant, gives infimum zero. The theorem never establishes that Π_{φ_S}(f_T) retains teacher knowledge. The actual load-bearing assumption is that the student's learned quantization range, optimized for the student's activation distribution, is also a good projection range for teacher features. When teacher features lie largely outside [a^-_ℓ, a^+_ℓ], Eq. (6) clips them to the boundary and discards the teacher signal. Figure 1 is only a qualitative distribution plot; no quantitative coverage statistic is reported. Table 7 compares projection schemes but never ablates the feature loss to zero or to a constant target, and all rows include logit KD, so the accuracy gain cannot be attributed to the projection alone. Worse, joint optimization of φ_S with L_feat permits a degenerate solution: the student shrinks its range so that Π_{φ_S}(f_T) collapses toward the student's own output, driving L_feat toward zero without transferring teacher knowledge. Theorem 1 provides no protection against this. The paper's central mechanism therefore rests on an untested assumption, and the reported consistent gains (with one negative result in Table 11, W3A3) do not isolate that mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SQuaT, a label-free quantization-aware training (QAT) method combined with knowledge distillation (KD). The key idea is to quantize teacher intermediate features using the student's own quantization parameters, thereby projecting the teacher's full-precision features onto the student's quantization lattice before applying a feature-level distillation loss. The authors argue that prior teacher-aware projection methods leave an unattainable residual, creating an irreducible lower bound on the distillation loss, and that this bound is removed because SQuaT targets are attainable by construction. The paper proves a theorem to this effect (Theorem 1, Section 3.3) and presents experiments on CIFAR-10, CIFAR-100, ImageNet, Tiny-ImageNet, DeiT-Tiny, and BERT/GLUE, comparing against SQAKD and other baselines, with additional ablations on loss functions, projection schemes, and on-device throughput.","tokens_in":14538,"tokens_out":4368,"duration_ms":46568,"significance":"If the central claim is validated, student-aware projection is a simple and appealing idea that could improve label-free QAT in extreme low-bit regimes. The paper is strong in empirical breadth: it covers CNN and Transformer architectures, vision and NLP, multiple bit widths, several loss functions, and reports error bars and source code availability, which are concrete strengths. However, the theoretical contribution is elementary: Theorem 1 states a projection property that holds by definition, and it does not establish that the projected teacher features retain useful knowledge. The empirical evidence is the main support for the method, but it has gaps (e.g., no feature-loss ablation to zero or constant target, and one hyperparameter selected using test accuracy). The stress-test concern about the student's clipping range discarding teacher signal therefore lands directly and needs to be addressed.","major_comments":[{"comment":"Theorem 1 is a tautology rather than a substantive result. Since G is defined as the image of the student-aware projection and the target is chosen to be the projection of f_T onto G, part (b) holds by construction (select h = Pi(f_T)). The theorem does not address whether Pi(f_T) retains teacher knowledge, and the claimed 'elimination of the lower bound' is a direct consequence of defining the target as the projection. The paper must provide quantitative evidence that teacher features are not largely clipped by the student's learned range [a^-_l, a^+_l]. Please report, per distillation layer and bit width, the fraction of teacher feature components that fall inside the student's clipping range, or an equivalent coverage statistic. Without this, the projection could discard the teacher's signal and the method could still minimize the feature loss.","section":"Section 3.3, Eq. (13)-(14)"},{"comment":"Jointly optimizing the student's quantization parameters phi_S with the feature loss L_feat permits a degenerate solution: the student can shrink its clipping range so that both the student's quantized features and the projected teacher features collapse toward a small set of values, driving L_feat toward zero without transferring teacher knowledge. Theorem 1 does not protect against this collapse. Please add an analysis or experiment showing that the learned ranges remain informative, for example by reporting final a^-/a^+ values relative to teacher feature quantiles, or by evaluating a variant in which phi_S is frozen during feature distillation.","section":"Section 3.2.1, Eq. (7) and Algorithm 1"},{"comment":"The sensitivity analysis selects lambda_feat = 10 based on the best test accuracy on CIFAR-100 with ResNet-32 at 4-bit, and this value is then used for the corresponding headline row in Table 2. This is test-set overfitting. The hyperparameter should be chosen on a held-out validation split, or the paper should demonstrate that the relative ranking of SQuaT versus SQAKD is stable across a range of lambda_feat values.","section":"Section 4.5, Table 8"},{"comment":"The projection-scheme comparison in Table 7 always includes the logit-level KD loss and does not include a baseline with lambda_feat = 0 or with a constant/uninformative feature target. As a result, the accuracy improvement attributed to student-aware projection could be partially or entirely due to the presence of any auxiliary feature loss, or to implicit regularization, rather than to the specific projection mechanism. Please add a no-feature-loss baseline and a constant-target baseline to isolate the effect.","section":"Section 4.4, Table 7"},{"comment":"The claim that SQuaT 'surpasses the corresponding pretrained full-precision models' rests on margins of 0.01 and 0.04 percentage points on CIFAR-10 (Table 1, W4A4), with reported standard deviations of 0.01-0.03. These differences are not convincingly beyond noise. Please either provide significance tests or soften the claim to avoid overstating the result.","section":"Section 4.2, Tables 1-2"}],"minor_comments":[{"comment":"There are typos in table headers: 'R TE' should be 'RTE' in Table 5, and 'Accuarcy' should be 'Accuracy' in Table 11.","section":"Tables 5 and 11"},{"comment":"The learning curves in Figure 3 would be easier to interpret with explicit axis labels, a legend identifying the three projection schemes in each panel, and a note on which bit width each panel corresponds to.","section":"Figure 3"},{"comment":"Algorithm 1 updates phi_S for all layers L, not only the distillation layers K. Please clarify whether quantization parameters outside K are trained during feature alignment, and if so, how this interacts with the logit loss.","section":"Algorithm 1"},{"comment":"The approximation sign in Eq. (5) and the hyperparameter eta are introduced without a precise statement of how they relate to the EWGS method; a short clarifying sentence or a pointer to the EWGS formulation would help.","section":"Section 3.1.2, Eq. (5)"},{"comment":"For the Teacher-aware baseline, the number of calibration epochs and the calibration data used are not specified; adding this detail would make the comparison reproducible.","section":"Section 4.4, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The empirical scope is unusually broad for a short conference paper, and the code release is a plus. However, the theoretical section is thin and overclaims: Theorem 1 is a definitional projection property, not a proof that student-aware targets preserve teacher knowledge. The 'surpasses FP' statements for CIFAR-10 should be toned down or backed by significance tests. The missing ablation with lambda_feat=0 is important for attributing the gains. These issues are fixable with additional experiments and revised framing, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the hook is right and the trick is real. Using the student's own quantization parameters to quantize teacher features is a clean, cheap idea that I haven't seen in the cited prior work, and the paper backs it with a wide run of experiments: CNNs, ViT, BERT, 1-8 bit, multiple loss functions. The gains are consistent, with the expected larger wins at W1A1/W2A2 and on DeiT/BERT, and the authors report error bars and one negative result (Tiny-ImageNet W3A3), which gives me confidence the effect isn't cherry-picked. If I were doing label-free QAT, I'd try this first.\n\nWhere I'd push back: Theorem 1 is not a theorem about knowledge transfer. It says the projection is the nearest point in the attainable set, which is true by definition, and that the distance to the projection is zero because you chose the target to be the projection. That doesn't tell you whether the projected teacher features retain enough signal. The real assumption is that the student's learned clipping range covers the teacher's feature distribution well enough. The paper shows a qualitative distribution plot and compares projection schemes, but it never quantifies coverage, and the Table 7 ablation doesn't include a zero-feature-loss or constant-target control, so part of the gain could come from regularization rather than from the specific projection. The degenerate solution worry—student shrinks its range to zero to trivially match projected teacher features—is theoretically open, though in practice the logit loss seems to keep the student useful; still, the paper doesn't discuss it.\n\nThe lambda_feat=10 choice tuned on test accuracy for a headline cell is a minor sin; it affects one row of Table 2. And the 'surpasses the FP teacher' phrasing leans on 0.01-0.04 pp margins in some cells, which is noise; the larger margin on CIFAR-100 ResNet-32 W4A4 is more convincing.\n\nNet: the paper is honest and the method is useful. The theory should be reframed as 'removes the known-unattainable target' rather than 'eliminates the lower bound', and the coverage question deserves one quantitative experiment. That's revision material, not grounds for rejection.\n\nI'd send it to serious referees. It's a solid contribution to the QAT/KD subfield, and the code is available.","headline":"Simple student-aware projection trick that mostly works; the theory is a tautology, but the empirical results make it a useful label-free QAT method.","tokens_in":15033,"tokens_out":2833,"would_cite":true,"duration_ms":24417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantizing teacher features removes the loss floor in label-free quantization-aware training by projecting them onto the student's own quantization lattice, giving the student targets it can actually reach.","keywords":["Quantization-aware training","Knowledge distillation","Label-free QAT","Student-aware projection","Feature distillation","Low-bit quantization","Self-supervised learning","Neural network compression"],"falsifier":"Train a linear probe on the projected teacher features $\\tilde{f}_{T,\\ell} = \\Pi_{\\phi_{S,\\ell}}(f_{T,\\ell})$ at a distillation layer and compare its label-prediction accuracy with a probe on the raw teacher features $f_{T,\\ell}$; if the projected features are near chance while the raw features are informative, the student-aware projection has discarded the teacher's signal and cannot be the source of the reported accuracy gains.","tokens_in":1715,"feed_emoji":"🎯","tokens_out":4293,"duration_ms":76329,"temperature":0.7,"pith_summary":"The paper claims that when a quantized student is trained by knowledge distillation from a full-precision teacher, using the teacher's raw features as targets creates an irreducible floor on the distillation loss, because the student cannot represent values outside its quantization lattice. SQuaT removes this floor by running teacher features through the student's own quantization function, projecting them onto the discrete lattice the student actually uses. This makes every target attainable, and Theorem 1 proves that the best achievable expected squared distance to such a target is zero. Empirically, label-free quantization-aware training with this student-aware projection consistently beats logit-only and teacher-calibrated baselines, with the largest gains at 1- and 2-bit widths, and is robust across architectures and loss functions.","feed_headline":"Quantizing teacher features removes the loss floor in label-free QAT","feed_subtitle":"Projecting teacher features onto the student's quantization lattice gives targets the student can reach, improving extreme low-bit accuracy.","key_machinery":"The central object is the student-aware projection $\\Pi_{\\phi_{S,\\ell}}(z)$, defined as the student's forward quantization path applied to teacher features: normalize $z$ by the student's learned range $[a^-_\\ell, a^+_\\ell]$, round onto the $b$-bit uniform grid, and apply the type-specific linear mapping. It carries the argument by guaranteeing that the projected teacher feature lies in the set of outputs the student can actually produce, so the squared-error distillation loss over that target can be driven to zero; all empirical gains are attributed to this alignment.","core_discovery":"On its own terms, the central discovery is that the distributional mismatch between full-precision teacher features and low-bit student features creates an unattainable residual. For a fixed student quantization map $\\Pi$, the expected squared distance from any attainable student output to the teacher feature is at least $\\mathbb{E}\\|\\Pi(f_T)-f_T\\|^2$, and this lower bound grows as teacher values fall outside the student's clipping range. SQuaT eliminates the bound by replacing the target $f_T$ with the student-quantized target $\\Pi(f_T)$, which lies in the student's attainable set, making zero expected residual reachable. The paper treats this projection as a paradigm shift from teacher-aware to student-aware projection and validates it across CNN, ViT, and transformer architectures, multiple loss functions, and bit-widths from 1 to 8.","pith_inferences":["The same 'project into the student's representable set' principle should transfer to any distillation target with a constrained output space, such as binary or ternary embeddings, hashed features, or learned embedding tables, not just uniform quantizers.","The theorem bounds the squared-error loss, not task accuracy; the observed gains suggest the loss floor was indeed harmful, but a direct test would be to correlate the measured lower bound with final accuracy across layers and bit-widths.","The method's reliance on the student's learned range could be a liability early in training, when the range is still changing; an inference is that a warm-up or range-regularization schedule could stabilize or further improve SQuaT.","Because the projection uses the student's quantization parameters, the teacher signal itself becomes a function of the student's state, coupling the target distribution to training dynamics; this may reduce the information content of targets compared to full-precision features, so the net gain depends on how much teacher signal survives rounding."],"forward_implications":["In label-free settings, a student trained with SQuaT can match or exceed the accuracy of supervised QAT baselines in several configurations, including surpassing the full-precision teacher under 4-bit quantization on CIFAR-10 and CIFAR-100.","The largest gains are concentrated in extreme low-bit regimes (1- and 2-bit), where the representational mismatch between teacher and student is most severe and the lower bound is largest.","The method works across CNNs, Vision Transformers, and BERT, and with L1, L2, KL, and cosine feature losses, so the benefit is tied to the projection principle rather than a specific architecture or loss.","No teacher-side calibration pretraining is needed, removing the additional overhead required by teacher-quantizer-based approaches such as QFD.","Models trained with SQuaT at INT8 on an edge device achieve a 6.49x throughput speedup over FP32 inference while maintaining competitive accuracy."],"supporting_citations":[{"why":"Provides the label-free QAT baseline (SQAKD) that SQuaT must beat and motivates the logit-level distillation setting.","marker":"Zhao and Zhao, 2024"},{"why":"Introduces QFD, the teacher-side calibrated feature quantizer that SQuaT contrasts with, establishing the need for student-side alignment.","marker":"Zhu et al., 2023"},{"why":"Defines EWGS, the QAT training framework whose forward/backward quantization path is reused to build the student-aware projection.","marker":"Lee et al., 2021"},{"why":"Provides the general knowledge distillation formulation that the label-free QAT setting is built on.","marker":"Hinton et al., 2015"},{"why":"Supplies the Straight-Through Estimator used for gradient propagation in the transformer-based QAT experiments.","marker":"Bengio et al., 2013"},{"why":"Motivates feature-level self-supervised learning, justifying the move from logit-only to feature-level distillation.","marker":"Chen et al., 2020"}],"fun_headline_variants":["Student-quantized teacher features nullify the KD lower bound","Label-free QAT: project teacher features onto student's grid","SQuaT: student-aware teacher quantization removes loss floor","Quantize teacher with student's map to reach zero residual","No unattainable targets: student-aware quant in KD"],"cache_read_input_tokens":17152,"weakest_assumption_plain":"The load-bearing premise is that the student's learned quantization range, which is optimized for the student's own activation distribution, is also a suitable range for projecting teacher features; if teacher feature values fall mostly outside that range, the projected targets would discard much of the teacher's knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Student-quantized teacher features nullify the KD lower bound","Label-free QAT: project teacher features onto student's grid","SQuaT: student-aware teacher quantization removes loss floor","Quantize teacher with student's map to reach zero residual","No unattainable targets: student-aware quant in KD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1477,"prompt_tokens":950,"completion_tokens":527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":566,"tokens_out":527,"duration_ms":108717,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:53:42.472094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a linear probe on the projected teacher features $\\tilde{f}_{T,\\ell} = \\Pi_{\\phi_{S,\\ell}}(f_{T,\\ell})$ at a distillation layer and compare its label-prediction accuracy with a probe on the raw teacher features $f_{T,\\ell}$; if the projected features are near chance while the raw features are informative, the student-aware projection has discarded the teacher's signal and cannot be the source of the reported accuracy gains.","supporting_citations":[],"review_version":1}