Pith. sign in

REVIEW 4 major objections 4 minor 22 references

Data-Free Knowledge Distillation for LiDAR-Aided Beam Tracking in MmWave Systems

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A student model trained only on synthetic LiDAR data generated from a pretrained teacher's feature statistics can match or beat that teacher at predicting current and future mmWave beams, without access to real training data or labels.

desk verdict A credible application of standard DF-KD to LiDAR-aided beam tracking, but the unstated provenance of the teacher's feature statistics in Eq. (5) needs to be resolved before the 3.8% claim can be trusted. read the letter →

arxiv 2509.19092 v2 pith:BWRA5BBJ submitted 2025-09-23 eess.SP

Abolfazl Zakeri , Nhan Thanh Nguyen , Ahmed Alkhateeb , Markku Juntti This is my paper
classification eess.SP
keywords data-freeknowledgedistillationbeamtrackingLiDAR-aidedmmWaveinversionsyntheticdatagenerationlogitmatchingCNN-GRUvehicularcommunications
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that beam tracking in mmWave vehicular systems can be done without any real training data at the base station. The authors propose a data-free knowledge distillation pipeline: a two-layer generator turns random noise into synthetic LiDAR-like sequences, guided only by statistics of the pretrained teacher's internal features, its activation norms, and its output entropy, and then a smaller student is trained on those synthetic sequences by matching the teacher's logits. The central result is that this student, with 25,408 parameters versus the teacher's 96,640, slightly outperforms the teacher on Top-1 accuracy, by about 3.8%, and that the metadata loss, aligning the mean and variance of the teacher's last-layer features, is the key ingredient. The paper also argues that a simple mean-squared-error loss on logits can replace the standard Kullback-Leibler distillation loss, reducing hyperparameter tuning. If true, this would let infrastructure deploy efficient beam predictors with no dataset storage or privacy burden.

What carries the argument

The load-bearing mechanism is knowledge inversion: a two-layer fully connected generator with ReLU and tanh activations maps Gaussian noise to synthetic LiDAR-shaped sequences. The generator is trained by backpropagating a loss whose main term is the metadata loss, mean squared error between the mean and variance of the teacher's last-layer hidden features on synthetic data and precomputed statistics from real data, plus an activation-norm maximizer and an output-entropy minimizer. The student is then trained on the synthetic sequences by matching the teacher's raw logits (Eq. 10, MSE) or softened distributions (Eq. 9, KL), with no ground-truth beam labels. The metadata loss is what makes th

What would settle it

Retain the exact teacher and student setup, but recompute the metadata mean and variance using only the training split of the real dataset. If the student's test Top-1 accuracy then drops well below the teacher's, the reported 3.8% gain in Fig. 5(a) depended on metadata leakage from the test split. Alternatively, compute the distance between synthetic and real LiDAR in the teacher's feature space, such as maximum mean discrepancy; if the distance is large while accuracy stays high, the student is succeeding for reasons other than distributional fidelity.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that data-free knowledge distillation is viable for LiDAR-aided beam tracking. A generator is trained to synthesize LiDAR input sequences from random noise, with a loss combining metadata (mean and variance of the teacher's last hidden-layer features), an activation-norm regularizer, and an output-entropy term. The resulting synthetic data, together with either KL divergence or a proposed MSE logit-matching loss, trains a CNN-GRU student that matches or exceeds the teacher's Top-1 and Top-5 beam prediction accuracy for current and three future slots. The authors report that the metadata loss is the dominant contributor, and that the MSE student loss pe

Load-bearing premise

The load-bearing premise is that synthetic LiDAR generated solely from the teacher's last-layer feature statistics is similar enough to real LiDAR that a student trained only on those synthetic sequences generalises to real test data, yet the paper validates this only through final accuracy and never reports whether the metadata statistics used in Eq. (5) came from the training or evaluation split.

Editorial extensions

If this is right

  • A base station can deploy a beam-tracking student without storing or transmitting real LiDAR data, since the student is trained entirely on synthetic sequences derived from teacher statistics.
  • The 3.8% accuracy gain over the teacher suggests data-free distillation acts as a regularizer, so a smaller model may generalise slightly better than the large teacher on the same test distribution.
  • The MSE logit loss removes temperature tuning, making the student-training stage simpler and more reproducible than standard KD.
  • The metadata loss's dominance implies that aligning the teacher's last-layer feature statistics is sufficient to generate useful synthetic sensing data in this task, which can guide generator design in similar multimodal problems.
  • If the method transfers to other modalities, communication systems that already have a pretrained teacher can generate training data for new students at the edge with no dataset access.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the claim holds, the same generator-metadata inversion could plausibly work with other sensing inputs such as camera images or radar point clouds, because the teacher's last-layer statistics may be modality-agnostic enough to invert.
  • An identically sized student trained on real LiDAR with the same architecture would isolate whether the 3.8% gain is a genuine distillation benefit or an artifact of the synthetic-noise regularisation.
  • The paper validates generator fidelity only through final accuracy; a direct distributional comparison in feature space, such as maximum mean discrepancy, would test whether the generator truly reproduces the real LiDAR distribution rather than exploiting a shortcut in the teacher.
  • Because the method relies on teacher feature statistics, it inherits the teacher's blind spots: if the teacher was trained on a biased or narrow LiDAR distribution, the synthetic data will inherit that bias, so dataset coverage remains a hidden requirement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a data-free knowledge distillation (DF-KD) framework for LiDAR-aided mmWave beam tracking. A generator is trained by knowledge inversion to synthesize LiDAR-like sequences from random noise, using a loss combining metadata statistics (mean/variance of the teacher's last-layer features over real data), activation norm, and output entropy. The student (a smaller CNN-GRU) is then trained exclusively on synthetic samples, using either KL divergence or a proposed MSE logit-matching loss. Experiments on the DeepSense dataset Scenario 8 report that the DF-KD student slightly outperforms the teacher in Top-1/Top-5 accuracy, and that the MSE loss is a simpler alternative to standard KD. The paper claims that this is the first DF-KD study for LiDAR-aided beam tracking.

Significance. If the central claim holds—that a student trained only on synthetic data generated without raw LiDAR samples can match or beat a teacher trained on the real dataset—then the contribution is practically meaningful for scenarios where data storage, transmission, or privacy prevents access to raw training data at the base station. The paper also provides a useful empirical comparison of KL vs. MSE distillation losses and shows the dominant role of the metadata loss. The use of a real, public dataset (DeepSense) and the availability of code are strengths. However, the strongest claim (3.8% Top-1 improvement over the teacher) rests on the provenance of the metadata statistics and on the absence of test-set leakage, which is not established in the manuscript. The lack of distributional diagnostics and the single-run nature of all results further temper confidence.

major comments (4)
  1. [Section III-B, Eq. (5)] The provenance of the metadata statistics µ and σ² is not specified. The metadata loss in Eq. (5) is the only channel through which real-data information enters the generator, and hence the student. Footnote 1 states that 'the original validation data used in this paper resides at the BS,' which raises the concern that these statistics may have been computed on the evaluation split. If so, the generator is trained to match teacher feature statistics on the test distribution, and the reported 3.8% Top-1 gain over the teacher is an artifact of test leakage. The authors must state explicitly which subset (train/validation/test) of DeepSense was used to compute µ and σ², and should provide distributional diagnostics (e.g., MMD, FID, or nearest-neighbor distance) between the synthetic samples and real samples from the training split. Final accuracy alone cannot distinguish genuine generalizat
  2. [Section IV, Figs. 3–5] All reported accuracies appear to be from a single run, with no error bars, confidence intervals, or significance tests. The central claim of 'slightly outperforming' the teacher by 3.8% in Fig. 5(a) could easily be within run-to-run variance for a model trained for 500 epochs with random initialization and random synthetic-data generation. The authors should report mean and standard deviation over multiple seeds, and ideally a paired significance test, before claiming that DF-KD outperforms the teacher. This is load-bearing because the abstract and conclusion emphasize matching or exceeding teacher performance.
  3. [Section III-B, Eqs. (5)–(8)] The term 'data-free' is used loosely. Computing the metadata statistics requires passing real data (or its features) through the pretrained teacher, which means raw LiDAR samples or their features must be available at the time the generator is trained. The abstract says 'without access to raw LiDAR samples,' but Eq. (5) requires real-data statistics. This is not necessarily a flaw if the statistics are computed once offline and only the statistics (not the raw data) are retained, but the distinction should be stated precisely. As written, the 'data-free' claim is overstated and the reader cannot tell whether the framework actually avoids storing raw data at the BS.
  4. [Section IV, Fig. 3] The choice of 'best-performing generator' for the student-loss comparison (Fig. 4) and for the final DF-KD results (Fig. 5) is not defined with an objective criterion. If the generator is selected based on the same test accuracies that are later reported, this introduces selection bias. The authors should specify whether the generator and student hyperparameters (α, β, T, γ, hidden size) were selected on a validation set separate from the reported test set, and if so, which validation set was used.
minor comments (4)
  1. [Section III-B, Eq. (6)] The notation is inconsistent: the loss is defined as a negative mean over the batch, but the subscript 'act.' and the use of ˜fT(G(˜X)) is not fully defined. Also 'ReLu' should be 'ReLU'.
  2. [Section IV, Eq. (9)] The KL loss is written with a leading T² factor but the temperature dependence of the softmax is not explained in the text; please clarify the standard KD convention and why T² is included.
  3. [Section IV, Fig. 5 caption] The caption says 'the proposed DF-KD with the MSE loss' but the text in Section IV also compares DF-KD with KL loss; please make the correspondence between figures and methods explicit.
  4. [General] The phrase 'the provided DF-KD' in Section IV should be 'the proposed DF-KD'. There are also several minor grammatical issues throughout (e.g., 'the teacher’ features' in Section III-B).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; student is scored on real test data, and the teacher/dataset are public prior work.

full rationale

The derivation chain is not circular. The student model is trained exclusively on synthetic LiDAR sequences produced by the generator, and its Top-1/Top-5 accuracy is measured on the real DeepSense test/validation split, an external benchmark that is independent of the generator and student fitting procedure. The teacher network and dataset derive from prior public work [1,10] with overlapping authors, but they are external, reproducible artifacts; the paper does not invoke any uniqueness theorem or unverified self-citation to rule out alternatives. The metadata loss in Eq. (5) is the standard DF-KD knowledge-inversion mechanism (cf. [18]): it aligns the teacher's last-layer feature statistics between synthetic and real data, and the student's output matching via Eq. (10) is a training loss, not a definition of the reported accuracy. The only substantive concern is that the manuscript does not specify which real-data split provides the metadata statistics; if those statistics were computed on the evaluation split, the reported test accuracy would be leakage-inflated. However, that would be a benchmark-integrity/correctness issue, not a by-construction circularity, and the paper's text does not establish that the evaluation data is used to fit the generator. The footnote 1 statement that validation data resides at the BS is a limitation worth noting, but it does not by itself demonstrate that the metadata comes from that split. No load-bearing step reduces to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three external inputs: a pretrained teacher from the authors' own prior work ([10]), the public DeepSense dataset ([1]), and hand-chosen loss weights plus temperatures. No new physical or conceptual entities are postulated; the generator and synthetic sequences are method artifacts, not independently testable inventions. The most important audit item is the metadata statistics (mu, sigma^2) in Eq. (5), whose computation set (train vs validation) is unspecified; that single ambiguity carries most of the risk to the headline accuracy claim.

free parameters (5)
  • alpha (generator activation loss weight) = 1e-4
    Weights the activation norm term in generator loss (8); chosen by hand, not swept; Fig. 3 compares the weighted blend against each component alone.
  • beta (generator entropy loss weight) = 1e-2
    Weights the entropy term in generator loss (8); hand-chosen. The generator's output distribution, and hence the student's training data, depends on this balance.
  • KD temperature T = 5
    Softening temperature for KL distillation in Eqs. (9) and (11); described as fine-tuned in Section IV. Required for the KL baseline but not for the proposed MSE loss.
  • KD mixing weight gamma = 0.7
    Weights KL versus cross-entropy in the standard KD loss (11); used only for the comparison baseline, not for the proposed DF-KD.
  • student hidden size = 32
    Reduced from the teacher's 128 hidden neurons (Section IV), giving 25,408 vs 96,640 trainable parameters. This capacity choice determines the expected accuracy gap between teacher and student.
assumptions (5)
  • domain assumption The pretrained teacher from [10] encodes reliable LiDAR-to-beam knowledge and its soft outputs provide a valid training signal.
    Section IV 'Model and Parameters' adopts the [10] model as teacher without retraining; the student's success depends on the teacher's soft targets and features being informative.
  • domain assumption The teacher's last-layer feature mean/variance over real data (the metadata in Eq. 5) is a sufficient representation of the real data distribution for the generator to invert.
    Section III-B: the metadata loss is the dominant generator objective. The paper does not specify which real-data subset produced mu and sigma^2, so representativeness and no-leakage are assumed.
  • domain assumption The generator's synthetic outputs, shaped as K frames with zero-padding of length V, occupy the same input space as real preprocessed LiDAR sequences.
    Section III-B: 'The output is shaped as a sequence of K frames, each with D features matching to the real LiDAR input format. Then, a zero-padding is applied with length V, the same as in the real dataset [10].' Requires that all real-data preprocessing is captured by this format.
  • domain assumption DeepSense Scenario 8 and its train/validation split are used as in [10], without re-splitting.
    Section IV: 'Similarly to [10], we choose Scenario 8 from the DeepSense dataset'; the split logic and teacher test loss (1.3172) are inherited from [10].
  • standard math Argmax over softmax-normalized logits (Eqs. 3-4) is a valid surrogate for SNR-maximizing beam selection.
    Section II-B: the ML objective replaces the SNR-maximization in Eq. (3); the validity of top-K beam candidates depends on the label-generation procedure in [10].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data-Free Knowledge Distillation for LiDAR-Aided Beam Tracking in MmWave Systems." pith.science (2026). https://pith.science/paper/BWRA5BBJ

@misc{pith2026250919092,
  author       = {Pith},
  title        = {Pith review of: Data-Free Knowledge Distillation for LiDAR-Aided Beam Tracking in MmWave Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWRA5BBJ}},
  note         = {Machine review of arXiv:2509.19092}
}
read the original abstract

We propose a data-free knowledge distillation (DF- KD) framework for LiDAR-aided mmWave beam tracking, where the objective is to predict the optimal current and future beams from a sequence of past LiDAR measurements. Specifically, we propose a knowledge inversion approach where a generator synthesizes LiDAR-like sequences from random noise, using a metadata loss to align the teachers internal feature statistics of synthetic and real data, without access to raw LiDAR samples. The student model is then trained exclusively on the synthetic data using either the Kullback- Leibler (KL) divergence loss or a proposed mean squared error (MSE) loss between the teachers and students raw output logits. Simulation results on the DeepSense dataset demonstrate the effectiveness of the proposed approach. In particular, the proposed convolutional neural network-gated recurrent unit (CNN-GRU) teacher architecture yields superior DF-KD student performance compared to GRU-only alternatives, and the MSE loss achieves performance comparable to the standard KD loss while requiring fewer hyperparamete

Figures

Figures reproduced from arXiv: 2509.19092 by the authors.

Figure 1
Figure 1. A schematic of training the generator G using the pretrained teacher T takes the pretrained teacher and statistical priors as input and proceeds in two main steps. In step (1), the generator G is trained using the pretrained teacher. In step (2), the student model is trained using both the pretrained teacher and the generator. The details of these two steps are explained in the following sections. B. Synthetic Data … view at source ↗
Figure 2
Figure 2. A schematic of training the student S using the pretrained teacher T and pretrained generator G choice. Additionally, inspired by [21], we also incorporate a mean squared error (MSE) loss between the teacher’s and student’s logits. We numerically compare the performance of the KL and the MSE losses in Section IV. KL Divergence Loss: KL divergence measures how one probability distribution diverges from another. In KD… view at source ↗
Figure 3
Figure 3. The top-1 and top-5 accuracies of the teacher and proposed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The top-1 and top-5 accuracies of the teacher and proposed [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The Top-1 and Top-5 accuracy of the teacher and proposed [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 5 linked inside Pith

  1. [18]

    Data-free knowledge distillation for deep neural networks,

    R. G. Lopes, S. Fenu, and T. Starner, “Data-free knowledge distillation for deep neural networks,” inProc. NeurIPS Workshop Deep Learn. Without Labels, (Long Beach, CA, USA), 2017. [Online]. Available: https://arxiv.org/abs/1710.07535

    arXiv 2017

  2. [17]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

  3. [21]

    Understanding knowledge distillation,

    T. Kim, J. Oh, N. Kim, S. Cho, and S. Yun, “Understanding knowledge distillation,”OpenReview (ICLR Submission), 2019

    2019

  4. [1]

    Deepsense 6G: A large-scale real-world multi-modal sensing and communication dataset,

    A. Alkhateeb, G. Charan, T. Osman, A. Hredzak, J. Morais, U. Demirhan, and N. Srinivas, “Deepsense 6G: A large-scale real-world multi-modal sensing and communication dataset,”IEEE Commun. Mag., vol. 61, no. 9, pp. 122–128, Sep. 2023

    2023

  5. [2]

    Multimodality in mmWave MIMO beam selection using deep learning: Datasets and challenges,

    J. Gu, B. Salehi, D. Roy, and K. R. Chowdhury, “Multimodality in mmWave MIMO beam selection using deep learning: Datasets and challenges,”IEEE Commun. Mag., vol. 60, no. 11, pp. 36–41, 2022

    2022

  6. [3]

    Leveraging sensing at the infrastructure for mmWave communication,

    A. Ali, N. Gonzalez-Prelcic, R. W. Heath, and A. Ghosh, “Leveraging sensing at the infrastructure for mmWave communication,”IEEE Commun. Mag., vol. 58, no. 7, pp. 84–89, Jul. 2020

    2020

  7. [4]

    Machine learning-based mmwave MIMO beam tracking in V2I scenarios: Algo- rithms and datasets,

    A. Oliveira, D. Suzuki, S. Bastos, I. Correa, and A. Klautau, “Machine learning-based mmwave MIMO beam tracking in V2I scenarios: Algo- rithms and datasets,” inProc. IEEE Latin-American Conf. on Commun. (LATINCOM), pp. 1–5, Medellin, Colombia, Dec. 2024

    2024

  8. [5]

    Harnessing multimodal sensing for multi-user beamforming in mmWave systems,

    K. Patel and R. W. Heath, “Harnessing multimodal sensing for multi-user beamforming in mmWave systems,”IEEE Trans. Wireless Commun., vol. 23, no. 12, pp. 18725–18739, Dec. 2024

    2024

Show all 22 references
  1. [6]

    Environment semantic com- munication: Enabling distributed sensing aided networks,

    S. Imran, G. Charan, and A. Alkhateeb, “Environment semantic com- munication: Enabling distributed sensing aided networks,”IEEE Open J. Commun. Soc., vol. 5, pp. 7767–7786, Dec. 2024

    2024

  2. [7]

    Multi-modality sensing in mmWave beamforming for connected vehicles using deep learning,

    M. B. Mollah, H. Wang, M. A. Karim, and H. Fang, “Multi-modality sensing in mmWave beamforming for connected vehicles using deep learning,”IEEE Trans. on Cogn. Commun. Netw., Early Access, 2025

    2025

  3. [8]

    Constrained multimodal sensing-aided communications: A dynamic beamforming design,

    A. Zakeri, N. T. Nguyen, A. Alkhateeb, and M. Juntti, “Constrained multimodal sensing-aided communications: A dynamic beamforming design,” inProc. IEEE Global Commun. Conf., Accepted, 2025. Available at: https://arxiv.org/abs/2505.10015

    2025 arXiv

  4. [9]

    M2BeamLLM: Multimodal sensing-empowered mmWave beam pre- diction with large language models,

    C. Zheng, J. He, C. G. Kang, G. Cai, Z. Yu, and M. Debbah, “M2BeamLLM: Multimodal sensing-empowered mmWave beam pre- diction with large language models,”arXiv preprint arXiv:2506.14532, Jun. 2025

    2025 arXiv

  5. [10]

    LiDAR aided future beam prediction in real-world millimeter Wave V2I communications,

    S. Jiang, G. Charan, and A. Alkhateeb, “LiDAR aided future beam prediction in real-world millimeter Wave V2I communications,”IEEE Commun. Lett., vol. 12, no. 2, pp. 212–216, Feb. 2023

    2023

  6. [11]

    Deep learning on multimodal sensor data at the wireless edge for vehicular network,

    B. Salehi, G. Reus-Muns, D. Roy, Z. Wang, T. Jian, J. Dy, S. Ioannidis, and K. Chowdhury, “Deep learning on multimodal sensor data at the wireless edge for vehicular network,”IEEE Trans. Veh. Technol., vol. 71, no. 7, pp. 7639–7655, Jul. 2022

    2022

  7. [12]

    Multimodal deep learning empowered millimeter-Wave beam prediction,

    B. Shi, M. Li, M.-M. Zhao, M. Lei, and L. Li, “Multimodal deep learning empowered millimeter-Wave beam prediction,” inProc. IEEE Veh. Technol. Conf., pp. 1–6, Singapore, Jun. 2024

    2024

  8. [13]

    Salehihikouei,Leveraging Deep Learning on Multimodal Sensor Data for Wireless Communication: From mmWave Beamforming to Digital Twins

    B. Salehihikouei,Leveraging Deep Learning on Multimodal Sensor Data for Wireless Communication: From mmWave Beamforming to Digital Twins. PhD thesis, Northeastern University, 2024

    2024

  9. [14]

    Vision-position multi-modal beam prediction using real millimeter wave datasets,

    G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” inProc. IEEE Wireless Commun. and Networking Conf., pp. 2727–2731, Austin, TX, USA, Apr. 2022

    2022

  10. [15]

    Multimodal deep learning-empowered beam prediction in future THz ISAC systems,

    K. Zhang, W. Yu, H. He, S. Song, J. Zhang, and K. B. Letaief, “Multimodal deep learning-empowered beam prediction in future THz ISAC systems,”arXiv preprint arXiv:2505.02381, May 2025

    2025 arXiv

  11. [16]

    Resource-efficient beam prediction in mmWave communications with multimodal realistic simulation framework,

    Y . M. Park, Y . K. Tun, W. Saad, and C. S. Hong, “Resource-efficient beam prediction in mmWave communications with multimodal realistic simulation framework,”arXiv preprint arXiv:2504.05187, Apr. 2025

    2025

  12. [19]

    A comprehensive survey on knowledge distillation,

    A. M. Mansourian, R. Ahmadi, M. Ghafouri, A. M. Babaei, E. B. Golezani, Z. Y . Ghamchi, V . Ramezanian, A. Taherian, K. Dinashi, A. Miri, and S. Kasaei, “A comprehensive survey on knowledge distillation,”arXiv preprint arXiv:2503.12067, Mar. 2025

    2025

  13. [20]

    Improving knowledge distillation via regularizing feature direction and norm,

    Y . Wang, L. Cheng, M. Duan, Y . Wang, Z. Feng, and S. Kong, “Improving knowledge distillation via regularizing feature direction and norm,” in Lecture Notes in Computer Science, pp. 20–37, Springer Nature, 2024

    2024

  14. [22]

    Self-distillation from the last mini-batch for consistency regularization,

    Y . Shen, L. Xu, Y . Yang, Y . Li, and Y . Guo, “Self-distillation from the last mini-batch for consistency regularization,” inCVPR, pp. 11943–11952, IEEE/CVF, Jun. 2022

    2022

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.