Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A shared discrete codebook lets an unpaired student model learn disease-level knowledge from a stronger medical modality, improving OCT-to-fundus from 64.5 to 70.2 macro-F1 and CT-to-chest-X-ray from 73.8 to 76.3 with no added inference…

desk verdict Useful new distillation recipe with real gains, but the paper doesn't yet show the gains come from teacher knowledge rather than codebook regularization; needs a no-knowledge teacher control. read the letter →

arxiv 2607.27357 v1 pith:FMQ5UC3N submitted 2026-07-29 cs.CV eess.IV

Dillan Imans , Phuoc-Nguyen Bui , Duc-Tai Le , Hyunseung Choo This is my paper · ORCID
classification cs.CVeess.IV
keywords unpairedcross-modaldistillationsharedsemanticcodebookmedicalimageclassificationknowledgedistributionalignmentclass-conditionalOCT-to-fundusCT-to-chest-X-ray
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to transfer diagnostic knowledge between two medical imaging modalities when no patient-level pairing exists, such as teaching a fundus-photo model from OCT scans or a chest-X-ray model from CT scans. The method represents each image as a distribution over a common set of learned code vectors, then aligns teacher and student code distributions both globally and per disease class, sidestepping the need for paired samples or directly comparable feature spaces. On two unpaired settings, the student improves from 64.5 to 70.2 macro-F1 and from 73.8 to 76.3 macro-F1, beating all evaluated distillation baselines; at inference only the student encoder and classifier remain. A sympathetic reader would care because it suggests that disease-level structure, rather than pixel or feature correspondence, is a sufficient bridge for cross-modal medical transfer.

What carries the argument

The central object is the shared semantic codebook: a set of $K$ unit-norm vectors in a common projection space, maintained as a buffer updated by exponential moving average rather than gradient descent. Both teacher and student projected embeddings are softly assigned to these codes by temperature-scaled cosine similarity, turning each image into a distribution over the same vocabulary. The load-bearing mechanism is the pair of distribution-alignment losses: a global term that matches teacher and student batch-marginal code usage under forward KL divergence, and a class-conditional term that matches per-class mean code distributions with squared error. A frozen, randomly initialized teacher projection head keeps the teacher-side distribution stationary while the student projector is trained; the paper invokes the random-projection geometry phenomenon to justify that class structure survives this untrained projection. Diversity guards, including entropy regularization on student code usage and restart of dead codes, prevent collapse onto a few dominant codes.

What would settle it

Train the student with the same objective but replace the frozen random teacher projection with a learned projection fit to preserve teacher class separability, or with no projection at all, and compare on the same splits; if the gains disappear or reverse, the frozen projection is not preserving the class structure the method relies on. Alternatively, measure teacher class separability in the projected codebook space before any student training: if classes are already indistinguishable there, the alignment signal is not semantic.

Watch

Extended reading notes

Core claim

The central claim is that unpaired cross-modal knowledge distillation can be achieved by routing both teacher and student through a shared discrete codebook and matching their aggregate code-usage statistics, instead of matching instances or continuous features. The teacher's frozen feature extractor and the student's trainable one are each projected into a shared low-dimensional space, softly assigned to a set of $K$ learnable code vectors, and compared through two losses: a global KL divergence between batch-marginal code distributions and a per-class squared-error term between class-mean code distributions. The codebook is evolved by exponential moving average, with entropy regularization and dead-code restart to keep the vocabulary diverse. On OCT-to-fundus retinal classification the student model rises from 64.5 to 70.2 macro-F1, and on CT-to-chest-X-ray pneumonia classification from 73.8 to 76.3, outperforming all evaluated distillation baselines. Because all teacher-side and codebook modules are discarded at inference, the method adds no deployment cost.

Load-bearing premise

The load-bearing premise is that a frozen, randomly initialized projection of the teacher's features preserves enough disease-class structure for the shared codebook to carry genuine semantic signal; if that projection scrambles class structure, the teacher-side alignment target becomes noise and the measured gains would not reflect real knowledge transfer.

Editorial extensions

If this is right

  • If the paper is right, disease-level semantic transfer works across unpaired, heterogeneous modalities without paired samples or direct feature matching.
  • The same codebook mechanism transfers unchanged between very different modality pairs, namely OCT to fundus and CT to chest X-ray.
  • At inference the student model is identical to a plain student encoder plus classifier, so the transfer comes at zero deployment cost.
  • Rare classes receive weaker class-conditional alignment because they co-occur less often in both mini-batches, making their transferred code signatures less stable across training runs.
  • SSCD recovers only part of the gap to the teacher upper bound, indicating that some teacher information is modality-specific and cannot be expressed from the student image alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the alignment is distributional rather than instance-level, the method should generalize to settings where teacher and student labels are only partially overlapping or where the student set is unlabeled, as long as class-conditional teacher statistics are available.
  • The frozen random projection assumption could be tested directly by ablating a learned teacher projection; if performance holds, the shared codebook is robust to that choice, and if not, part of the measured gain is a projection artifact.
  • The per-class alignment table suggests that co-occurrence-aware or class-balanced sampling should stabilize rare-class transfer, which the paper itself names as future work.
  • If the codebook codes turn out to be semantically interpretable, the method could double as an explanation tool for what the student model actually learned from the teacher modality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Shared Semantic Codebook Distillation (SSCD) for unpaired cross-modal medical image classification. Teacher and student encoders project their features into a shared low-dimensional space, softly assign them to a common discrete codebook, and are trained to align batch-marginal and class-conditional code distributions, with the codebook updated by EMA and kept diverse via entropy regularization and dead-code restart. The teacher and codebook are discarded at inference. Experiments on OCT→fundus (6-class MultiEYE) and CT→CXR (binary COVIDx/NIH) report macro-F1 gains of +5.7 and +2.5 over the student-only baseline, outperforming the evaluated baselines (DistillGlobMean, DistillClsMean, FDDM). The method is well specified, includes ablations and three-seed error bars, and releases code.

Significance. If the central claim holds, SSCD provides a practical, generic mechanism for transferring disease-level knowledge between unpaired, heterogeneous medical modalities without paired data or direct feature matching, at no inference cost. The paper's strengths include a clear method description, two heterogeneous benchmark settings, ablations of each loss term, and public code. However, the key interpretive claim—that the gains arise from the teacher's disease knowledge rather than from the codebook losses acting as a regularizer—is not currently supported by the experiments as designed. The missing no-knowledge teacher control is therefore load-bearing for the paper's central contribution.

major comments (3)
  1. [Sec. 3.5, Eq. (7)–(9); Sec. 4.4, Table 4] The central claim that SSCD transfers disease-level knowledge is not yet supported because no control separates the teacher's class information from the regularizing effect of the auxiliary codebook losses. The losses in Eq. (7) and Eq. (9) operate on aggregate, detached teacher distributions, so they can improve a student's marginal and class-conditional behavior even if the teacher codes carry no class structure. Table 4 reports cosine similarities between teacher and student class-mean code distributions (0.60–0.80 and 0.49–0.55), but these values are not compared against a null model such as a randomly initialized teacher, a label-shuffled teacher, or a teacher whose codes are random; high similarity could arise simply because both distributions are close to the shared marginal enforced by Ldist. The authors should run a no-knowledge teacher control (e.g., an untrained teacher encoder, or a teacher trained on shuffled labels) through the identical SSCD pipeline and report whether the +5.7/+2.5 F1 gains persist. If they do, the results would indicate a codebook-induced regularizer rather than cross-modal semantic transfer, and the paper's framing would need to change accordingly.
  2. [Sec. 3.3, 'Frozen teacher projection'] The justification for freezing the teacher projector g_T at random initialization relies on an analogy to the Johnson–Lindenstrauss phenomenon, but JL guarantees apply to linear maps, while g_T is a two-layer nonlinear ReLU MLP. The paper's cited empirical evidence is the 'teacher class separability in Fig. 2', yet Fig. 2 shows UMAP of backbone features h_T, not of the projected embeddings z_T used for codebook assignment; it therefore does not directly validate the crucial assumption that class structure survives the random nonlinear projection. Because all teacher-side signal flows through g_T, a degradation of class structure there would make the distillation target noisy. The authors should either (a) measure class separability or k-NN accuracy on z_T itself, or (b) ablate against a learned or fine-tuned g_T, to demonstrate that the frozen random projection is not a major source of information loss.
  3. [Sec. 4.1, 'Baselines'] OCT-CoDA, the strongest prior unpaired cross-modal method on OCT→fundus, is excluded from the main comparison because it requires LLM-generated disease concepts and a pretrained vision-language backbone. The stated reasons are practical, but the paper's contribution claims that SSCD 'outperforms all evaluated distillation baselines' and that what 'remains open' is a generic framework without external modality-specific priors. As written, the comparison does not establish that SSCD is competitive with the previous state of the art on the MultiEYE benchmark; it only establishes competitiveness with the reimplemented FDDM and simple mean-matching baselines. The authors should either include an OCT-CoDA comparison under the original protocol, or clearly state that the claim is limited to methods that do not use external language/vision-language resources.
minor comments (5)
  1. [Throughout] There are several LaTeX artifacts in the captions, e.g., 'T able 1' and 'T able 2', which should be fixed.
  2. [Sec. 4.4, Table 4] The per-class cosine similarities are reported without confidence intervals or a statistical test against a null distribution; since the mean values are moderate (0.49–0.55 on CT→CXR), a permutation or bootstrap test, or a comparison with a random-projection baseline, would make the alignment claim more convincing.
  3. [Sec. 3.4] The initialization of the codebook C is not specified; please state how the K code vectors are initialized before the EMA updates begin.
  4. [Sec. 4.1] The hyperparameter selection procedure for the baselines sweeps λ over a grid, but the text does not state whether SSCD's own hyperparameters (τ, λdist, λcls-cb, λent, K, d, γ) were selected on validation; reporting this would clarify the comparison's fairness.
  5. [Sec. 3.5] In Eq. (8), the notation N_c^T and N_c^S is used inconsistently with the earlier N_T and N_S; consider unifying the notation for count variables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SSCD's reported improvements are measured on held-out test sets, and no derived quantity reduces by construction to a fitted input.

full rationale

The paper's central claims are empirical: the headline macro-F1 gains are obtained by evaluating the trained student on held-out test data, not by construction from the loss definitions. Equations (7) and (9) define distillation objectives in terms of teacher and student code distributions, but these are optimization losses rather than predictions derived from fitted parameters; the reported test metrics are independent of those losses by measurement. The shared codebook is evolved by EMA from the same teacher-student projections it aligns, but this is the proposed mechanism itself rather than a circular step: the codebook is not tuned to reproduce the reported improvements, and no equation is equivalent to the final macro-F1 by definition. The paper contains no load-bearing self-citations: references [18] and [19] are baseline and dataset sources from other groups, and the Johnson-Lindenstrauss citation [12] is used only as a heuristic motivation for the frozen random projection, with the paper explicitly providing empirical checks (Fig. 2 and Table 4) rather than relying on the citation as proof. The reviewer concern that a no-knowledge teacher control is missing is about experimental attribution (whether gains come from semantic transfer or from codebook-induced regularization), which is a validity question, not evidence of circularity. Under the specified patterns, no step reduces to its own inputs, so the appropriate score is 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

SSCD introduces no new physical or mathematical entities; it borrows a learned codebook and EMA updates from VQ/SSL literature. The central claim depends on several hand-chosen hyperparameters (temperature, loss weights, codebook size, momentum) and on the domain assumptions listed above.

free parameters (8)
  • assignment temperature tau = 0.1
    Temperature in the soft assignment over code similarities (Eq. 4); chosen by hand, controls sharpness of code assignments.
  • loss weights lambda_dist and lambda_cls-cb = 0.5 each
    Weights for the distribution and class-conditional alignment losses in the total objective (Eq. 14); set by hand, not tuned on validation.
  • entropy weight lambda_ent = 0.05
    Weight for the student marginal entropy penalty; kept small to act as a collapse guard rather than competing with L_dist.
  • codebook size K = 256
    Number of shared codes in the codebook; chosen by hand.
  • EMA momentum gamma = 0.99
    Momentum for the online EMA codebook updates (Eqs. 10-11).
  • projection dimension d = 128
    Dimensionality of the shared embedding space after the projection heads; chosen by hand.
  • dead-code threshold eta = 1
    Count threshold below which a code is re-seeded to a random embedding.
  • restart period R = 100 steps
    Interval between checks for inactive codes.
assumptions (4)
  • domain assumption The teacher and student datasets share the same label space but contain no patient-level correspondence.
    Defines the unpaired problem setting in Sec. 3.1; the method is designed for this condition.
  • ad hoc to paper A random, frozen nonlinear MLP projection of a trained backbone approximately preserves class structure (Johnson-Lindenstrauss phenomenon).
    This justifies freezing g_T at random initialization in Sec. 3.3; the JL theorem applies to linear embeddings, and the extension to a nonlinear ReLU MLP is an assumption supported only by self-reported empirical observation.
  • domain assumption Matching global and class-conditional distributions over a shared codebook is sufficient to transfer diagnostic knowledge across heterogeneous modalities.
    The core design premise of the distillation objective in Sec. 3.5; the paper does not provide an independent semantic grounding for the codebook.
  • domain assumption EMA updates with hard top-1 assignments and dead-code restart produce a stable and diverse codebook that acts as a common semantic vocabulary.
    Adopts standard practice in VQ training (Sec. 3.6); the stability is not formally guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification." pith.science (2026). https://pith.science/paper/FMQ5UC3N

@misc{pith2026260727357,
  author       = {Pith},
  title        = {Pith review of: Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FMQ5UC3N}},
  note         = {Machine review of arXiv:2607.27357}
}
read the original abstract

Cross-modal knowledge distillation can transfer diagnostic knowledge from a strong but costly teacher modality to a cheaper and more deployable student modality. In medical image analysis, however, the two modalities are often unpaired: they are collected from different patient cohorts and occupy geometrically incompatible feature spaces. This makes instance-level distillation invalid and direct feature matching unreliable. To address these challenges, we propose Shared Semantic Codebook Distillation (SSCD), which compares teacher and student representations through a shared discrete codebook. Each image is represented as a distribution over a common, modality-agnostic vocabulary, and knowledge is transferred by aligning these distributions across modalities, both globally and class-conditionally, without requiring paired samples or directly comparable raw features. The codebook is evolved online by exponential moving average and kept diverse through entropy regularization and dead-code restart. At inference, all teacher-side and codebook modules are discarded, leaving only the student encoder and classifier. On two heterogeneous unpaired settings, OCT-to-fundus retinal disease classification and CT-to-chest-X-ray pneumonia classification, SSCD improves the student from 64.5 to 70.2 macro-F1 and from 73.8 to 76.3 macro-F1, respectively, outperforming all evaluated distillation baselines on both settings. Code and pretrained models are available at https://github.com/DillanImans/SSCD-unpaired-distillation

Figures

Figures reproduced from arXiv: 2607.27357 by the authors.

Figure 1
Figure 1. Overview of SSCD. The two images shown per cohort illustrate the two settings evaluated (OCT→fundus and CT→CXR), each trained independently with a single teacher and single student modality. Teacher and student features are projected into a shared space and softly assigned to a common codebook, where their code distributions are aligned overall and per class. A teacher encoder fT is trained on the teacher modality a… view at source ↗
Figure 2
Figure 2. UMAP of backbone features at inference, teacher (top) and student (bottom), colored by class, for (a) OCT→fundus and (b) CT→CXR. does not guarantee that any particular teacher image and student image will be mapped to similar codes. The method is therefore best understood as disease￾level semantic transfer, not instance-level cross-modal matching. Finally, SSCD recovers only part of the gap between the student-only … view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 8 canonical work pages

  1. [1]

    Advances in neural information processing systems29(2016)

    Aytar, Y., Vondrick, C., Torralba, A.: Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems29(2016)

    2016

  2. [2]

    New England journal of medicine357(22), 2277–2284 (2007)

    Brenner,D.J.,Hall,E.J.:Computedtomography—anincreasingsourceofradiation exposure. New England journal of medicine357(22), 2277–2284 (2007)

    2007

  3. [3]

    In: Proceedings of the European conference on computer vision (ECCV)

    Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of the European conference on computer vision (ECCV). pp. 132–149 (2018)

    2018

  4. [4]

    Advances in neural information processing systems33, 9912–9924 (2020)

    Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems33, 9912–9924 (2020)

    2020

  5. [5]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)

    2021

  6. [6]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873–12883 (2021)

    2021

  7. [7]

    COVIDx CT-3: A Large-scale, Multinational, Open-Source Benchmark Dataset for Computer-aided COVID-19 Screening from Chest CT Images

    Gunraj, H., Tuinstra, T., Wong, A.: Covidx ct-3: A large-scale, multinational, open-source benchmark dataset for computer-aided covid-19 screening from chest ct images. arXiv preprint arXiv:2206.03043 (2022)

    work page Pith review arXiv 2022

  8. [8]

    In:ProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition

    Gupta, S., Hoffman, J., Malik, J.: Cross modal distillation for supervision transfer. In:ProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition. pp. 2827–2836 (2016)

    work page 2016

Show all 23 references
  1. [9]

    arXiv preprint arXiv:1503.02531 (2015)

    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

    2015 arXiv

  2. [10]

    science254(5035), 1178–1181 (1991) 16 D

    Huang, D., Swanson, E.A., Lin, C.P., Schuman, J.S., Stinson, W.G., Chang, W., Hee, M.R., Flotte, T., Gregory, K., Puliafito, C.A., et al.: Optical coherence to- mography. science254(5035), 1178–1181 (1991) 16 D. Imans et al

    1991

  3. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huo, F., Xu, W., Guo, J., Wang, H., Guo, S.: C2kd: Bridging the modality gap for cross-modal knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16006–16015 (2024)

    2024

  4. [12]

    Contemporary mathematics26(189-206), 1 (1984)

    Johnson, W.B., Lindenstrauss, J., et al.: Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics26(189-206), 1 (1984)

    1984

  5. [13]

    In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3967–3976 (2019)

    2019

  6. [14]

    arXiv preprint arXiv:2208.06366 (2022)

    Peng, Z., Dong, L., Bao, H., Ye, Q., Wei, F.: Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366 (2022)

    2022 arXiv

  7. [15]

    Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets (2015),https://arxiv.org/abs/1412.6550

    2015 arXiv

  8. [16]

    arXiv preprint arXiv:1910.10699 (2019)

    Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019)

    2019 arXiv

  9. [17]

    Advances in neural information processing systems30(2017)

    Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems30(2017)

    2017

  10. [18]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Wang, L., Dai, W., Jin, M., Ou, C., Li, X.: Fundus-enhanced disease-aware dis- tillation model for retinal disease classification from oct images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 639–648. Springer (2023)

    2023

  11. [19]

    IEEE Transactions on Medical Imaging44(4), 1711–1722 (2024)

    Wang, L., Qi, C., Ou, C., An, L., Jin, M., Kong, X., Li, X.: Multieye: Dataset and benchmark for oct-enhanced retinal disease recognition from fundus images. IEEE Transactions on Medical Imaging44(4), 1711–1722 (2024)

    2024

  12. [20]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classi- fication and localization of common thorax diseases. In: Proceedings of the IEEE conference on computer vision and patte...

    2017

  13. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, Z., Li, D., Luo, C., Xie, C., Yang, X.: Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8637–8646 (2023)

    2023

  14. [22]

    In: International Conference on Learning Representations

    Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W., Zhang, W., Cui, B., Zhang, M.,Leskovec,J.:Vqgraph:Rethinkinggraphrepresentationspaceforbridginggnns and mlps. In: International Conference on Learning Representations. vol. 2024, pp. 16490–16509 (2024)

    2024

  15. [23]

    arXiv preprint arXiv:1612.03928 (2016)

    Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016)

    2016 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.