{"id":"0ba4fb6a-4c49-4752-b68b-ceb41406ae8a","arxiv_id":"2506.18220","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A ViT-to-CNN knowledge-distillation pipeline for retinal disease classification is reported with 89% student accuracy, but the deployment claim, baseline comparison, and reported numbers are internally unreliable.","lead":"This paper applies a known knowledge-distillation recipe, in which a Vision Transformer teacher teaches a small CNN student, to four-class retinal fundus disease classification and reports an 89% student accuracy with a 97.4% parameter reduction. It is a course-project report whose claimed edge deployment on a Jetson Nano is contradicted by the authors' own quantization note.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported student per-class accuracies are internally contradictory (§5.1.2 vs §5.1.3), so the headline 89%/93%-retention claim is not reproducible from the paper's own tables.","rationale":"The reader's weakest_assumption is the uncontrolled baseline, and that is a legitimate concern: if the non-distilled student was trained with different hyperparameters or for fewer epochs, the claimed 69-point improvement could be an artifact. However, I see an even more load-bearing problem in the paper's internal arithmetic. Section 5.1.2 gives the distilled student per-class accuracies as Cataract 85.6%, DR 91.6%, Glaucoma 68.7%, Normal 97.4%, which are consistent with the recall column of Table 2 and yield an overall accuracy near 89%. Section 5.1.3 then reports a different set of student per-class accuracies — Cataract 85%, DR 82%, Glaucoma 86%, Normal 81% — and uses those to compute '6–8 percentage points lower' than the teacher and '93% retention'. These two sets cannot both be correct for the same 673-image test set. This means the headline result is not reproducible from the paper's own tables and figures. The baseline issue is important but secondary: if the distilled student's 89% figure itself is not trustworthy, the comparison to a 20% baseline is moot. I therefore keep the reader's REJECT verdict, but with a slightly different primary justification: the central quantitative claim is internally inconsistent. I also note the paper's own quantization note concedes the deployment speed measurement is unreliable, further supporting rejection, but the numeric contradiction is the decisive point.","tokens_in":20813,"tokens_out":5914,"duration_ms":65235,"concrete_test":"Recompute overall accuracy and per-class recall from the confusion matrix in Figure 14 and the precision/recall/F1 table in Table 2, then compare with the per-class accuracies stated in §5.1.3. If the two sets disagree beyond rounding (e.g., Normal 97.4% vs 81%, Glaucoma 68.7% vs 86%), the 89% accuracy and 93% retention claims are internally unsupported. This is a purely arithmetic check on the paper's own reported artifacts and should be done before any external re-training experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own numbers for the distilled student on the same test set are mutually inconsistent. In §5.1.2 and Table 2, per-class accuracies (recalls) are Cataract 86%, DR 92%, Glaucoma 69%, Normal 97%, which sum to roughly 596/673 = 88.6% overall and match the claimed 89%. In §5.1.3, the student per-class accuracies are instead Cataract 85%, DR 82%, Glaucoma 86%, Normal 81%, which would give roughly 82–83% overall and are incompatible with Table 2 (e.g., Normal 97% vs 81%, Glaucoma 69% vs 86%). The §5.1.3 values are the ones used to support the '93% retention' claim, so the central quantitative result cannot be verified from the paper's own data. The baseline comparison (§5.1.1, 20% non-distilled accuracy) is a separate concern because training details and seeds are absent, but the internal contradiction is more fundamental: even before asking whether the baseline was controlled, the headline numbers are not self-consistent. The quantization note at the end of §5.1.3 also concedes that the Jetson throughput measurement was unreliable, but the numeric inconsistency alone is sufficient to block the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a cross-architecture knowledge distillation framework that compresses a Vision Transformer (ViT) teacher into a CNN student for four-class retinal fundus image classification, with deployment on an NVIDIA Jetson Nano. The teacher is reported to achieve 92.87% test accuracy, while the student claims 89% accuracy and roughly 93% retention of teacher performance with 97.4% parameter reduction. The proposed framework includes a Partitioned Cross-Attention (PCA) projector, a Group-Wise Linear (GL) projector, and a multi-view robust training scheme, applied to a dataset of 6,727 fundus images. The paper also reports a non-distilled baseline at 20% accuracy and a near-random ROC performance, which the authors attribute to the value of distillation.","tokens_in":21028,"tokens_out":7349,"duration_ms":74448,"significance":"If the reported results were reliable, the paper would demonstrate a practical path to deploying transformer-level diagnostic accuracy on low-power edge hardware, which is clinically relevant for resource-limited settings. The authors address class imbalance with weighting and condition-specific augmentation, and the problem is well motivated. However, the experimental validation contains internal contradictions and missing controls that currently prevent verification of the central claims. The paper ships no code or data, and the deployed quantization results are explicitly acknowledged as invalid. As a result, the significance cannot be assessed without substantial experimental rework.","major_comments":[{"comment":"The per-class accuracies for the distilled student on the test set are mutually inconsistent. In Table 2 (§5.1.2), the recalls are Cataract 0.86, DR 0.92, Glaucoma 0.69, Normal 0.97, which are consistent with the stated 89% global accuracy. In §5.1.3, the text reports student accuracies of Cataract 85%, DR 82%, Glaucoma 86%, and Normal 81%, which yield approximately 82–83% overall and are incompatible with Table 2 (e.g., Normal 97% vs. 81%). The abstract's 89% accuracy matches Table 2, but the '93% retention' claim is supported only by the §5.1.3 numbers. This internal contradiction means the headline results are not reproducible from the paper's own tables. The authors must provide a single, consistent evaluation and clarify which numbers correspond to the final test set.","section":"§5.1.2 vs §5.1.3"},{"comment":"The non-distilled baseline is reported at 20% accuracy (Table 1), which is near chance for a four-class problem and implausibly low for an ImageNet-pretrained CNN fine-tuned on 5,000 images. The paper gives no training details for this baseline: no number of epochs, learning rate, optimizer, augmentation, or initialization beyond 'ImageNet pre-trained weights'. Without evidence that the baseline was trained with the same budget and hyperparameters as the distilled student, the claimed 69-point improvement cannot be attributed to the proposed distillation framework. A controlled ablation with identical training settings is required.","section":"§5.1.1"},{"comment":"The quantization and deployment section contains an explicit admission that 'the model sizes were the same so quantization might not have been saved properly' and that the throughput increase is 'believed to be due to the GPU taking time to warm up.' This invalidates the reported Jetson Nano deployment results, which are a core motivation of the paper. The claim that the student model is 'ready for deployment' is unsupported by the presented measurements. The authors must either provide valid quantization and inference measurements or remove deployment claims from the abstract and conclusions.","section":"§5.1.3 (Quantization)"},{"comment":"The teacher model is described as pretrained with I-JEPA self-supervised learning in the abstract and Section 2, and Section 5.1 reports 20 epochs of self-supervised pretraining. However, Section 3.1.3.2 states that the ViT is 'initialized with ImageNet pre-trained weights', and the augmentation description for pretraining mentions 'contrastive learning' (Section 4.2.3), which is not I-JEPA. This inconsistency between the proposed method and the actual implementation makes it unclear what pretraining procedure was used. The authors should clarify the teacher's exact pretraining and how it aligns with the I-JEPA description.","section":"§3.1.3.2 and §5.1"},{"comment":"The paper omits critical experimental details: the values of the loss weights λ1, λ2, and α, the number of views and crop sizes in the multi-view generator, the learning rate schedule, the batch size, the number of random seeds, and the full training configuration of the baseline model. Without these details, the experiments are not reproducible. Provide a complete hyperparameter table and report results over multiple seeds with standard deviations to assess variance.","section":"General reproducibility"}],"minor_comments":[{"comment":"The phrase 'The teacher model has 97.4 percent more parameters than the student model' is mathematically incorrect; the teacher has approximately 38.5 times the student's parameters, and later text correctly states '97.4% reduction'.","section":"Abstract"},{"comment":"There are two sections labeled '5.1.3' with different content; renumber the sections to avoid confusion.","section":"Section 5.1.3 (duplicate)"},{"comment":"Figure 5 is used for both 'Sample Images (with augmentation)' and 'Self-Supervised Learning Pre-Training'; one should be renumbered. Several figures are referenced by wrong numbers (e.g., Figure 15 is called 'Multi-Class ROC Curve' but the text talks about Figure 14).","section":"Figures"},{"comment":"The PCA attention equation writes A_s = softmax(Q^T K_s / sqrt(d_k)), but the dimensions of Q and K are not defined consistently; this makes the formula ambiguous. Clarify the tensor shapes.","section":"Section 2.3.1"},{"comment":"Some references are incomplete (e.g., the Hugging Face model page has no access date) and the I-JEPA figures are credited to a Medium post rather than the original paper; provide complete citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be a course project report (E6692 at Columbia). The experimental inconsistencies and missing controls are severe, but the underlying methodology is plausible and the application is meaningful. If the authors can redo the evaluation with a consistent test set, a controlled baseline, and valid deployment measurements, a resubmission could be considered. However, if the contradictory numbers cannot be reconciled, the paper should be rejected. The editor may want to check whether the course attribution and data sources meet the journal's standards for novelty and rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a Columbia course-project report that applies Sun et al.'s cross-architecture distillation to retinal fundus images. The topic is sensible, and the I-JEPA-pretrained ViT teacher plus disease-specific augmentations are reasonable tweaks. The authors are also transparent about the quantization failure, which is more than many project reports do. But the central claim — a 2.2M-parameter student retaining 89% accuracy and ~93% of the teacher's performance — collapses when you check the numbers. Section 5.1.2 and Table 2 report student recalls of 86/92/69/97 for Cataract/DR/Glaucoma/Normal, which reproduce the stated 89% weighted accuracy. Section 5.1.3, the section used for the 93% retention claim, gives student accuracies of 85/82/86/81, which average to about 83% and cannot both be true. This is not a minor typo; it is the central quantitative result. The baseline non-distilled student is also suspect: 20% accuracy on a 4-class task from an ImageNet-pretrained CNN is near random, and no training details, seeds, or ablations are given, so the '69-point improvement' is uninterpretable. The quantization note at the end of §5.1.3 concedes that the Jetson throughput measurement was unreliable because the model sizes were the same and the GPU was warming up; that undercuts the deployment evidence, yet the abstract and discussion continue to lean on it. Finally, the abstract calls the framework 'novel' while the body correctly credits Sun et al. for the PCA/GL projectors and adversarial training; what's actually new is the I-JEPA initialization and task-specific augmentations, which are incremental. So the stress-test holds up on reading the paper. There is a kernel of a useful technical report here — the compression numbers are striking, and the clinical setting is meaningful — but as a research preprint it is not reproducible. My recommendation: reject at this stage. If the authors reconcile the per-class numbers, control and document the baseline, and drop the dubious quantization claim, it could become a modest workshop paper. As it stands, I would not send it to a serious referee.","headline":"A course-project application of Sun et al.'s cross-architecture KD to retinal images, but the paper's own per-class numbers contradict, so the 89%/93% retention claim is not currently verifiable.","tokens_in":21615,"tokens_out":3935,"would_cite":false,"duration_ms":40942,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 2.2M-parameter CNN distilled from a ViT hits 89% on four eye-disease classes.","keywords":["cross-architecture knowledge distillation","retinal fundus image classification","vision transformer distillation","CNN student model","edge deployment","Jetson Nano","self-supervised pre-training","medical image analysis"],"falsifier":"Retrain the non-distilled student with the same optimizer, learning-rate schedule, epochs, class weighting, augmentations, and random seeds as the distilled run; if it reaches accuracy well above 20%, the reported gain is not attributable to the PCA/GL projectors. A second check would be to run the frozen teacher's own confusion matrix on the same test set to see whether its Cataract/Glaucoma confusions match the student's, which would indicate the distillation preserved the teacher's biases rather than inventing new ones.","tokens_in":20564,"feed_emoji":"👁","tokens_out":5986,"duration_ms":60224,"temperature":0.7,"pith_summary":"This paper tries to show that a vision-transformer teacher's diagnostic skill for retinal fundus images can be moved into a much smaller CNN student and then run on a low-power edge board. The distilled student reaches 89% test accuracy on four classes (Normal, Diabetic Retinopathy, Glaucoma, Cataract) while keeping about 93% of the teacher's 92.87% performance, and it shrinks parameters from 85.8 million to 2.23 million (a 97.4% reduction) with a model size of 8.79 MB. The authors claim the key to this is a set of cross-architecture distillation components, two projectors and a multi-view adversarial training scheme, rather than the student architecture alone, since the same CNN without distillation scored only 20%.","feed_headline":"2.2M-parameter CNN keeps 93% of ViT eye-disease accuracy","feed_subtitle":"Distilled student hits 89% on four retinal classes and fits in an 8.79 MB edge model.","key_machinery":"The load-bearing mechanism is a combination of three distillation losses grafted onto a frozen ViT teacher. The Partitioned Cross-Attention (PCA) projector gives the CNN its own query/key/value convolutions so its attention maps can be matched to the teacher's via KL divergence; the Group-Wise Linear (GL) projector splits student channels into groups and applies small linear transforms to align CNN features with transformer embeddings in feature space; and a multi-view generator with an adversarial discriminator encourages student features to match the teacher's distribution across augmented crops. The total distillation loss is a weighted sum of the PCA, GL, and adversarial losses, added to a standard classification loss.","core_discovery":"On its own terms, the paper's central claim is that a CNN student trained with the CrossArchitectureKD framework can absorb the global, attention-based reasoning of a ViT teacher and reproduce most of its clinical classification behavior at edge-deployable size. The teacher, a ViT base model pre-trained with I-JEPA self-supervised learning and fine-tuned on 6,727 fundus images, reaches 92.87% test accuracy. The distilled student reaches 89% overall accuracy, with per-class performance of 85.6% for Cataract, 91.6% for Diabetic Retinopathy, 68.7% for Glaucoma, and 97.4% for Normal, and macro precision/recall/F1 of 0.90/0.86/0.88. The paper presents the non-distilled student baseline at 20% accuracy as the contrast that isolates the value of the distillation components.","pith_inferences":["A controlled retraining of the non-distilled student with the exact optimizer, epochs, augmentation, class weighting, and seeds used for the distilled student would test whether the 20%-versus-89% gap is really caused by the distillation components or partly by an undertrained baseline.","The same PCA/GL projector pair could be applied to other cross-architecture transfers in medical imaging, such as ViT-to-CNN distillation for chest X-rays or pathology slides, where global context matters and edge deployment is desirable.","The paper's own note that quantization produced identical model sizes and that the throughput gain may be a GPU warm-up artifact suggests the edge-deployment comparison needs a properly saved INT8 model and averaged repeated runs to measure real latency and memory benefits.","If the 93% retention generalizes beyond this dataset, the natural next clinical question is whether the 6-8 point accuracy drop per class changes the triage threshold or referral rate in a real screening workflow."],"forward_implications":["A 2.2-million-parameter CNN (8.79 MB) can plausibly run four-class retinal triage on memory-limited edge hardware, opening a path to screening where ophthalmologists are scarce.","The 69-point gap between the distilled student (89%) and the non-distilled baseline (20%) implies that the projector-based distillation signal, not the CNN backbone, is what transfers the teacher's diagnostic behavior.","Per-class results suggest the method transfers unevenly: Normal and DR retain high accuracy, while Glaucoma (68.7%) and Cataract (85.6%) are the weak spots and the main source of confusion.","Because the teacher's feature maps and attention patterns are frozen during training, the same framework could be reused with any improved teacher model without retraining the student from scratch."],"supporting_citations":[{"why":"Supplies the cross-architecture knowledge distillation framework (PCA and GL projectors, adversarial training) that the paper adapts to retinal images; it is the central method being transferred.","marker":"[4]"},{"why":"Defines the ViT base model and the pretrained weights the teacher is initialized from before fine-tuning on the fundus dataset.","marker":"[3]"},{"why":"Contextualizes the retinal-disease detection task with an existing foundation model for retinal images, framing the clinical use case and standard for disease detection.","marker":"[7]"}],"fun_headline_variants":["Edge-ready CNN distills ViT's eye-disease IQ at 93% retention","89% accuracy from 2.2M-parameter CNN on Jetson Nano","Tiny CNN inherits ViT's retinal diagnostic skill with 93% fidelity","Tiny CNN learns from ViT to spot eye diseases on budget hardware","Distilling 100M+ ViT into 2.2M CNN for portable eye screening"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's strongest contrast, a 69-point jump from the non-distilled 20% baseline to the distilled 89%, rests on the unstated assumption that the two students were trained with the same budget, initialization, and hyperparameters; the paper gives no training details, seeds, or ablations for the baseline, and 20% accuracy for an ImageNet-pretrained CNN is implausibly close to random.","fun_headline_variants_meta":{"raw":{"variants":["Edge-ready CNN distills ViT's eye-disease IQ at 93% retention","89% accuracy from 2.2M-parameter CNN on Jetson Nano","Tiny CNN inherits ViT's retinal diagnostic skill with 93% fidelity","Tiny CNN learns from ViT to spot eye diseases on budget hardware","Distilling 100M+ ViT into 2.2M CNN for portable eye screening"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2856,"prompt_tokens":989,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1759}},"tokens_in":605,"tokens_out":1867,"duration_ms":13426,"temperature":1.0,"reasoning_tokens":1759,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:28.989199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the non-distilled student with the same optimizer, learning-rate schedule, epochs, class weighting, augmentations, and random seeds as the distilled run; if it reaches accuracy well above 20%, the reported gain is not attributable to the PCA/GL projectors. A second check would be to run the frozen teacher's own confusion matrix on the same test set to see whether its Cataract/Glaucoma confusions match the student's, which would indicate the distillation preserved the teacher's biases rather than inventing new ones.","supporting_citations":[{"cited_title":"Cross-Architecture Knowledge Distillation","cited_arxiv_id":"2207.05273","evidence_quote":"Contextualizes the retinal-disease detection task with an existing foundation model for retinal images, framing the clinical use case and standard for disease detection."}],"review_version":1}