{"id":"2c2239c9-f10f-49db-a706-b9bf620a2785","arxiv_id":"2507.12157","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"TGDA trains fine-grained-recognition students from scratch using attention maps and soft labels from a fine-tuned, pretrained teacher, reporting large gains on three benchmarks.","lead":"This paper introduces TGDA, a two-stage training recipe that uses a fine-tuned teacher to generate attention-based crops and soft labels so that a fine-grained classifier can be trained from random weights. The authors report large accuracy gains on Aircraft, Cars, and CUB while using far fewer parameters and less data than pretrained baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'from scratch' claim is contradicted by the pretrained teacher: Section 4 fine-tunes ResNet-101 teachers for 50 epochs, so every TGDA result may simply distill ImageNet knowledge through the teacher.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the teacher is fine-tuned from an ImageNet-pretrained initialization, and no from-scratch teacher experiment is provided. I agree with this reading. The paper's abstract, introduction, and conclusion all make claims about eliminating reliance on pretraining and training systems 'entirely from scratch,' while Section 4 explicitly describes fine-tuning the teacher. Since the teacher's outputs drive both the augmentations and the distillation loss in Stage 2, the student's success cannot be attributed to a pretraining-free pipeline. This is an internal inconsistency between the claimed contribution and the experimental protocol, not a disagreement with an external consensus. The proposed concrete test—retraining the teacher from random initialization—would settle the question directly. If the from-scratch teacher yields comparable student accuracy, the claim could be salvaged; if not, the paper's headline claim fails, although the engineering recipe for training compact students from scratch may still have value. The existing verdict of REJECT is therefore appropriate, and my read does not change it.","tokens_in":9874,"tokens_out":3429,"duration_ms":41614,"concrete_test":"Retrain the CAL teacher from random initialization on each of Aircraft, Cars, and CUB, using the same target dataset and a budget comparable to the student (800 epochs with AdamW, or at least enough for convergence), then run the full Stage-2 TGDA pipeline with that from-scratch teacher under the same settings as Tables 1 and 3. If student accuracy drops by more than the claimed margins over the pretrained-teacher baselines (roughly 2 points at 448x448 or 10 points at 128x128), the central 'from scratch' claim is falsified; if accuracy holds within noise, the claim survives. Run three seeds and report mean plus/minus standard deviation to also address the missing error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that TGDA 'demonstrates that high-performance FGIR systems can be trained entirely from scratch' and 'eliminates the reliance on large-scale pretraining.' This is not supported by the methodology. In Section 4, the teacher is described as follows: 'ResNet-101 teachers are fine-tuned for 50 epochs with SGD based on the CAL recipe.' Fine-tuning, in this context, means starting from an ImageNet-pretrained checkpoint; the paper never states otherwise and never trains a teacher from random initialization. Stage 2 (Section 2.2) then uses that teacher's soft labels and part attention maps as the only learning signal beyond the ground-truth labels: L_total = alpha * L_CE(I_org) + beta * [L_KD(I_org) + L_KD(I_aug)]. The student's parameters are randomly initialized, but the knowledge entering through the teacher is not. The teacher's attention maps select the crops, and its soft predictions define the distillation targets. Therefore, the reported gains—up to 23% over DSSD at 128x128 and up to 2.4% over CEKD at 448x448—are confounded: they could be entirely a transfer of ImageNet-trained feature and attention priors through the teacher. If so, the paper has not shown that pretraining can be replaced; it has shown that pretraining can be moved into a separate model. This is an internal inconsistency between the claim ('trained entirely from scratch', 'without any reliance on external pretrained weights') and the experimental setup, not merely a disagreement with community consensus. The absence of a from-scratch teacher ablation, error bars, and released code compounds the problem, but the decisive issue is the pretrained teacher.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TGDA, a two-stage knowledge-distillation framework for fine-grained image recognition (FGIR) that aims to train student models from random initialization using a fine-grained-aware teacher to generate attention-based augmentations and soft supervision. The authors introduce two task-specific backbone families, LRNets for low-resolution FGIR and ViTFS for hardware-efficient vision transformers, and report that these from-scratch students outperform pretrained baselines on FGVC-Aircraft, Stanford-Cars, and CUB-200-2011 in both high- and low-resolution settings.","tokens_in":10120,"tokens_out":8947,"duration_ms":96180,"significance":"If the claims were fully supported, the work would be significant: it would challenge the field's reliance on ImageNet pretraining for FGIR and open the door to task-specific architectures. The paper has several strengths: a clear two-stage framework, a thoughtful reassessment of backbone design for low-resolution inputs, hardware-aware transformer modifications, and extensive comparisons across three benchmarks. However, the central 'from scratch' claim is undercut by the experimental design, as detailed in the major comments, and the present version does not establish the claimed independence from pretraining.","major_comments":[{"comment":"The central claim that TGDA enables training 'from scratch' without reliance on pretrained weights is contradicted by the method itself. Section 4 states that 'ResNet-101 teachers are fine-tuned for 50 epochs with SGD based on the CAL recipe'; fine-tuning conventionally starts from an ImageNet-pretrained checkpoint, and the paper never states otherwise or trains a teacher from random initialization. In Stage 2 (Section 2.2), the student's only supervision beyond ground-truth labels is the teacher's soft predictions and attention maps, so the teacher's pretrained knowledge is propagated into the from-scratch student. The claimed independence from pretraining is therefore not demonstrated; the results could be a transfer of ImageNet-trained features through the teacher. The manuscript must either train teachers from scratch and re-run all experiments, or explicitly reframe the contribution as 'from-scratch student with a pretrained teacher,' which would substantially reduce the novelty claimed in the title and abstract.","section":"§1, §2.2, §4"},{"comment":"The ablation study does not isolate the contribution of the teacher's pretrained knowledge. The rows without 'Two-Stage' report near-random accuracies (e.g., 1.7% on Aircraft, 0.6% on Cars, 10.1% on CUB for the no-component baseline), and the jump to 89.0% on Aircraft when 'Two-Stage' is added is attributed to the two-stage training design; however, the 'Two-Stage' condition uses a fine-tuned, pretrained teacher. The table also lacks an explicit 'Teacher' column, so it is unclear whether the non-two-stage rows even include a teacher. As written, Table 4 confounds the effect of a separate teacher with the effect of pretrained initialization, and it cannot support the claim that PAM and two-stage training are sufficient for from-scratch success.","section":"§5.3, Table 4"},{"comment":"The comparisons against pretrained baselines are not apples-to-apples. TGDA's student receives supervision from a ResNet-101 teacher that has been fine-tuned on the target dataset from ImageNet initialization, whereas the baselines are standard backbones directly fine-tuned. Thus the reported gains (up to 2.4 points over CEKD at 448×448 and up to 23 points over DSSD at 128×128) may reflect distillation from a much stronger pretrained teacher rather than a genuine advantage of from-scratch training. A control experiment in which the teacher is trained from scratch—or, at minimum, in which the student is trained with a from-scratch teacher—is needed to support the paper's conclusions. Without such a control, the headline results are consistent with the alternative hypothesis that pretraining has merely been moved into the teacher.","section":"§5.1, Tables 1 and 2; §5.2, Table 3"}],"minor_comments":[{"comment":"Reference [6] is cited for CUB-200-2011, but [6] is an empirical study of self-supervised vision transformers; the correct CUB citation appears to be [25].","section":"§4, References"},{"comment":"The parameter counts for LRNet versus standard ResNets are not reported in the table; the text refers to Figure 1, but the exact reductions should be stated to support the '20.6x less parameters' claim.","section":"§5.2, Table 3"},{"comment":"The loss weight α is not reported; please provide the value and, ideally, a sensitivity analysis for β and temperature.","section":"§4"},{"comment":"Table 4 would benefit from an explicit 'Teacher' column and from error bars or multiple seeds, given the huge variance between rows.","section":"§5.3, Table 4"},{"comment":"Minor wording issues: 'Tab. 3 present' should be 'Tab. 3 presents,' and 'orders of magnitude less data' should be quantified (Table 2 shows 6K images for ViTFS-T, which is roughly three orders of magnitude less than ImageNet-21k, not simply 'orders of magnitude' without specifying the base).","section":"§5.2, Abstract"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim of training FGIR systems entirely from scratch is not supported by the experiments as described, because the teacher is fine-tuned from a pretrained checkpoint and no from-scratch teacher experiment is provided. The missing experiment is essential rather than cosmetic; without it, the paper cannot be distinguished from a standard knowledge-distillation method that transfers pretrained knowledge to a smaller student. I recommend that the editor consider this a load-bearing flaw that is not addressable by text revision alone. The incorrect citation of the CUB dataset (reference [6] instead of [25]) may also indicate a need for careful proofreading in any revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the actual deliverable is a training recipe that lets a randomly initialized student beat ImageNet-pretrained baselines on three FGIR benchmarks; that is a real result if the numbers hold. Second, the headline claim—high-performance FGIR without reliance on pretraining—does not hold as stated. The teacher is a CAL-trained ResNet-101 that starts from ImageNet weights. Section 4 says the teachers are 'fine-tuned for 50 epochs,' and the student's supervision beyond ground-truth labels comes entirely from that teacher's soft labels and part-attention maps. So TGDA moves pretraining into the teacher rather than removing it. The stress-test note is correct, and I do not think the paper has a response short of adding a from-scratch teacher ablation.\n\nWhat is genuinely new: the two-stage design, with a frozen part-attention teacher generating attention crop/drop views for a student trained from random init, is a sensible combination of WS-DAN/CAL, Hinton-style KD, and DADKD-style augmentation. The LRNet backbone—delayed downsampling, five blocks—and ViTFS—BatchNorm, conv stem, register tokens—are plausible architecture modifications. The reported low-resolution gains are large: up to 23 points over DSSD, and the ViTFS-T result (matching a ViT-B/21k with 15x fewer parameters on 6k images) is striking if true. These would be a solid empirical contribution even without the 'from scratch' framing.\n\nThe soft spots are real but mostly fixable. Single runs, no error bars, no released code. The ablation in Table 4 is confusing: the rows without PAM or Two-Stage show accuracies between 0.6 and 27.1, which looks like a broken baseline or a very different experimental setup; the reader cannot tell what 'neither component' means. That table needs a rewrite. The deeper issue is conceptual: the paper conflates 'student trained from random init' with 'system trained from scratch.' That is a dependency, not a formal circularity, but it undercuts the abstract's promise.\n\nWho this is for: people working on efficient FGIR, low-resolution recognition, or distillation with scarce data. They will get a useful recipe and backbone ideas worth testing. The paper deserves a serious referee, not a desk reject, but it needs a from-scratch teacher experiment, error bars, and a clearer ablation before acceptance. I would send it out with a request for exactly those additions.","headline":"Useful distillation recipe and backbone designs, but the 'from scratch' claim does not survive contact with the pretrained teacher.","tokens_in":10775,"tokens_out":1911,"would_cite":false,"duration_ms":21045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"High-performance fine-grained image recognition can be trained entirely from scratch, the paper argues, using teacher-guided augmentation that distills part-attention knowledge into a randomly initialized student.","keywords":["fine-grained image recognition","training from scratch","knowledge distillation","data augmentation","part attention maps","low-resolution recognition","vision transformers","efficient inference"],"falsifier":"Train the TGDA pipeline with a teacher that is itself trained from random initialization on the target dataset only, and compare student accuracy; if it falls to the low single-digit levels of the no-teacher ablation, the reported gains come from the teacher's ImageNet-pretrained knowledge rather than from the mechanism itself.","tokens_in":9622,"feed_emoji":"🔍","tokens_out":5597,"duration_ms":60728,"temperature":0.7,"pith_summary":"This paper sets out to show that fine-grained image recognition, which must tell apart near-identical bird species, car models, or aircraft, does not require backbones pretrained on ImageNet. It proposes Teacher-Guided Data Augmentation (TGDA), a two-stage process in which a teacher trained with part-attention maps first generates augmented crops of each image, and the frozen teacher's soft predictions then supervise a student trained from random initialization. The claim is that TGDA-trained students match or beat pretrained state-of-the-art methods on three benchmarks, and that this unlocks backbones designed specifically for the task: LRNets for low-resolution input and ViTFS, a batch-normalized vision transformer family. If true, practitioners could drop pretraining entirely, saving data and compute and enabling deployment-friendly architectures.","feed_headline":"From-scratch training beats ImageNet-pretrained on fine-grained vision","feed_subtitle":"Teacher-guided augmentation lets small task-specific models match or surpass pretrained baselines with far less data.","key_machinery":"The load-bearing mechanism is the Part Attention Module (PAM), a learnable convolutional unit that produces multiple attention maps highlighting semantically meaningful object regions. These maps are used in two data-aware augmentations: attention cropping, which zooms into discriminative regions, and attention dropping, which masks them to force complementary feature learning. The same maps, combined with the teacher's soft predictions on original and augmented views, give the student rich supervision without any pretrained weight initialization. The framework is two-stage: the teacher is trained first with its PAM, then frozen while supervising the randomly initialized student; prior one-stage co-training, in which an untrained student also teaches the teacher, is shown to be harmful.","core_discovery":"The paper's central claim is that high-performance fine-grained image recognition can be trained entirely from scratch, provided the student is guided by a teacher that has learned where discriminative parts are. In TGDA, a fine-grained-aware teacher built on a part attention module produces attention maps; these maps drive attention cropping and attention dropping to create diverse augmented views, and the teacher's soft labels on both original and augmented images form the distillation signal for a randomly initialized student. On ResNet-18/34 at 448x448, TGDA outperforms the previous distillation-based state of the art by up to 2.4 accuracy points; with the dedicated LRNet backbone at 128x128 it beats prior low-resolution methods by up to 23 points while using up to 20.6x fewer parameters. The paper also reports that ViTFS-T, trained with TGDA on only a few thousand images, matches a ViT-B pretrained on ImageNet-21k with 15.3x fewer trainable parameters.","pith_inferences":["Inference: because the teacher is itself ImageNet-pretrained, the honest reading of \"from scratch\" is narrow: TGDA replaces direct pretraining of the deployable student with transfer of pretrained knowledge through the teacher, and the paper does not demonstrate that a from-scratch teacher would work as well.","Inference: the method's dependence on part attention suggests it should transfer best to domains where classes differ in localized parts; for categories that differ mainly in global appearance, attention-based cropping and dropping may add less.","Inference: a natural extension is to test TGDA on medical or agricultural fine-grained tasks, where ImageNet pretraining is often mismatched to the imaging modality and a task-designed backbone could show larger gains."],"forward_implications":["Fine-grained recognition systems can be built without ImageNet pretraining, so backbones can be chosen for deployment constraints such as resolution, parameters, and hardware support rather than for what pretrained weights happen to exist.","Small task-specific backbones like LRNets become practical for low-resolution input, where ImageNet-pretrained architectures lose the fine detail needed to separate near-identical classes.","Batch-normalized vision transformers like ViTFS can be trained from random initialization, making transformer inference faster on hardware that lacks optimized LayerNorm support.","The reported gains come with far less training data and compute, lowering the barrier to entry for resource-constrained labs and applications."],"supporting_citations":[{"why":"Supplies the knowledge distillation loss form (KL divergence between teacher and student soft outputs) used in TGDA's Stage 2.","marker":"[10]"},{"why":"Introduces the part attention module and attention-based cropping/dropping that TGDA's teacher uses to generate augmented views.","marker":"[11]"},{"why":"DSSD is the prior state of the art in low-resolution FGIR that TGDA and LRNet are compared against and beat by up to 23 points.","marker":"[16]"},{"why":"Provides the counterfactual attention learning recipe used to fine-tune the ResNet-101 teacher and is another fine-grained-aware teacher architecture.","marker":"[19]"},{"why":"CEKD is the previous distillation-based data augmentation approach for FGIR that TGDA outperforms at high resolution and uses in its ablation.","marker":"[31]"},{"why":"Defines class activation maps, the static CAM-based augmentation signal that TGDA replaces with learnable part attention maps.","marker":"[36]"}],"fun_headline_variants":["From-scratch fine-grained recognition beats pretrained baselines","Teacher-guided augmentation matches ImageNet-pretrained with less data","Scratch-trained models surpass pretrained on fine-grained tasks","No pretraining needed: From-scratch FGIR with teacher guidance","TGDA: Train fine-grained vision from scratch, beat pretrained"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that TGDA removes reliance on pretraining assumes that the pretrained knowledge entering through the teacher, which is fine-tuned from an ImageNet-pretrained ResNet-101, does not count as reliance on pretraining.","fun_headline_variants_meta":{"raw":{"variants":["From-scratch fine-grained recognition beats pretrained baselines","Teacher-guided augmentation matches ImageNet-pretrained with less data","Scratch-trained models surpass pretrained on fine-grained tasks","No pretraining needed: From-scratch FGIR with teacher guidance","TGDA: Train fine-grained vision from scratch, beat pretrained"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1440,"prompt_tokens":1047,"completion_tokens":393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":663,"tokens_out":393,"duration_ms":4198,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:52:56.345306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the TGDA pipeline with a teacher that is itself trained from random initialization on the target dataset only, and compare student accuracy; if it falls to the low single-digit levels of the no-teacher ablation, the reported gains come from the teacher's ImageNet-pretrained knowledge rather than from the mechanism itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the part attention module and attention-based cropping/dropping that TGDA's teacher uses to generate augmented views."},{"cited_title":"Dynamic semantic structure distillation for low-resolution fine-grained recognition","cited_arxiv_id":null,"evidence_quote":"DSSD is the prior state of the art in low-resolution FGIR that TGDA and LRNet are compared against and beat by up to 23 points."},{"cited_title":"Counterfactual Attention Learn- ing for Fine-Grained Visual Categorization and Re-identification","cited_arxiv_id":null,"evidence_quote":"Provides the counterfactual attention learning recipe used to fine-tune the ResNet-101 teacher and is another fine-grained-aware teacher architecture."},{"cited_title":"CEKD:Cross ensemble knowledge distillation for augmented fine-grained data","cited_arxiv_id":null,"evidence_quote":"CEKD is the previous distillation-based data augmentation approach for FGIR that TGDA outperforms at high resolution and uses in its ablation."}],"review_version":1}