{"id":"9b9cc1b0-5365-48ae-8671-4d8569270abc","arxiv_id":"2504.14826","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"TripleD distills large image-restoration datasets into a 2% subset selected by entropy-based complexity scores and then fine-tuned with a CNN, reportedly preserving 90-95% of full-dataset performance.","lead":"This paper describes a method to train image restoration models on a small, curated subset of a large training set, using complexity scores and a fine-tuning network. The authors report that using just 2% of the data maintains 90-95% of full-training performance, enabling single-GPU training on high-resolution datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 90–95% claim is not tied to the proposed selection mechanism: no random-2%-subset baseline appears in Tables 1–11, so simple downsampling may explain the results.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing gap: the paper never compares against a random subset of equal size. This is not a stylistic omission; it directly undermines the central claim that complexity-based selection plus distribution matching is what delivers 90–95% of full-training performance at 1–5% data. If a random 2% subset performs equally well, the proposed selection mechanism is unnecessary and the method reduces to a resource-saving trick already known in the literature. The paper does contain real experimental effort: multiple tasks, standard benchmarks, and ablations over CNN depth, downsampling resolution, feature extractors, and distillation ratio. However, those ablations are all conditional on the selected subset and do not test selection against random sampling. Additional issues—unreported manual entropy labels, missing code, and inconsistent loss notation—compound the reproducibility problem, but the missing random baseline alone is sufficient to reject the paper as written. I therefore agree with the reader's REJECT verdict and find no reason to change it.","tokens_in":11955,"tokens_out":2866,"duration_ms":26850,"concrete_test":"Run the exact Table 1 protocol on Rain100L, GoPro, and SIDD with a random 2% subset using the same number of samples, epochs, optimizer, and CNN fine-tuning configuration, and compare final PSNR/SSIM against TripleD and the full-data baseline. Also run the ViT-selected 2% subset with the CNN fine-tuning module removed. If random selection matches TripleD within 0.3 dB, or if removing CNN fine-tuning costs less than 0.5 dB, the claimed contribution of the selection and distribution-matching mechanism is unsupported. For reproducibility, the authors should release the selected subset indices and the complexity-score code.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that TripleD's complexity-based selection plus CNN fine-tuning preserves 90–95% of full-training performance with 1–5% of data. For that claim to hold, the selected subset and the distribution-matching module must contribute beyond arbitrary subsampling. The manuscript never reports training Restormer, PromptIR, or UHDFormer on a random 2% subset under the same budget, nor on the ViT-selected subset without the CNN fine-tuning step. Table 1 compares TripleD against full-data multi-GPU Restormer and full-data single-GPU Restormer, but not against random or uniform 2% sampling. The ablations in Tables 2–8 vary CNN layers, downsampling resolution, distillation proportion p, and feature extractors, but always within the selected subset; they do not test whether the selection itself beats random selection. Table 4 is suggestive: ResNet-50, Mamba, and MLP-Mixer all score within roughly 1.5 dB of ViT (30.50–32.08 dB), indicating the choice of complexity scorer is not critical. Meanwhile, all comparison methods in Tables 1, 9, 10, and 11 are retrained on a small-scale dataset distilled from the respective backbone using the same TripleD pipeline, so the reported gains relative to single-GPU full training could stem from reduced optimization difficulty or from chunked-processing artifacts rather than from the selection or distillation mechanism. Without a random-subset baseline, the central contribution—that complexity-aware distillation is what enables 90–95% performance at 2% data—is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TripleD, a distribution-aware dataset distillation framework for image restoration. TripleD selects a small subset (1-5%) of training pairs using a ViT-16 that predicts image-complexity scores from manually revised ImageNet entropy labels, then fine-tunes the selected subset's feature distribution with an 8-layer CNN and also synthesizes latent-space samples with a diffusion model (SD3). The distilled data are used to train restoration backbones (Restormer, PromptIR, UHDFormer) on a single RTX 3090 GPU. Experiments cover multi-task restoration (deraining, deblurring, denoising), all-in-one restoration, and UHD low-light enhancement, reporting that 2% of the data preserves 90-95% of full-training PSNR/SSIM.","tokens_in":12214,"tokens_out":11351,"duration_ms":94065,"significance":"The problem addressed - reducing the training cost of large image-restoration models - is important, and the paper demonstrates a concrete resource-saving result: training UHDFormer on a single consumer GPU. The experimental scope is broad, with 11 tables and ablations of CNN depth, downsampling resolution, feature extractors, and subset proportion. These are genuine strengths. However, the paper's central attribution claim is not yet supported: no random-subsample baseline is reported, so the results may reflect simple data downsampling rather than the proposed complexity-aware selection. The reliance on a manually annotated scorer and the inconsistent loss definitions also prevent reproducibility. The contribution is potentially significant, but the evidence presented is insufficient as it stands.","major_comments":[{"comment":"The central claim that the proposed selection mechanism enables 90-95% of full-dataset performance with 1-5% of data is never tested against a random or uniform subsample. Table 1 compares TripleD only with multi-GPU and single-GPU full-data training; Tables 9-11 compare models trained with TripleD with pre-trained or retrained baselines. The ablations in Tables 2-8 vary components of the pipeline but always operate on the ViT-selected subset, so they do not isolate the selection step. Please add a random-2%-subset baseline (using the same training budget, optimizer, and gradient-accumulation settings) and a 'first-2%' or uniform-sampling baseline for each task, and report PSNR/SSIM with standard deviations over multiple seeds.","section":"Section 4.2, Tables 1, 9-11"},{"comment":"The paper does not specify the composition and size of the final training set. Section 3.2 selects a subset B of size 1-2%; Sections 3.3-3.4 generate a synthetic dataset Bs; Figure 2 states that 'both the distilled real and synthetic subsets are used to train the restoration network.' The abstract and Figure 1 claim 'only 2% of the data,' but if the model trains on B union Bs, the effective training data may be larger than 2%. Please report the exact number of training pairs used in each experiment and clarify how the 1-5% claim is computed.","section":"Section 3.2-3.4, Figure 2"},{"comment":"The loss used for the CNN fine-tuning step is not consistently defined. Eq. (3) defines LDC as a gradient-matching cosine-similarity loss, but the text after Eq. (4) says 'LDC denotes KL dispersion.' The notation IFa and IFb is also undefined. Please reconcile these definitions, specify the exact objective and its weights, and describe how the CNN is trained jointly with the restoration model.","section":"Section 3.4, Eq. (4)"},{"comment":"The complexity scorer requires manual revision of ImageNet entropy labels ('we manually conduct a significant amount of revision annotation', Section 1), but the paper does not describe what the revision consists of, how many labels were revised, or whether the revised labels are released. This is a load-bearing component because the selection mechanism is the core contribution; without this information the method is not fully specified or reproducible. Please provide the annotation protocol and make the revised labels available.","section":"Section 1 and Section 3.2"},{"comment":"The feature-extractor ablation in Table 4 shows that ResNet-50, Mamba, and MLP-Mixer all achieve PSNR within 1.58 dB of ViT (30.50-32.08 dB), suggesting that the specific choice of complexity scorer has a minor effect. Combined with the absence of a random baseline, this leaves open the possibility that the gains come from the reduced training set size and the CNN fine-tuning rather than from the entropy-based selection. Please discuss this and, if possible, compare against a no-selection (random) control.","section":"Section 4.6, Table 4"},{"comment":"Figure 5 introduces a comparison with 'GSDD' in the diversity analysis, but GSDD is never defined or cited, and no quantitative GSDD results appear in the tables. In addition, Table 7 compares SD3 against StyleGAN2 but not against the selected real subset without synthetic data, so the contribution of the latent distillation module is not established. Please define GSDD and include the missing no-synthetic baseline.","section":"Section 3.3 and Figure 5"}],"minor_comments":[{"comment":"The abstract says '500 savings in computing resources'; this should be written as '500x' or '500% savings' to be meaningful.","section":"Abstract"},{"comment":"In Table 3, the row labeled 'Ours' should be labeled '128x128', which is the downsampling resolution used by TripleD; the current label is not a resolution.","section":"Table 3"},{"comment":"Table 9's caption says 'pre-trained PromptIR model without TripleD' while the text says 'retrained'; please align the wording with the actual protocol.","section":"Table 9 caption"},{"comment":"Figure 8 lists per-image PSNR values (e.g., 'Ours 43.80') that differ from the dataset-average PSNR in Table 10 (35.01); please state explicitly that the figure shows per-sample PSNR, not the benchmark average.","section":"Figure 8"},{"comment":"Eq. (2) uses class-based IPC notation (M = C x IPC) that is not meaningful for restoration datasets without classes; please define how the subset size is determined for regression-style restoration tasks.","section":"Section 3.1, Eq. (2)"},{"comment":"Section 4.1 says 'dynamically selecting 2% of the data in each epoch,' but Section 3.2 describes a one-time static selection of the top 1-2% by entropy; please clarify whether the selection changes across training epochs.","section":"Section 4.1 vs. Section 3.2"},{"comment":"The text refers to YOLOv5 in Section 4.7, but reference [10] is YOLOX; please correct the citation or the reference.","section":"Reference [10]"},{"comment":"Table 1 lists identical PSNR/SSIM values for RealBlur-J and RealBlur-R under each method; please verify whether this is a copy-paste error.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an important practical problem and has a broad experimental sweep, but the missing random-subset control is a fundamental gap. I am recommending major revision only because the missing experiments and clarifications are in principle supplyable; if the authors cannot provide random-subset baselines that show a clear advantage for complexity-aware selection, the paper should be rejected. The writing also needs substantial editing before it can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper applies dataset distillation to image restoration using a ViT-computed complexity score to pick a 2% subset and a CNN to align feature distributions. The package is coherent, but the central claim — that the complexity-aware selection is what buys the 90–95% — is untested, because no random-subset baseline appears anywhere in Tables 1–11. That is the load-bearing flaw.\n\nWhat's new and good: applying distillation to restoration with a two-stage curriculum (easy samples first, harder later) is a sensible adaptation, and the paper does cite prior super-resolution distillation (ref [8]) rather than pretending the field is empty. The experiments cover many tasks: deraining, deblurring, denoising, all-in-one, and UHD low-light, and the ablation tables do check a few components. Notably, removing CNN fine-tuning drops from 32.08 to 30.75, and the feature-extractor ablation (Table 4) shows ViT is not special — MLP-Mixer scores 32.05. That last result actually undercuts the centrality of the ViT scorer.\n\nSoft spots, in proportion. The missing random baseline is the big one. The stress-test note is right: all comparison methods are retrained on the same distilled subset, so the comparisons cannot tell you whether selection matters. If random 2% does just as well, the method reduces to \"train on fewer patches,\" which is not a distillation contribution. The second issue is reproducibility: the manual entropy-label revision is described vaguely, no code or data is shipped, and the loss definitions are internally inconsistent — Eq. (3) defines gradient-matching cosine similarity but the text calls it KL dispersion. Third, the 1% result in Table 5 (25.33 dB) contradicts the abstract's \"1~5% of the data\" claim; only 2% and up approach 90%. These are not trivial nits; they stack into a paper that reads like a promising workshop draft rather than a finished submission.\n\nWho this is for: practitioners who want to train restoration models on small budgets might find the resource numbers interesting, but they would be unwise to adopt the recipe without a random baseline. A serious referee could fix this paper — adding random and selection-without-CNN baselines, cleaning up the losses, shipping code — so it deserves review, but the current version should not be accepted as-is.\n\nRecommendation: send it to review with a request for major revision, and make sure the reviewers specifically demand the missing baselines.","headline":"A plausible distillation pipeline for image restoration that is undone by the absence of a random-subset baseline; the central selection mechanism is unverified.","tokens_in":12771,"tokens_out":3006,"would_cite":false,"duration_ms":26302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TripleD claims that training image restoration models on a 1-5% complexity-selected subset preserves about 90-95% of full-data quality and fits 4K training on one consumer GPU in under eight hours.","keywords":["dataset distillation","image restoration","subset selection","vision transformer","entropy-based complexity","latent diffusion","distribution matching","efficient training"],"falsifier":"Train the same restoration network on a randomly chosen 2% subset, run it through the same CNN fine-tuning and diffusion synthesis, and compare PSNR/SSIM on Rain100L and GoPro; if the random subset matches or beats the complexity-selected one, the entropy scorer is not doing the work the paper assigns it.","tokens_in":11716,"feed_emoji":"🖼️","tokens_out":7985,"duration_ms":68325,"temperature":0.7,"pith_summary":"This paper tries to establish that dataset distillation, previously applied almost entirely to classification, works for image restoration, a dense pixel-prediction task with no classification margin to absorb errors. It proposes a pipeline named TripleD that scores image complexity with a pretrained vision Transformer, keeps the top 1-2% of samples, synthesizes extra compact training pairs in latent space with a diffusion model, and fine-tunes the small dataset's feature distribution with an 8-layer CNN. On deraining, deblurring, denoising, all-in-one restoration, and 4K low-light benchmarks, training on this tiny subset reaches about 90-95% of full-dataset PSNR/SSIM and fits on one consumer GPU in under eight hours. The practical reason to care is that, if correct, researchers can iterate on large restoration models without large GPU clusters, and the framework points a way to distill other dense prediction datasets.","feed_headline":"One GPU, 2% of data, ~90% of full training quality","feed_subtitle":"TripleD trains restoration models on 1-5% of the data, cutting compute so 4K fits on one consumer GPU.","key_machinery":"The load-bearing object is the complexity-selected, distribution-aligned mini-dataset produced by three cooperating components. A ViT-16 with a sigmoid head, trained on ImageNet with manually revised entropy labels, assigns each downsampled image a complexity score in [0,1], and the top 1-2% form the real sub-dataset. A diffusion model generates synthetic degraded/clean pairs whose latent codes match the selected real samples, replacing the GAN used in prior latent distillation. An 8-layer CNN over 3x3 convolutions then adjusts the feature distribution of the combined small set, supervised by L2 and KL terms plus a gradient-matching loss, while the restoration model trains. The two-stage curriculum, easy low-complexity samples first and harder ones later, organizes this small dataset into a training schedule.","core_discovery":"The central claim is that a 2% subset of training images, chosen by a ViT-based complexity score and adjusted in feature space by a CNN, is enough to train restoration networks to near full-data performance: about 90-95% in PSNR/SSIM across ten benchmarks including Rain100L deraining, GoPro deblurring, SIDD denoising, and the UHD-LOL4K low-light set. The paper also reports that this holds for all-in-one models that must handle several degradations at once, and that the 2% operating point is near the knee of the curve, since raising the subset to 5% or 10% yields only small gains. The mechanism is not simple subsampling: the distilled subset is augmented by diffusion-synthesized pairs and passed through an 8-layer CNN trained jointly with the restoration network, so that the feature distribution of the small set is pulled toward the original dataset.","pith_inferences":["The authors leave implicit that the entropy scorer's contribution is untested against random sampling; a natural next experiment is a random 2% subset passed through the same CNN and diffusion pipeline, which would separate the selection mechanism from the distribution-alignment machinery.","The manual revision of entropy labels makes the scorer closer to a learned perceptual-complexity measure than to true information-theoretic entropy, which would explain why cosine distance, standard deviation, and KL divergence performed poorly as complexity scores.","Because distribution matching happens in latent space rather than task-specific pixels, the recipe could transfer to other dense regression problems such as super-resolution, inpainting, or depth estimation.","The easy-to-hard curriculum is a testable hypothesis the ablations do not isolate: with the same 2% subset, shuffling the order would show whether the ordering itself contributes or only the selection matters."],"forward_implications":["A single mid-range GPU can train restoration transformers that currently require multi-GPU setups, because the 2% subset keeps memory and training steps low.","UHD (4K) low-light restoration models can be trained in under eight hours instead of days, making dataset-scale iteration practical for small labs.","The same distilled-subset recipe works across degradation types such as rain, blur, noise, haze, and low light, so users do not need a separate distillation scheme for each task.","Because gains saturate between 2% and 10% subset size, the 2% point is close to the optimal trade-off between fidelity and training cost.","Models trained on the distilled subset still transfer to downstream object detection and segmentation, so reduced training data does not lock out downstream users."],"supporting_citations":[{"why":"Supplies the dataset-distillation objective and gradient-matching formulation that TripleD adapts to image restoration.","marker":"[32]"},{"why":"Provides the pre-trained vision Transformer used to score image complexity for subset selection.","marker":"[9]"},{"why":"Contributes the generative latent distillation idea, where synthetic latents are matched in the generator's latent space; TripleD replaces the GAN with a diffusion model.","marker":"[5]"},{"why":"Defines the Restormer architecture and full-dataset baselines whose PSNR/SSIM TripleD aims to approach with 2% data.","marker":"[37]"},{"why":"Rain100L and Rain100H supply the deraining benchmarks used for most ablations and the main quantitative comparisons.","marker":"[35]"},{"why":"GoPro supplies the motion-deblurring benchmark and baseline that demonstrate the 2%-subset gain.","marker":"[19]"},{"why":"SIDD supplies the real denoising benchmark used to show the method preserves most full-data performance on clean noise.","marker":"[1]"}],"fun_headline_variants":["2% of data distills to 90% performance for image restoration","TripleD: one GPU, tiny data, near-full restoration quality","Distilled 2% dataset trains 4K restoration on a single GPU","Image restoration distilled: 2% data, one GPU, 8 hours","Small data, big restoration: TripleD distills 2% to 90% quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains depend on the assumption that a vision Transformer's entropy-style complexity score really identifies the most training-worthy images, but no random 2% subset is tested as a baseline, so the selection mechanism could be contributing little beyond aggressive downsampling.","fun_headline_variants_meta":{"raw":{"variants":["2% of data distills to 90% performance for image restoration","TripleD: one GPU, tiny data, near-full restoration quality","Distilled 2% dataset trains 4K restoration on a single GPU","Image restoration distilled: 2% data, one GPU, 8 hours","Small data, big restoration: TripleD distills 2% to 90% quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1445,"prompt_tokens":988,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":604,"tokens_out":457,"duration_ms":4302,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:39:30.888821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same restoration network on a randomly chosen 2% subset, run it through the same CNN fine-tuning and diffusion synthesis, and compare PSNR/SSIM on Rain100L and GoPro; if the random subset matches or beats the complexity-selected one, the entropy scorer is not doing the work the paper assigns it.","supporting_citations":[{"cited_title":"Dataset distillation via optimal transport","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset-distillation objective and gradient-matching formulation that TripleD adapts to image restoration."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained vision Transformer used to score image complexity for subset selection."},{"cited_title":"Generalizing dataset dis- tillation via deep generative prior","cited_arxiv_id":null,"evidence_quote":"Contributes the generative latent distillation idea, where synthetic latents are matched in the generator's latent space; TripleD replaces the GAN with a diffusion model."},{"cited_title":"Restormer: Efficient transformer for high-resolution image restoration","cited_arxiv_id":null,"evidence_quote":"Defines the Restormer architecture and full-dataset baselines whose PSNR/SSIM TripleD aims to approach with 2% data."},{"cited_title":"Deep joint rain detection and removal from a single image","cited_arxiv_id":null,"evidence_quote":"Rain100L and Rain100H supply the deraining benchmarks used for most ablations and the main quantitative comparisons."},{"cited_title":"Deep multi-scale convolutional neural network for dynamic scene deblurring","cited_arxiv_id":null,"evidence_quote":"GoPro supplies the motion-deblurring benchmark and baseline that demonstrate the 2%-subset gain."},{"cited_title":"A high-quality denoising dataset for smartphone cameras","cited_arxiv_id":null,"evidence_quote":"SIDD supplies the real denoising benchmark used to show the method preserves most full-data performance on clean noise."}],"review_version":1}