{"id":"26db86cd-942b-432f-9741-fc5d56d777f3","arxiv_id":"2411.15746","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PR-MIM recovers the accuracy lost when masked image modeling throws away tokens by reconstructing them with a lightweight convolution and spreading kept tokens, making pre-training 28% cheaper and 36% lighter without hurting ViT-B/16 performance.","lead":"This paper introduces a way to make masked image modeling pre-training cheaper by reconstructing discarded tokens with a tiny convolution, recovering the accuracy lost when some tokens are skipped. It matters because large vision models can be pre-trained with roughly 28% less compute and 36% less memory without losing downstream performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own coverage worry is statistically negligible at ρe=0.75; the real load-bearing gap is the missing variance information and the missing ViT-L/16 800-epoch baseline behind the 'lossless' claim.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and I agree that the paper requires additional controls before 'lossless' can be taken as established. However, I disagree that the furthest-sampling coverage assumption is the weakest point. A simple back-of-the-envelope calculation shows that 25% unmasked tokens make the 'no xu neighbor in a 7×7 window' event essentially impossible (probability ~10^−7 per window), so the benefit of progressive reconstruction does not depend on the unverified coverage property of furthest sampling; the method would work even with random throwing as far as basic coverage is concerned. The genuine load-bearing uncertainty is statistical: all headline numbers come from single runs, and the exact equality 83.3 vs 83.3 could easily be seed luck. The missing ViT-L/16 800-epoch partial-reconstruction row is a separate, concrete omission that undercuts the model-scaling argument. These concerns are addressable by rerunning with multiple seeds and filling the baseline, so the verdict remains CONDITIONAL rather than moving to REJECT. My proposed test directly measures the effect of seed variance and supplies the missing control, which is the decisive check for the central claim.","tokens_in":94,"tokens_out":11829,"duration_ms":291077,"concrete_test":"Run four independent pre-training seeds for ViT-B/16 on ImageNet-1K at 800 epochs under three conditions: standard MAE (ρd=0), partial reconstruction (ρd=50, random throwing), and PR-MIM (ρd=50). Report mean±std of fine-tuned top-1 accuracy. If PR-MIM's mean falls within 0.1 of MAE's mean with overlapping standard deviations, the 'lossless' claim is statistically supported; if the gap is comparable to the partial-reconstruction gap, it is not. In the same run, add the missing ViT-L/16 800-epoch partial-reconstruction (ρd=50) condition to test the 'larger models are more fragile' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption targets the 7×7-window coverage guarantee for furthest sampling. Under the paper's settings (ρe=0.75, ViT-B/16), this is not load-bearing: a 7×7 window contains 49 token positions, and the probability that it contains no unmasked token xu is 0.75^49 ≈ 1.7×10^−7. Hence even random throwing virtually always leaves every thrown token with at least one unmasked neighbor, so the 'no meaningful loss' failure mode described in Sec. 3.3 essentially never occurs, and furthest sampling is not needed to avoid it. The central claim—lossless performance at ρd=50—therefore does not rest on the unverified coverage property. What it does rest on is the single-run accuracy pair 83.3 vs 83.3 in Table 3; typical seed variance for 800-epoch MAE pre-training is on the order of 0.1–0.3, so an exact match to one decimal place does not establish losslessness. Additionally, Table 3 omits the ViT-L/16 800-epoch partial-reconstruction baseline, despite the paper using that row to argue that larger models are more fragile; without this control, the scaling narrative and its claimed recovery are unsupported at the long schedule.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PR-MIM, a modification of partial-reconstruction masked image modeling (MIM) that recovers the quality lost when a subset of masked tokens is dropped from the reconstruction loss. The method has two components: a progressive reconstruction scheme that reconstructs thrown tokens via a cheap 7×7 depth-wise convolution over the decoder's spatial output, and a furthest-sampling strategy that selects the retained masked tokens to be maximally dispersed. The authors apply PR-MIM to MAE, SimMIM, MFF, TEC, LocalMIM, and GreenMIM, reporting that at a 50% token-throwing ratio, ViT-B/16 pre-trained for 800 epochs matches standard MAE's 83.3% top-1 accuracy while saving 28% FLOPs and 36% memory. Downstream results on ImageNet robustness, iNaturalist, ADE20K segmentation, and COCO detection are also presented.","tokens_in":14613,"tokens_out":5401,"duration_ms":46433,"significance":"If the accuracy claims hold, PR-MIM is a practically valuable and conceptually clean contribution: it gives a near-cost-free way to retain the supervision signal of thrown tokens, is orthogonal to several existing MIM frameworks, and is supported by ablations of the depth-wise convolution kernel size, spatial aggregation design, and sampling strategy. The breadth of the evaluation (six frameworks, three model sizes, and 100/300/800-epoch schedules) is a strength, and the reported FLOPs/memory savings are concrete and reproducible from the tables. The main reservation is that the central 'lossless' claim is established only through single-run accuracy pairs, with no variance information, and one key scaling baseline (ViT-L/16 at 800 epochs under partial reconstruction) is missing. These are fixable by additional experiments or by tempering the abstract's wording.","major_comments":[{"comment":"The 'lossless performance' claim rests entirely on single-run top-1 accuracies such as 83.3 (MAE) versus 83.3 (PR-MIM) for ViT-B/16 at 800 epochs, with the partial-reconstruction baseline at 83.0. Without seed-level variance or at least multiple runs, an exact match to one decimal place does not establish losslessness; typical seed variance for 800-epoch MAE pre-training is on the order of 0.1–0.3 top-1. Please report mean±std over at least three seeds for the headline comparisons (MAE, partial reconstruction, and PR-MIM at 800 epochs for ViT-B/16) and adjust the abstract's wording if the confidence intervals show a residual gap.","section":"Abstract; Table 3"},{"comment":"The scaling claim that larger models are more fragile to partial reconstruction and that PR-MIM recovers the degradation is supported at 100 epochs (82.4 vs 83.2 vs 83.3) but not at 800 epochs, where the table lists only the MAE baseline (85.4) and PR-MIM (85.4) and omits the MAE-with-partial-reconstruction row. Adding the ViT-L/16 800-epoch partial-reconstruction baseline is necessary to support the long-schedule scaling narrative.","section":"Table 3, ViT-L/16 rows"},{"comment":"The comparison with AMT is not apples-to-apples: in the MAE rows PR-MIM uses a 50% throwing ratio versus AMT's 40%, and in the SimMIM rows the total pre-training epochs differ (200 for AMT vs 100 for PR-MIM). The text states that PR-MIM 'outperforms AMT using even a higher throwing ratio,' which is only meaningful if the comparison is made at matched training budgets. Please either run matched schedules and throwing ratios or provide a clear cost-equivalence argument (e.g., FLOPs-matched) that justifies the comparison.","section":"Table 6"}],"minor_comments":[{"comment":"Eq. (1) maximizes the sum of pairwise distances among retained tokens, while Alg. 1 greedily maximizes the minimum distance to already-selected tokens. These are different objectives; the text should explicitly state that Alg. 1 is a heuristic approximation to Eq. (1) and comment on the mismatch.","section":"Sec. 3.3, Eq. (1) and Alg. 1"},{"comment":"The motivation for furthest sampling is that a thrown token may have no retained or unmasked token within the 7×7 window, making the loss meaningless. At the tested throwing ratios this event is exceedingly rare (for ρe=0.75 and ρd=0.5, the probability that all 48 surrounding positions are thrown is below 10^-20), so the paper should quantify the actual frequency of the insufficient-neighbor case to justify this motivation, or present it as a heuristic for improving aggregate supervision quality rather than as a necessity.","section":"Sec. 3.3"},{"comment":"The y-axis 'L2 norm of gradient difference' is undefined. Please specify which gradients are compared (e.g., encoder backbone gradients at a particular layer, or total loss gradients) and how the norm is aggregated over the batch and over training steps before plotting.","section":"Sec. 4.2, Fig. 4"},{"comment":"The SimMIM rows are formatted ambiguously: the 'Epochs' column value appears to apply to both the baseline and the '+PR-MIM' row, but the reader must infer this. Add an explicit entry or footnote for the +PR-MIM row (e.g., '800') to avoid confusion.","section":"Table 4"},{"comment":"Typographical and unit issues: 'we obverse' should be 'we observe' (end of Sec. 3.1 of the introduction); '7.3 · 10−3G floating-point operations per second' should be 'GFLOPs' (operations, not per second); and the GFLOPs value in Sec. 3.2 should be written as 7.3×10−3 GFLOPs for consistency.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the proposed method is likely to interest the MIM community. The missing ViT-L/16 800-epoch partial-reconstruction baseline and the absence of any variance information are the main reasons I cannot recommend acceptance in the current form; both are fixable with additional experiments or a more cautious abstract. I would also encourage the editor to check whether the authors have followed the journal's policy on releasing code, since the abstract promises public code but the submission does not provide a link."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PR-MIM is a genuinely simple efficiency trick that appears to work. The progressive reconstruction module (7x7 depth-wise conv on the decoder output) is cheap and clever, and furthest sampling is a reasonable add-on. The evaluation is unusually broad—MAE, SimMIM, LocalMIM, MFF, TEC, GreenMIM, plus segmentation, detection, and robustness—and the consistent pattern (PR-MIM recovers most or all of the partial-reconstruction degradation) is convincing as a trend.\n\nThe stress-test note is right that the Sec. 3.3 coverage worry is a non-issue at the standard 0.75 mask ratio: the probability that a thrown token has no unmasked neighbor inside a 7x7 window is about 10^-7, so the 'no meaningful loss' failure mode essentially never occurs with random throwing, let alone furthest sampling. The paper's motivation for furthest sampling is therefore weaker than presented, but the ablation shows a small real gain, so it is not a fatal flaw. The fact that furthest sampling is essentially farthest-point sampling without a citation is a minor miss.\n\nThe real soft spots are statistical and editorial. All results are single runs, and 'lossless' rests on an exact match to one decimal (83.3 vs 83.3). Typical seed variance for 800-epoch MAE is on the order of 0.1-0.3, so losslessness is not statistically established. More concretely, Table 3 omits the ViT-L/16 800-epoch partial-reconstruction baseline—the exact row the paper needs to support its claim that larger models are more fragile and that PR-MIM fixes that. That control is essential for the scaling narrative. Also, no code is released yet, which matters for an efficiency paper with implementation-dependent numbers.\n\nFor the right reader (anyone working on efficient MIM or MAE variants), this is worth a serious referee. With code and the missing baseline, I would be happy to see it accepted; as is, it is a conditional. I would send it to review.","headline":"A simple, broadly validated efficiency trick for MAE-style pre-training, with the main caveats being single-run numbers, a missing ViT-L/16 control, and no code yet.","tokens_in":15213,"tokens_out":1627,"would_cite":true,"duration_ms":15082,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that masked-image pre-training can throw away half its masked tokens and still match full accuracy, if the thrown tokens are reconstructed from nearby retained tokens and the retained tokens are spread as far apart as…","keywords":["masked image modeling","partial reconstruction","progressive reconstruction","furthest sampling","self-supervised representation learning","vision transformer","pre-training efficiency","ImageNet-1K"],"falsifier":"Run Algorithm 1 on the $14\\times 14$ token grid of a ViT-B/16 image with a 75% masking ratio and throwing ratios of 25%, 50%, and 65%, and count how many thrown tokens have no retained or unthrown token within a $7\\times 7$ neighborhood; if a non-negligible fraction of thrown tokens is uncovered while the accuracy gain persists, then the mechanism the paper credits for the gain is not the one doing the work.","tokens_in":14132,"feed_emoji":"🧩","tokens_out":9872,"duration_ms":85468,"temperature":0.7,"pith_summary":"Masked image modeling (MIM) pre-training is accurate but expensive, and a known shortcut called partial reconstruction throws away a subset of masked tokens so the decoder has less to process, saving compute but degrading the learned representations. This paper claims to eliminate that degradation with two additions: a progressive reconstruction step in which a tiny $7\\times 7$ depthwise convolution reconstructs the thrown tokens from the retained ones, and a furthest-sampling rule that keeps the retained tokens as spread out as possible. With a 50% throwing ratio on ViT-B/16, the method reports ImageNet-1K accuracy identical to standard MAE while using about 72% of the FLOPs, 64% of the memory, and 75% of the pre-training time. If true, the result means MIM pre-training can be accelerated by roughly a quarter to a third without losing the quality of the learned representations.","feed_headline":"Recover thrown tokens for free, keep masked-image accuracy","feed_subtitle":"A 7-by-7 depthwise convolution reconstructs dropped patches, saving 28% FLOPs at equal ImageNet accuracy.","key_machinery":"The load-bearing object is the progressive reconstruction scheme: after the standard MAE decoder reconstructs the retained masked tokens, a $7\\times 7$ depthwise convolution aggregates the decoded unmasked and retained tokens in spatial layout to fill in the thrown token positions, costing about $7.3\\times 10^{-3}$ GFLOPs. The companion mechanism is furthest sampling, a greedy farthest-point selection over the token grid that keeps retained masked tokens as dispersed as possible so that every thrown token has nearby support inside the $7\\times 7$ window. Together they supply supervision from every masked token while keeping the decoder's input sequence short, which is what preserves representation quality at reduced pre-training cost.","core_discovery":"The paper's central claim is that the performance loss from partial reconstruction comes not from the shorter decoder sequence but from thrown tokens being abandoned, so the fix is to keep every masked token in the loss while preserving the cost saving. PR-MIM does this with a progressive reconstruction scheme: after the standard decoder reconstructs the retained masked tokens, a $7\\times 7$ depthwise convolution rearranges the decoder outputs into spatial form, fills the thrown positions with zeros, and predicts those tokens from neighboring retained and unthrown tokens, at an added cost of only $7.3\\times 10^{-3}$ GFLOPs. A furthest-sampling strategy then selects the retained masked tokens so they are dispersed as widely as possible, giving each thrown token local context to aggregate. At $\\rho_d=50\\%$, the paper reports ViT-B/16 ImageNet-1K fine-tuning accuracy of 83.3% at 800 epochs, matching full MAE, with FLOPs at 0.72x, memory at 0.64x, and pre-training time at 0.75x.","pith_inferences":["Editorial: the paper tests static images, but the same spatial aggregation should transfer to higher-resolution or video inputs, where the $7\\times 7$ convolution cost is essentially independent of sequence length; this is a straightforward extension the paper does not run.","Editorial: furthest sampling's first retained token is chosen randomly, so coverage near image borders is not controlled; a deterministic seed, such as a corner or grid anchor, would remove that variance and is a one-line modification of Algorithm 1.","Editorial: because PR-MIM reconstructs thrown tokens spatially rather than discarding them, it could be combined with attention-based token selection to save even more compute; the paper shows compatibility with many frameworks but does not test that combination.","Editorial: the reported gradient analysis predicts that PR-MIM should also improve convergence under shorter schedules, which could be checked directly by comparing learning curves at 100 versus 400 pre-training epochs."],"forward_implications":["At 50% throwing, PR-MIM reports ImageNet-1K fine-tuning accuracy equal to full MAE for ViT-B/16 (83.3% at 800 epochs) and ViT-L/16 (85.4% at 800 epochs), with decoder FLOPs cut to 0.72x and memory to 0.64x.","The same combination works across MAE, SimMIM, TEC, GreenMIM, LocalMIM, and MFF, so the cost saving is not tied to one decoder design and extends to Swin-based architectures.","At a 65% throwing ratio the reported 100-epoch ViT-B/16 accuracy (81.7%) matches the full-budget baseline, so the savings can be pushed beyond 50% without measured loss.","The method also closes most of the partial-reconstruction gap in robustness benchmarks, iNaturalist transfer, ADE20K segmentation, and COCO detection, with additional cost only of order $10^{-3}$ GFLOPs.","Because thrown tokens remain in the loss, gradient estimates are closer to those of standard MAE, which the paper argues makes stochastic pre-training more accurate at shorter effective epochs."],"supporting_citations":[{"why":"Supplies the asymmetric encoder-decoder baseline, masking recipe, and the full-budget accuracy and cost numbers against which the lossless claim is measured.","marker":"[24]"},{"why":"Represents the prior partial-reconstruction approach that uses temporal redundancy to throw tokens; the paper argues this route is unavailable for static images.","marker":"[53]"},{"why":"Attention-driven token throwing that PR-MIM is compared with in Table 6; the paper argues attention maps add cost and fail on SimMIM.","marker":"[38]"},{"why":"A unified-encoder MIM framework where partial reconstruction previously degraded accuracy; PR-MIM claims to eliminate the drop at 30% throwing.","marker":"[62]"},{"why":"A local multi-scale reconstruction framework used to verify that PR-MIM retains accuracy with HOG targets.","marker":"[52]"},{"why":"A hierarchical Swin-based MIM framework used to show the method is compatible beyond plain ViTs.","marker":"[29]"},{"why":"A target-encoder MIM framework with iBOT-style targets used to test PR-MIM against another reconstruction target.","marker":"[18]"},{"why":"A pixel-reconstruction framework that PR-MIM accelerates and even improves by 0.2% top-1 accuracy.","marker":"[36]"},{"why":"Cited in Sec. 3.3 to support the premise that a masked token should be reconstructed by aggregating local information.","marker":"[6]"}],"fun_headline_variants":["PR-MIM: tiny conv revives discarded tokens, cuts FLOPs 28%","Partial reconstruction without loss: PR-MIM recycles masked tokens","7x7 depthwise conv resurrects dropped patches in masked image modeling","PR-MIM: furthest sampling plus cheap conv matches full MAE","Throw-away tokens no longer wasted: PR-MIM's progressive reconstruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the greedy furthest sampling leaves every thrown token with at least one retained or unthrown token inside the $7\\times 7$ window of the depthwise convolution; Sec. 3.3 admits that a thrown token with no such neighbors cannot receive a meaningful loss, but no coverage statistics or worst-case guarantee are reported for the throwing ratios tested.","fun_headline_variants_meta":{"raw":{"variants":["PR-MIM: tiny conv revives discarded tokens, cuts FLOPs 28%","Partial reconstruction without loss: PR-MIM recycles masked tokens","7x7 depthwise conv resurrects dropped patches in masked image modeling","PR-MIM: furthest sampling plus cheap conv matches full MAE","Throw-away tokens no longer wasted: PR-MIM's progressive reconstruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000981,"raw_usage":{"total_tokens":4169,"prompt_tokens":953,"completion_tokens":3216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3119}},"tokens_in":569,"tokens_out":3216,"duration_ms":21998,"temperature":1.0,"reasoning_tokens":3119,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:56:28.955193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on the $14\\times 14$ token grid of a ViT-B/16 image with a 75% masking ratio and throwing ratios of 25%, 50%, and 65%, and count how many thrown tokens have no retained or unthrown token within a $7\\times 7$ neighborhood; if a non-negligible fraction of thrown tokens is uncovered while the accuracy gain persists, then the mechanism the paper credits for the gain is not the one doing the work.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Supplies the asymmetric encoder-decoder baseline, masking recipe, and the full-budget accuracy and cost numbers against which the lossless claim is measured."},{"cited_title":"Videomae v2: Scaling video masked autoencoders with dual masking","cited_arxiv_id":null,"evidence_quote":"Represents the prior partial-reconstruction approach that uses temporal redundancy to throw tokens; the paper argues this route is unavailable for static images."},{"cited_title":"Good helper is around you: attention-driven masked image modeling","cited_arxiv_id":null,"evidence_quote":"Attention-driven token throwing that PR-MIM is compared with in Table 6; the paper argues attention maps add cost and fail on SimMIM."},{"cited_title":"Simmim: A simple framework for masked image modeling","cited_arxiv_id":null,"evidence_quote":"A unified-encoder MIM framework where partial reconstruction previously degraded accuracy; PR-MIM claims to eliminate the drop at 30% throwing."},{"cited_title":"Masked image modeling with local multi-scale reconstruction","cited_arxiv_id":null,"evidence_quote":"A local multi-scale reconstruction framework used to verify that PR-MIM retains accuracy with HOG targets."},{"cited_title":"Green hierarchical vision transformer for masked image modeling","cited_arxiv_id":null,"evidence_quote":"A hierarchical Swin-based MIM framework used to show the method is compatible beyond plain ViTs."},{"cited_title":"Improving pixel-based mim by reduc- ing wasted modeling capability","cited_arxiv_id":null,"evidence_quote":"A pixel-reconstruction framework that PR-MIM accelerates and even improves by 0.2% top-1 accuracy."},{"cited_title":"Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction","cited_arxiv_id":"2206.00790","evidence_quote":"Cited in Sec. 3.3 to support the premise that a masked token should be reconstructed by aggregating local information."}],"review_version":1}