{"id":"81d704e3-b792-48b1-8104-6c18805bdb93","arxiv_id":"2412.10730","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A cluster-masked autoregressive plus multi-task pretraining framework lifts xLSTM vision backbones to 83.4% top-1 accuracy on ImageNet-1K, roughly one point above the ViL-B baseline.","lead":"This paper introduces MAL, a pretraining recipe for xLSTM image models that groups image patches into clusters, predicts them sequentially, and adds depth and segmentation tasks during pretraining. On ImageNet-1K classification, the base model reaches 83.4% top-1 accuracy, about one point above the prior xLSTM backbone and several points above efficient attention and Mamba baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 83.4% is tied to a 20%-masking inpainting variant that contradicts the causal cluster-autoregressive objective in Eq. (3); without code or a precise mask construction, the central mechanism is unverified.","rationale":"The reader's verdict is CONDITIONAL, and my concern reinforces the need for a condition without moving the verdict to a harsher one. The reader's weakest_assumption focused on hyperparameters being selected on the same validation set, which is a legitimate statistical concern. My stronger, mechanism-level concern is that Table 6's best configuration is not the objective in Eq. (3): causal next-cluster prediction has no masking ratio, while Table 6 explicitly uses 1%-70% masking and calls high ratios 'inpainting.' This is an internal inconsistency rather than merely a disagreement with community practice. If the winning variant is random-token inpainting with 20% masking, then the paper's title and abstract overstate the role of autoregressive cluster masking, and the method's novelty is substantially reduced. I do not reject the paper because the empirical results could still be reproducible and useful, but the authors must provide code or a precise mask construction, and ideally compare the causal objective against the inpainting objective under identical budgets. The ADE20K pretraining-to-evaluation overlap is another concern, but I treat it as secondary because the ImageNet classification claim is the headline. No ad hominem is intended; this is a request for internal consistency and reproducibility.","tokens_in":12953,"tokens_out":6565,"duration_ms":62071,"concrete_test":"Release code or specify the exact mask for the 20% cluster-masked run, then compare two MAL-B pretraining variants: strict causal next-cluster prediction (Eq. 3, cluster size 4x4, no random masking) versus the Table 6 recipe with 20% random token masking on 144 tokens. Fine-tune both on ImageNet-1K and check which reproduces 83.4%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on the 'cluster-masked autoregressive' pretraining mechanism, but the formal objective and the best-performing configuration are inconsistent. Eq. (3) defines next-cluster prediction with no masking ratio, and Eq. (4) is a strict causal lower-triangular mask. In contrast, Section 4.5 / Table 6 sets the pretraining sequence length to 144 tokens and varies a 'masking ratio' from 1% to 70%, stating that masking multiple tokens 'transforms the task into an inpainting problem.' With the optimal cluster size 4x4 on a 192x192 input, there are only 9 clusters, so a 20% masking ratio over 144 tokens cannot be expressed in cluster units (20% of 9 is not an integer). If the 83.4% result is actually obtained with 20% random token masking and length-preserving reconstruction, then the improvement is not produced by the claimed autoregressive cluster-prediction objective; it would be closer to MAE-style inpainting. The manuscript provides no code and no precise algorithm showing how clusters and masking ratios interact, so the mechanism behind the headline number is not verifiable from the text alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAL, a self-supervised pretraining framework for xLSTM-based vision backbones. MAL combines a 'cluster-masked' autoregressive objective, in which spatially adjacent patches are grouped into clusters that serve as prediction units, with a two-stage pretraining procedure: first image autoregression on ImageNet-1K, then multi-task pretraining on depth estimation (NYU Depth v2) and semantic segmentation (ADE20K) jointly with autoregression. The authors report consistent accuracy improvements over supervised baselines and over the ViL xLSTM backbone: MAL-B reaches 83.4% top-1 on ImageNet-1K versus 82.4% for ViL-B, and MAL-S improves ADE20K mIoU to 47.5 versus 46.3 for ViL-S. The paper also reports robustness gains on ImageNet variants and an ablation study covering prediction-unit size, scanning order, masking ratio, decoder design, and multi-task pretraining.","tokens_in":13185,"tokens_out":5540,"duration_ms":48571,"significance":"If the results hold, the paper would be a useful contribution as the first autoregressive pretraining study for xLSTM vision backbones, and the combination of cluster-masked prediction with multi-task encoder-decoder pretraining is a reasonable direction. The paper's strengths include the systematic ablation over prediction-unit granularity and prediction order, the inclusion of out-of-domain ImageNet evaluations, and the segmentation results on ADE20K. However, the significance is substantially weakened by internal inconsistencies in the definition of the pretraining objective and in the ablation tables, and by the absence of a precise algorithmic description or code for the cluster-masked procedure. The central claim that MAL 'sets a new benchmark' therefore cannot be assessed reliably from the manuscript in its current form.","major_comments":[{"comment":"The formal cluster-autoregressive objective and the best-performing configuration are mutually inconsistent. With the chosen 4×4 cluster size on a 192×192 input there are only (192/16/4)^2 = 9 prediction units, but Table 6 says the pretraining sequence is 144 tokens and reports a masking ratio of 20% (28 tokens), which cannot be expressed in units of 9 clusters. The paper never defines how random token masking and cluster prediction units interact, so the mechanism behind the headline 83.4% is not verifiable from the text; please provide the exact mask construction and state explicitly whether the 83.4% result uses the Eq. (3) objective or a length-preserving inpainting objective.","section":"§3.2.1, Eq. (3) and §4.5, Table 6"},{"comment":"The causal mask is written as content_mask_ij = 0 for i < j and −∞ for i ≥ j, which by the paper's own definition ('0 allows the i-th token to attend to the j-th token') lets later tokens attend to earlier ones and blocks self/previous attention; this is the opposite of the stated property that 'each token attends only to itself and preceding tokens'. If this is a sign or indexing typo, it must be corrected, because the autoregressive property of the encoder-decoder is load-bearing for the method.","section":"§3.3.2, Eq. (4)"},{"comment":"The hyperparameters (cluster size, masking ratio, decoder depth and width) are selected by maximizing ImageNet validation top-1 accuracy on the same validation set used to report the headline 83.4%, so the comparison against ViL-B (82.4%) is not a comparison of a fixed architecture but of the best cell of a searched grid. The paper should report an evaluation protocol that separates configuration selection from reporting, or provide multiple seeds with error bars; otherwise the 1.0% claimed gain over ViL-B may be partially due to selection noise.","section":"§4.5, Tables 4, 6, 7 and §4.2, Table 1"},{"comment":"The ablation story is internally inconsistent. Table 4 reports 83.4% for 9 cluster units under what the text describes as autoregressive pretraining, while Table 8 reports 82.5% for MAL-B with autoregression only and credits multi-task pretraining for the rise to 83.4%. If Table 4's configuration already includes multi-task pretraining, then the 1.5-point cluster-size gain and the 0.9-point multi-task gain cannot both be read off as stated. Please specify exactly which pretraining stages and which hyperparameters each ablation uses.","section":"§4.5, Tables 4 and 8"},{"comment":"The throughput for MAL-T is reported as 4108 imgs/s in Table 1 but as 1301 imgs/s in the text. Since efficiency is one of the paper's stated advantages, this discrepancy must be resolved before the efficiency claims can be assessed.","section":"§4.2 and Table 1"}],"minor_comments":[{"comment":"The caption reads 'all image sizes are 2242'; this should be '224×224' or '224^2'.","section":"Table 1 caption"},{"comment":"The section begins with a sentence fragment: 'Unified Architecture has become increasingly popular...' should be a complete sentence with a subject.","section":"§2.2"},{"comment":"References [1] and [2] are the same paper (Vision-LSTM by Alkin et al.) and should be merged.","section":"References"},{"comment":"The masking ratios are labeled as 1%, 10%, 20%, 30%, 50%, and 70%, but for a 144-token sequence the corresponding token counts are 1, 14, 28, 43, 72, and 100, which are approximations (e.g., 14/144 = 9.7%); the rounding convention should be stated.","section":"§4.5, Table 6"},{"comment":"The decoder applies the same content mask as the encoder, but if the decoder is reconstructing all tokens with position embeddings, the role of the causal mask in the decoder should be explained, especially in light of the length-preserving reconstruction described in Table 6.","section":"§3.3.4, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not currently reproducible: no code is provided, no seeds or error bars are reported, and the precise algorithm connecting cluster sizes to masking ratios is absent. The ablation tables are used both for hyperparameter selection and for headline reporting, which inflates the apparent gains. I would ask the authors to provide a precise algorithmic specification (or code), to clarify the evaluation protocol, and to reconcile Tables 4, 6, and 8 before the paper can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is the first to put xLSTM under autoregressive pretraining for vision, and the empirical story is mostly consistent: MAL-B gets 83.4% on ImageNet, about a point above ViL-B, with similar gains on robustness sets and a 1.2 mIoU jump on ADE20K over ViL-S. The ablations are reasonably thorough—prediction unit size, order, masking ratio, decoder shape, and the multi-task head all get tested. If I just read Tables 1–8, this is a credible recipe for squeezing more out of xLSTM backbones.\n\nThe soft spots are real though. The biggest one is the mismatch between the stated method and the best configuration. Eq. (3) defines next-cluster prediction, and Eq. (4) is a strict causal mask. But the masking ablation in Table 6 is run with a 144-token sequence and reports masking ratios like 20% (28 tokens). With the chosen 4x4 cluster size on a 192x192 input you only have 9 clusters, so 20% in token space does not correspond to any integer number of clusters. The paper itself says masking multiple tokens 'transforms the task into an inpainting problem.' So the headline 83.4% may be coming from a masked-inpainting objective, not the cluster-autoregressive one claimed in the abstract. This is not a minor wording issue; it means the mechanism behind the main result is not actually specified in the text.\n\nThere are also smaller issues. The text says MAL-T throughput is 1301 imgs/s while Table 1 lists 4108. No error bars or seeds. The hyperparameters (masking ratio, cluster size) were selected on ImageNet val, then the same val set is used to report the final number. The comparison set is missing standard self-supervised baselines like MAE or AIM. And 'setting a new benchmark' is a stretch for a ~1 point gain.\n\nCounter to that, the citations are fine; they credit ARM for cluster AR and ViL for xLSTM, and the claim of first xLSTM AR vision pretraining looks correct. The paper ships no code or data, which is a big reason the masking issue cannot be resolved by the reader.\n\nBottom line: the paper deserves a serious referee because the question is timely and the empirical gains are plausible. But it is not acceptable in this form. Ask for code, error bars, a precise definition of the cluster-masked mask, and a direct comparison to MAE/AIM-style pretraining. If the authors can show the 83.4% actually comes from the cluster-AR objective, it becomes a useful contribution.\n\nWould I take it to review? Yes, but with major revision.","headline":"First xLSTM autoregressive vision pretraining with modest gains, but the headline result appears to come from an inpainting variant not the claimed cluster-AR objective.","tokens_in":13747,"tokens_out":5697,"would_cite":false,"duration_ms":48033,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cluster-masked and multi-task pretraining raises xLSTM vision to 83.4% top-1 on ImageNet-1K.","keywords":["xLSTM","autoregressive pretraining","cluster-masked masking","multi-task learning","visual representation learning","ImageNet-1K","semantic segmentation","masked image modeling"],"falsifier":"Run the MAL-B pretraining pipeline with the cluster size and masking ratio selected on a separate validation split (or a fixed rule chosen before seeing ImageNet-1K), then compare with ViL-B under identical fine-tuning; if the resulting top-1 is no better than ViL-B's 82.4%, the cluster-masked and multi-task pretraining claim would not survive an unbiased estimate. Alternatively, keep the default 9-unit/20% configuration but replace cluster targets with 1x1 patch targets and identical compute; if the gap between 83.4% and the patch-unit result disappears, the cluster mechanism, not the two-stage multitask schedule, is the source of the gain.","tokens_in":1981,"feed_emoji":"🖼️","tokens_out":3440,"duration_ms":118917,"temperature":0.7,"pith_summary":"xLSTM is an extended LSTM architecture with exponential gating and parallel matrix memory that can be turned into a vision backbone, but the paper argues its visual potential has not been realized because it has mostly been trained with standard supervised recipes. MAL is a two-stage pretraining framework designed to fix this: first, an autoregressive stage that predicts clusters of patches rather than individual pixels or patches, using a causal content mask so each cluster attends only to earlier clusters; second, a multi-task stage that adds depth estimation and semantic segmentation to the same encoder-decoder setup. The paper reports that MAL-B reaches 83.4% top-1 on ImageNet-1K, compared with 82.4% for the supervised ViL-B baseline and 81.9% for Vim-B, and that MAL-S improves ADE20K segmentation mIoU by 1.2 over ViL-S. If these results hold, the paper has found a self-supervised pretraining pathway that lets xLSTM backbones scale and compete with attention- and Mamba-based vision models rather than lagging behind them.","feed_headline":"Cluster-masked pretraining lifts xLSTM to 83.4% on ImageNet","feed_subtitle":"A two-stage pretraining recipe beats supervised baselines with xLSTM at the same model size.","key_machinery":"The load-bearing mechanism is the cluster-masked autoregressive pretraining scheme, driven by a causal content mask over the serialized image: position $i$ is allowed to attend only to itself and earlier positions, with forbidden attention set to $-\\infty$ in the attention logits. The prediction unit is a spatial cluster $c_i\\in\\mathbb{R}^{H_c\\times W_c}$ formed by grouping adjacent $16\\times16$ patches (default $4\\times4$ clusters), which shortens the sequence and supplies a stronger local target than pixels or single patches. Around this, MAL alternates mLSTM (matrix-memory LSTM) blocks that scan the sequence forward from the top left and backward from the bottom right in the xLSTM encoder, and a separate transformer decoder with its own parameters reconstructs the input. The second stage attaches task-specific linear heads for depth and segmentation while keeping the same encoder-decoder, then discards the decoder at fine-tuning.","core_discovery":"The central claim is that the quality of autoregressive visual pretraining for xLSTM is governed less by the backbone and more by what the model is asked to predict and by auxiliary tasks. Grouping spatially adjacent patches into clusters as prediction units, with a cluster size of 4x4 patches (9 prediction units per 144-token sequence), outperforms both pixel- and patch-level units; the paper's ablations show 83.4% top-1 with 9 units versus 81.9% with 144 patch units and a 81.5% supervised baseline. A second pretraining stage that jointly performs autoregression, depth estimation, and segmentation raises accuracy further, from 82.5% to 83.4% for MAL-B, even though only the encoder is kept during fine-tuning and task heads are replaced by a linear classifier. The conclusion the paper draws is that cluster-masked and multi-task pretraining fully leverages the scaling potential of xLSTM and surpasses traditional supervised models.","pith_inferences":["I would expect cluster-masked targets to transfer to other causal vision sequence models, since the mechanism is defined at the serialization level rather than inside xLSTM; swapping xLSTM for Mamba or linear attention would isolate where the gain comes from.","Because the cluster size and masking ratio were selected by maximizing ImageNet-1K validation accuracy, I would treat the reported 83.4% as an optimistic estimate until the configuration is fixed on a separate split or chosen as a default before seeing the test set.","The +0.9-point multi-task gain comes from only two auxiliary datasets (NYU Depth v2 and ADE20K); I would test whether adding more tasks or larger auxiliary data continues to improve the encoder or saturates quickly.","The scheme sits between autoregression and inpainting, since masking several consecutive tokens makes the prediction task partially like masked image modeling; I would expect a unified loss to capture both, and the paper's masking-ratio ablations give evidence for that view."],"forward_implications":["MAL-B's 83.4% top-1 on ImageNet-1K would be the strongest reported result among xLSTM, Mamba, and attention-based vision backbones at base scale in the paper's comparison, beating ViL-B by 1.0 point and Vim-B by 1.5 points.","Replacing pixel- and patch-level prediction units with 4x4 clusters improves the same encoder's top-1 accuracy from 81.9% to 83.4%, making cluster-masked autoregression the recommended pretraining target for vision xLSTM.","A second pretraining stage with depth and segmentation tasks raises MAL-B from 82.5% to 83.4%, even though only the encoder and a linear classification head are kept for fine-tuning.","The same pretrained encoder transfers to ADE20K semantic segmentation, with MAL-S improving mIoU from 46.3 to 47.5 over ViL-S under the UperNet framework."],"supporting_citations":[{"why":"Supplies the supervised ViL baseline and the xLSTM vision backbone that MAL pretrains and then compares against in all tables.","marker":"[2]"},{"why":"Defines the xLSTM building blocks, including the mLSTM layers with matrix memory that the MAL encoder alternates.","marker":"[4]"},{"why":"Source of the cluster-based prediction-unit idea and the Mamba-based ARM baseline that MAL-B must beat in Table 1.","marker":"[40]"},{"why":"The pixel- and patch-based autoregressive pretraining baseline that MAL's cluster target replaces.","marker":"[6]"},{"why":"Supplies the fine-tuning recipe and the masking-ratio framing used in MAL's pretraining ablations.","marker":"[21]"},{"why":"Supports the claim that autoregressive image pretraining scales with model capacity, which MAL extends to xLSTM.","marker":"[19]"},{"why":"The Vision Mamba baseline whose accuracy and robustness numbers MAL is compared against in Tables 1 and 2.","marker":"[55]"},{"why":"Supplies the NYU Depth v2 data for the depth estimation task in the multi-task pretraining stage.","marker":"[34]"},{"why":"Supplies the ADE20K data for the segmentation task in multi-task pretraining and in the segmentation benchmark of Table 3.","marker":"[53]"}],"fun_headline_variants":["Cluster-masked xLSTM beats supervised baselines on ImageNet","Cluster-masked units improve xLSTM autoregressive pretraining","Multi-task pretraining lifts xLSTM to 83.4% on ImageNet","xLSTM vision boosted by cluster-masked multi-task pretraining"],"cache_read_input_tokens":15872,"weakest_assumption_plain":"The reported headline accuracy (83.4% for the base-size MAL model) is the result of choosing the cluster size and masking ratio by looking at ImageNet-1K validation accuracy, then reporting that chosen configuration against fixed baselines; if that selection is not accounted for, the claimed gain over ViL-B may be smaller than it appears.","fun_headline_variants_meta":{"raw":{"variants":["Cluster-masked xLSTM beats supervised baselines on ImageNet","Cluster-masked units improve xLSTM autoregressive pretraining","Multi-task pretraining lifts xLSTM to 83.4% on ImageNet","xLSTM vision boosted by cluster-masked multi-task pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000853,"raw_usage":{"total_tokens":3706,"prompt_tokens":942,"completion_tokens":2764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2686}},"tokens_in":558,"tokens_out":2764,"duration_ms":18737,"temperature":1.0,"reasoning_tokens":2686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:39:31.569018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MAL-B pretraining pipeline with the cluster size and masking ratio selected on a separate validation split (or a fixed rule chosen before seeing ImageNet-1K), then compare with ViL-B under identical fine-tuning; if the resulting top-1 is no better than ViL-B's 82.4%, the cluster-masked and multi-task pretraining claim would not survive an unbiased estimate. Alternatively, keep the default 9-unit/20% configuration but replace cluster targets with 1x1 patch targets and identical compute; if the gap between 83.4% and the patch-unit result disappears, the cluster mechanism, not the two-stage multitask schedule, is the source of the gain.","supporting_citations":[{"cited_title":"Alkin, M","cited_arxiv_id":null,"evidence_quote":"Supplies the supervised ViL baseline and the xLSTM vision backbone that MAL pretrains and then compares against in all tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the cluster-based prediction-unit idea and the Mamba-based ARM baseline that MAL-B must beat in Table 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The pixel- and patch-based autoregressive pretraining baseline that MAL's cluster target replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NYU Depth v2 data for the depth estimation task in the multi-task pretraining stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ADE20K data for the segmentation task in multi-task pretraining and in the segmentation benchmark of Table 3."}],"review_version":1}