{"id":"a5e45dc6-e6e1-482a-99fc-d590bfe84924","arxiv_id":"2506.07803","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reconstruction fidelity from vision encoder features reveals which models preserve more visual information, and linear feature-space operations can produce color edits.","lead":"The authors train a small neural network to redraw images from the internal features of vision encoders, then use the quality of those redrawings to compare how much visual information different encoders preserve. They find that encoders trained on image-based objectives reconstruct more accurately, and that simple rotations in feature space produce color changes in the reconstructed images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SigLIP vs. SigLIP2 comparison conflates training objective with multilingual training data, so the central attribution to the objective is not established; Section 7 contradicts Section 5.1's own admission that the data subsets differ.","rationale":"The reader's weakest assumption correctly identifies the SigLIP versus SigLIP2 comparison as confounded by training data. This is the single most load-bearing concern because the paper's headline conclusion is causal ('encoders pre-trained on image-based tasks retain significantly more image information than those trained on non-image tasks'), and the only direct evidence for that claim is the SigLIP/SigLIP2 pair. The paper itself provides the conflicting statements: Section 5.1 discloses the WebLI English-only versus multilingual difference, while Section 7 asserts identical datasets. No other experiment in the paper isolates the training objective, so the confound cannot be waved away. I considered whether the circularity of the color-swap experiment or the lack of public code is more serious, but those affect secondary claims or reproducibility rather than the central attribution. The reconstructor pipeline itself is plausible and the statistical tests are appropriate, so the paper merits a conditional verdict rather than rejection; the condition should be an objective-only control. Since the reader already issued CONDITIONAL for essentially this reason, my stress-test does not move the verdict.","tokens_in":12567,"tokens_out":5326,"duration_ms":74140,"concrete_test":"Obtain or train a contrastive-only SigLIP-style ViT-L/16 model on the same multilingual WebLI data distribution used for SigLIP2, then rerun the Section 5.1 reconstruction protocol at all four resolutions and compare CLIP/SigLIP2 cosine scores and Table 1 statistics. If the contrastive-only multilingual model's reconstruction scores close the gap to SigLIP2, the data difference, not the objective, explains the result; if the gap persists, the objective attribution is supported. If such a model is not available, the authors should at minimum report an ablation from the SigLIP2 paper or a public contrastive-only multilingual ViT-L model at matching resolutions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that image-based multitask objectives retain more image information than contrastive-only training—rests on the SigLIP/SigLIP2 comparison. Section 5.1 explicitly states that SigLIP was trained on the English-only WebLI subset while SigLIP2 used a multilingual corpus, and that SigLIP2 additionally uses captioning, self-distillation, and masked modeling. Section 7 nevertheless claims the two models 'share identical architectures, parameter counts, and datasets but differ only in their optimization objectives.' That is internally inconsistent. Because data distribution and scale are known to affect representation quality, the reconstruction gap in Table 1 and Figure 4 cannot be attributed to the training objective without a control that holds the pretraining data fixed. A secondary issue is that reconstruction fidelity under a single learned decoder and ℓ2 loss is treated as an unbiased information measure, but no decoder-independence check is provided; however, the data confound alone is sufficient to make the headline causal claim load-bearing and currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reconstruction-based interpretability method for vision encoders: a learned decoder maps frozen encoder features back to pixels, and reconstruction fidelity is used as a proxy for how much visual information the encoder preserves. The authors compare SigLIP and SigLIP2 across four resolutions, report that SigLIP2 reconstructions are statistically better under CLIP and SigLIP2 similarity scores, and extend the method to a range of ViT-based encoders. They also study feature-space manipulations: a learned orthogonal/self-conjugate matrix that swaps red and blue channels, a linear operator for blue-channel suppression, and a linear operator for colorization. The central claims are that image-based pretraining objectives retain more visual information than contrastive-only objectives, that higher resolution increases reconstruction fidelity, and that orthogonal feature rotations correspond to interpretable color edits.","tokens_in":12768,"tokens_out":4390,"duration_ms":50270,"significance":"If the central claims were established, the reconstruction pipeline would be a useful and inexpensive tool for probing vision-encoder representations, and the SigLIP/SigLIP2 comparison would provide a clean causal statement about training objectives. The paper ships code and weights, and the statistical testing on the SigLIP/SigLIP2 difference is thorough (permutation/wilcoxon with very small p-values). The feature-manipulation experiments, especially the color-swap operator, are creative and the visual results are striking. However, the main causal attribution is currently confounded, and several supporting analyses are either internally inconsistent or less decisive than the text suggests. The method itself is promising and the empirical material is substantial, but the paper needs additional control experiments and more careful framing before the headline conclusions can be accepted.","major_comments":[{"comment":"The central claim that SigLIP and SigLIP2 'differ only in their training objective' is contradicted within the manuscript. Section 5.1 states that SigLIP uses the English-only WebLI subset while SigLIP2 uses a multilingual corpus, and explicitly concedes that 'these subsets are not identical.' Section 7, however, asserts that these models 'share identical architectures, parameter counts, and datasets but differ only in their optimization objectives.' This is an internal inconsistency. Because training-data distribution and scale are known to affect representation quality, the statistically significant reconstruction gap in Table 1 cannot be attributed to the objective alone. A control that holds the pretraining data fixed (e.g., training a SigLIP-style contrastive-only model on the same multilingual corpus, or at minimum explicitly re-framing the conclusion as comparing a combination of objective and data) is required before the headline causal statement is supported.","section":"Section 5.1 / Section 7"},{"comment":"The claim that 'orthogonal rotations — rather than spatial transformations — control color encoding' (Abstract) is not established by the experiments. The hypothesis in Section 4.2.1 posits that the feature-space operator Af is orthogonal and self-inverse; the estimation then solves an orthogonal Procrustes problem, thereby imposing orthogonality rather than discovering it. The ablation in Figure 6 compares orthogonal self-conjugated, orthogonal, and linear solutions, but all three yield visually similar reconstructions (Figure 12, columns 4-7), and the linear operator, after normalization, also reproduces the swap. This does not demonstrate that orthogonality is a uniquely correct characterization. Furthermore, the text never tests spatial transformations or other non-orthogonal alternatives, so the comparative claim 'rather than spatial transformations' is unsupported. The authors should either soften the claim to 'orthogonal operators provide a good fit' or add experiments that distinguish among transformation families (e.g., by measuring reconstruction error quantitatively against a spatial-shift baseline).","section":"Section 4.2.1 / Section 5.2"},{"comment":"The eigenvalue analysis for blue-channel suppression is largely tautological and does not provide evidence for the proposed feature-space hypothesis. The argument that repeated application of Af converges to a projection and hence its eigenvalues are either 1 or have magnitude less than 1 holds for any linear contraction; it is not a distinctive prediction of the claim that feature-space operations mirror image-space operations. The empirical validation in Figure 7 is qualitative (visual inspection of eigenvalue scatter and reconstructions), with no quantitative metric or comparison to a null distribution. To make this experiment informative, the authors should compare the learned Af against a baseline (e.g., a random operator with the same spectral radius) and report a quantitative measure of how well the repeated operator matches the pixel-space projection.","section":"Section 4.2.2 / Section 5.3"},{"comment":"The use of reconstruction fidelity as an absolute measure of 'how much visual information an encoder preserves' is not validated for decoder dependence. All experiments use a single reconstructor architecture (four transformer blocks plus upsampling) trained on a fixed 115k-image COCO subset with an ℓ2 loss. Because the decoder and training distribution are fixed, the ranking of encoders in Figure 9 and Table 1 could reflect the decoder's inductive bias or COCO-specific coverage rather than intrinsic properties of the encoders. The paper's Limitations section acknowledges this, but no control experiment is provided (e.g., a different decoder architecture, a different training set, or a perceptual loss). Additionally, the evaluations in Appendix A use only CLIP and SigLIP2 encoders as scoring functions, which may favor encoders with similar pretraining; no reconstruction-based metrics (PSNR, LPIPS) are reported. Without these controls, the strong information-ranking claims are not fully supported.","section":"Section 3 / Section 5.1 / Appendix A"}],"minor_comments":[{"comment":"The sentence 'SigLIP models are variant of CLIP' contains a grammar error ('are variant' should be 'are variants').","section":"Section 5.1"},{"comment":"The column header 'Sequence dength' is misspelled ('dength' should be 'length').","section":"Table 2"},{"comment":"The caption refers to 'fisheye space' in column 4; this appears to be a typo for 'feature space'.","section":"Figure 13 caption"},{"comment":"The reference to 'Caron et al. (2024)' for WebLI in Section 5.1 is misplaced: WebLI is a web image-text dataset introduced in the context of SigLIP/PaLI, not in the cited 'Web-scale visual entity recognition' paper. Please cite the correct source for WebLI.","section":"References"},{"comment":"Radford et al. (2021a) and Radford et al. (2021b) are the same paper and should be merged into a single reference.","section":"References"},{"comment":"The notation 'Rθ ⊥ ⊥(Ai, Af) | E' for conditional independence is nonstandard and is not defined; a brief explanation would help readers.","section":"Section 6.1"},{"comment":"The colorization experiment does not compare against a trivial baseline (e.g., predicting the mean color of the training set), so it is unclear how much of the success is due to the linear feature-space mapping versus simple global color priors.","section":"Section 5.4"},{"comment":"The caption states 'n = 1000samples' but the space is missing; also the relationship between 'n=1000' and the bootstrap 'B=100,000' is not explained in the text.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a preprint with a public GitHub repository, but I did not verify the code. The main empirical finding (SigLIP2 > SigLIP in reconstruction similarity) is statistically solid, but the internal inconsistency about the pretraining data is a serious issue for a journal publication. The feature-manipulation experiments are visually compelling but overstated in their conclusions. I believe the authors can address the data confound either by running an additional controlled experiment or by substantially weakening the causal claim, and can tighten the transformation-family claims. The manuscript is within scope for a computer vision journal and has the potential to be a useful methodological contribution after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Alex, you should know two things about arXiv:2506.07803. First, the reconstruction pipeline itself is a reasonable and useful tool: train a transformer decoder to invert frozen ViT features, then use reconstruction quality as a proxy for how much pixel-level information an encoder preserves. Second, the paper's headline comparison—SigLIP2 reconstructs better than SigLIP—is real, but the attribution to training objective is not established, because the paper contradicts itself about the training data.\n\nWhat's actually new: the systematic SigLIP/SigLIP2 comparison across four resolutions, and the linear-operator analysis showing that color channel swaps correspond to near-orthogonal transformations in feature space. The colorization and channel-suppression experiments extend that idea. The inversion framework is known from Dosovitskiy & Brox and Mahendran & Vedaldi, so the novelty is in the application, not the invention.\n\nThe strengths: the statistics are solid—Wilcoxon and bootstrap with p-values, held-out evaluation for the color operators, and an ablation that compares orthogonal self-conjugated, orthogonal, and unconstrained linear operators. The eigenvalues clustering near +/-1 even without orthogonality constraints is a nice check. The conditional-independence argument for the reconstructor and operator training is a thoughtful way to address bias.\n\nThe soft spots are real, though. Section 5.1 explicitly says SigLIP uses the English-only WebLI subset while SigLIP2 uses a multilingual corpus, and lists the extra objectives. Then Section 7 says the models 'share identical architectures, parameter counts, and datasets but differ only in their optimization objectives.' That's an internal contradiction, and since data distribution and scale affect representation quality, the central claim about the objective is now load-bearing and unsupported. A control that holds pretraining data fixed would be needed. Also, reconstruction fidelity under a single decoder and L2 loss is not obviously an unbiased information measure; a decoder-independence check would help. The code link in the abstract isn't accessible from the submission, which is a reproducibility problem but a fixable one.\n\nBottom line: this is a useful paper for anyone working on vision-encoder interpretability or model selection. It deserves a serious referee, but the revision needs to either remove the identical-datasets claim or run the controlled comparison. I'd engage with it as a reviewer.","headline":"A useful reconstruction tool for probing vision encoders, but the central SigLIP-vs-SigLIP2 attribution is undercut by an internal contradiction about training data.","tokens_in":13311,"tokens_out":2613,"would_cite":true,"duration_ms":28762,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The central claim is that reconstruction fidelity measures how much visual information an encoder's features preserve: multitask-trained SigLIP2 beats contrastive-only SigLIP at every resolution, and orthogonal feature-space rotations…","keywords":["image reconstruction","vision encoder interpretability","feature inversion","contrastive pretraining","multitask pretraining","latent space manipulation","orthogonal transformations","color editing"],"falsifier":"Train a contrastive-only SigLIP model on the same multilingual WebLI data that SigLIP2 used: if its reconstruction fidelity matches SigLIP2's rather than SigLIP's, the paper's attribution of the gap to the training objective is falsified.","tokens_in":12355,"feed_emoji":"🖼️","tokens_out":11842,"duration_ms":109034,"temperature":0.7,"pith_summary":"The paper claims that how faithfully an image can be rebuilt from a vision encoder's internal feature tensor is a direct measure of how much visual information those features preserve. Applying this reconstruction test to SigLIP and SigLIP2, two encoder families that share architecture and parameter count but differ mainly in training objective, the paper finds that the multitask-trained SigLIP2 produces significantly higher-fidelity reconstructions than the contrastive-only SigLIP at every tested resolution. The same pipeline ranks a wider set of ViT encoders by feature informativeness and shows that higher input resolutions yield more invertible features. The paper further shows that learned linear operators on feature tensors behave like their pixel-space counterparts: an orthogonal self-inverse rotation swaps red and blue channels, a linear operator suppresses the blue channel like a projection, and a linear map performs credible colorization. If correct, the method turns any frozen vision encoder plus a small trained decoder into a general probe of what the encoder keeps.","feed_headline":"Multitask-trained vision encoders keep more of the picture","feed_subtitle":"A decoder trained on frozen vision features reveals which pretraining objective stores more image detail.","key_machinery":"The load-bearing object is the trained reconstructor $R_\\theta$: a small transformer composed of four blocks with multi-head self-attention and feed-forward sublayers, followed by upsampling layers interleaved with residual blocks, mapping a feature tensor $f = E(i)$ back to the original image $i$ under an $\\ell^2$ reconstruction loss $\\mathcal{L}_{\\mathrm{rec}} = \\mathbb{E}\\|i - R_\\theta(E(i))\\|_2^2$. Each spatial feature vector is channel-wise normalized before decoding, which suppresses norm outliers common in CLIP-style features. For the manipulation experiments, the central identity is the orthogonal Procrustes solution $Q = \\arg\\min_{Q} \\sum_j \\|f_j^* - Q f_j\\|^2$ subject to $Q^\\top Q = I_c$ and $QQ = I_c$, learned from paired features $(E(i), E(i^*))$ of images and their pixel-space transforms; applying $Q$ in latent space is then shown to reproduce the pixel-space transform after decoding, and eigenvalue analysis of learned operators links them to the algebra of their image-space counterparts.","core_discovery":"The central discovery, stated on the paper's own terms, is that the training objective determines how much of the original image survives in a vision transformer's feature tensor. Because SigLIP and SigLIP2 share architecture, parameter count, and the same WebLI data source, the reconstruction gap between them is attributed to pretraining: the multitask objective (sigmoid contrastive plus image captioning, self-distillation, and masked prediction) preserves structural detail that pure contrastive learning discards. Fidelity is measured by training a decoder on COCO image-feature pairs with a frozen encoder and scoring reconstructions on the COCO validation split with CLIP and SigLIP2 cosine similarity, and SigLIP2 wins at 224, 256, 384, and 512 pixels with statistically significant margins under both Wilcoxon signed-rank and bootstrap tests. The same pipeline yields two further claims: higher-resolution feature tensors retain more visual information, and pixel-space color operations correspond to simple linear or orthogonal operators in feature space, whose eigenvalue behavior mirrors the algebra of the corresponding image-space operation (self-inverse rotations for channel swaps, projection-like spectra for blue-channel suppression).","pith_inferences":["The cleanest follow-up control would be a contrastive-only SigLIP retrained on SigLIP2's exact multilingual data subset, since that comparison separates the training objective from the training data as causes of the reconstruction gap.","If the linear colorization result generalizes, the same paired-feature linear-mapping recipe could probe whether other semantic attributes such as texture, lighting, or object identity are likewise encoded linearly in the same feature space.","Because all reconstructions are scored through decoders and similarity measures trained on natural-image statistics, the encoder ranking may partly reflect distribution overlap with photographic images; stability of the ranking under a different reconstruction dataset would test that.","The paper's own limitation note names convolutional encoders as the next target, so under the paper's logic a ConvNeXt-style encoder trained with an image-reconstruction objective should reconstruct more faithfully than a contrastive-only ViT of comparable size."],"forward_implications":["Reconstruction fidelity becomes a model-agnostic proxy for how much visual information an encoder keeps, usable on any ViT-based encoder without training probes or linear readouts.","For applications that need dense, invertible image features, multitask pretraining such as SigLIP2's is preferable to contrastive-only pretraining, since it preserves more structural detail at equal architecture and parameter count.","Input resolution should be part of encoder selection alongside architecture and objective, because higher-resolution feature tensors consistently prove more informative.","Pixel-space color edits can be driven by simple learned operators in feature space, opening a route to predictable, semantically grounded image editing through latent-space manipulation.","The encoder rankings from the appendix suggest image-based or multitask pretraining and higher resolution, more than raw parameter count, drive reconstruction fidelity across model families."],"supporting_citations":[{"why":"Defines SigLIP, the contrastive-only encoder whose reconstruction fidelity is compared against SigLIP2's.","marker":"(Zhai et al., 2023)"},{"why":"Introduces SigLIP2, the multitask-trained encoder the paper argues preserves significantly more image information.","marker":"(Tschannen et al., 2025)"},{"why":"Supplies the COCO corpus used to train the reconstructor and the validation images used to score reconstruction fidelity.","marker":"(Lin et al., 2014)"},{"why":"Establishes the feed-forward decoder approach to feature inversion that this work adapts to transformer-based encoders.","marker":"(Dosovitskiy and Brox, 2016)"},{"why":"Provides CLIPScore, the cosine-similarity metric used to measure agreement between original and reconstructed images.","marker":"(Hessel et al., 2021)"},{"why":"Prior feature-inversion-by-optimization baseline that the paper's single-pass decoder improves on in speed and architecture coverage.","marker":"(Mahendran and Vedaldi, 2016)"},{"why":"Cited as the source of the WebLI pretraining data, whose English-only versus multilingual subsets define the confound in the SigLIP comparison.","marker":"(Caron et al., 2024)"},{"why":"CLIP serves both as one of the evaluated encoders and as the similarity model for scoring reconstruction quality.","marker":"(Radford et al., 2021a)"}],"fun_headline_variants":["Training objective determines how much image detail encoder retains","Image reconstruction reveals contrastive learning discards detail","Multitask vision encoders outperform contrastive in image fidelity","Frozen features decoded: which pretraining keeps more of the picture","Reconstruction scores expose what vision encoders truly preserve"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reconstruction gap between SigLIP and SigLIP2 comes from the training objective alone, even though the two models were trained on different data subsets (English-only versus multilingual WebLI) and every reconstruction is judged through a decoder trained on COCO.","fun_headline_variants_meta":{"raw":{"variants":["Training objective determines how much image detail encoder retains","Image reconstruction reveals contrastive learning discards detail","Multitask vision encoders outperform contrastive in image fidelity","Frozen features decoded: which pretraining keeps more of the picture","Reconstruction scores expose what vision encoders truly preserve"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3133,"prompt_tokens":934,"completion_tokens":2199,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2120}},"tokens_in":550,"tokens_out":2199,"duration_ms":18779,"temperature":1.0,"reasoning_tokens":2120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:25:03.952938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a contrastive-only SigLIP model on the same multilingual WebLI data that SigLIP2 used: if its reconstruction fidelity matches SigLIP2's rather than SigLIP's, the paper's attribution of the gap to the training objective is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SigLIP, the contrastive-only encoder whose reconstruction fidelity is compared against SigLIP2's."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the COCO corpus used to train the reconstructor and the validation images used to score reconstruction fidelity."},{"cited_title":"and Brox, T","cited_arxiv_id":null,"evidence_quote":"Establishes the feed-forward decoder approach to feature inversion that this work adapts to transformer-based encoders."},{"cited_title":"and Vedaldi, A","cited_arxiv_id":null,"evidence_quote":"Prior feature-inversion-by-optimization baseline that the paper's single-pass decoder improves on in speed and architecture coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as the source of the WebLI pretraining data, whose English-only versus multilingual subsets define the confound in the SigLIP comparison."}],"review_version":1}