{"id":"f6e01cb3-5a92-4b7e-ae3c-4ebc876e263d","arxiv_id":"2411.13073","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An ensemble of InfoNCE encoders, with their embedding spaces aligned by an unsupervised orthogonal transformation, improves OOD retrieval metrics on MNIST over single encoders.","lead":"This paper combines several pre-trained AI encoders by aligning their learned data representations before averaging them, which improves how well the combined model handles new or shifted data. The method is tested on MNIST, a small handwritten-digit dataset, and the gains are modest but consistent across in-distribution and shifted examples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ID-learned orthogonal alignment is assumed to transfer to OOD inputs, but no experiment or argument establishes this; the OOD gains hinge on an untested stability of R.","rationale":"The reader's weakest assumption identifies exactly the most load-bearing premise: the alignment matrix R learned on in-distribution data is applied unchanged to OOD inputs. For the abstract's claim to hold in the OOD setting, this transfer must be valid. The paper's theory does not establish transfer; Proposition 2 only shows that if f1(x) = R f2(x) exactly, the ensemble mean equals a single encoder's output, so it provides no argument that alignment should improve anything, let alone generalize across distribution shift. The empirical MNIST results are consistent with the method working, but they do not measure alignment quality on OOD inputs, and the ensemble has no variance estimate or significance testing. I considered whether the lack of code or the single-dataset evaluation is more load-bearing, but those affect reproducibility and scope, not the logical validity of the central claim. The transfer of R is the step where the argument is least secure: if R misaligns OOD embeddings, the reported OOD gains could be coincidental or driven by a different mechanism. A direct measurement of OOD alignment loss would settle this. Since the reader already conditioned the verdict on this and related issues, my read does not move the verdict; it sharpens the specific test needed.","tokens_in":9927,"tokens_out":4875,"duration_ms":53221,"concrete_test":"Compute the alignment loss from Eq. (2) on the OOD test set (colored and cropped) using the R learned on the ID training set, and compare it with the alignment loss on the ID test set. If the OOD loss is substantially larger, or if re-estimating R on a held-out portion of the OOD set changes ensemble R@1/MAP@R by more than a few points, then the single ID-learned R does not transfer and the OOD claim is not supported by the current protocol. This is a single computational check using the paper's own metric and data splits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central OOD claim depends on the orthogonal map R learned by minimizing Eq. (2) on the in-distribution training set remaining valid for colored/cropped test inputs. The theory (Prop. 1-2) assumes f1(x) = R f2(x) for all x, but this is only asserted for the training distribution. Distribution shift can change the relative embedding geometry of the two encoders nonlinearly; if it does, the same R will misalign OOD embeddings, and the Karcher mean will average semantically inconsistent directions. The paper reports no OOD alignment loss, no comparison between R learned on ID and R re-learned on OOD, and no robustness check across different anchor encoders. A secondary but related issue: under the exact assumption, Prop. 2 gives mean = f1(x), so the theory is a restatement of single-encoder recovery rather than a mechanism for ensemble gains; the empirical transfer of R is therefore the real load-bearing premise for the claimed OOD improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ensemble-InfoNCE, a method for ensembling self-supervised pre-trained encoders in the embedding space. The method learns an orthogonal transformation R that aligns the hyperspherical embedding spaces of M encoders on in-distribution pre-training data, then aggregates the aligned embeddings via the Karcher mean. The authors present two theoretical propositions claiming that encoder embedding spaces are related by orthogonal transformations and that the aligned ensemble recovers the ground-truth latents. They report experiments on MNIST in in-distribution, randomly colored, and randomly cropped settings, showing that the aligned ensemble improves Recall@1 and MAP@R over single encoders, whereas unaligned embedding-space ensembles degrade performance.","tokens_in":10116,"tokens_out":6206,"duration_ms":66183,"significance":"The paper identifies and addresses a real practical problem: naive averaging of misaligned embedding spaces from independently trained encoders degrades retrieval quality, and the comparison against unaligned embedding-space ensembles and weight-space ensembles is a useful sanity check. If the empirical gains are robust, the proposed method could offer a simple, unsupervised way to improve zero-shot OOD embedding quality. However, the theoretical contribution is largely a restatement of the paper's own assumption, the empirical evidence is restricted to a single dataset and a single ensemble that lacks variance estimates, and the transfer of the learned alignment to OOD inputs is an untested premise on which the OOD claim rests. The paper is honest in calling the experiments preliminary, but the central OOD claim needs stronger support.","major_comments":[{"comment":"The theoretical guarantee is a tautology under the paper's assumption. Prop. 1 follows immediately from f1(x)=R1z and f2(x)=R2z, and the proof of Prop. 2 (Eq. 5) substitutes f1(x)=Rf2(x) into the mean to obtain f1(x)=R1z. Thus the 'ensemble recovers the correct latents' result is simply the assumed single-encoder recovery property restated, and it provides no mechanism by which ensembling improves over a single encoder. The actual empirical gains must arise in the approximate-alignment regime, but the paper provides no analysis or bounds for that regime. This is load-bearing because the abstract and introduction present the theory as support for the method; I recommend either removing the theoretical-recovery claim or replacing it with an analysis of the mean under approximate alignment.","section":"Section 3.1, Prop. 2"},{"comment":"The OOD evaluation assumes that the orthogonal map R learned on the in-distribution training set (Eq. 2) remains a valid alignment for colored and cropped test inputs. The paper provides no argument or evidence for this transfer: there is no OOD alignment loss reported, no comparison between R learned on ID data and R re-estimated on OOD data, and no ablation varying the anchor encoder. If distribution shift changes the relative embedding geometry of the two encoders nonlinearly, the same R will misalign OOD embeddings and the Karcher mean will average semantically inconsistent directions. Because the OOD improvement is one of the paper's main claims, this untested transfer assumption needs to be addressed directly.","section":"Section 3.2 to Table 1"},{"comment":"The empirical claim that the aligned ensemble outperforms single models is not statistically established. The ensemble is a single deterministic combination of all M=5 models, so there is no variance estimate or confidence interval for the ensemble, and no significance test is reported. In the ID setting, the R@1 gain is 0.900 to 0.911, which is approximately 1.8 standard deviations of the single-model distribution; the OOD gains are larger, but without repeated ensembles or bootstrap resampling the reader cannot assess whether the differences are reliable. Please provide confidence intervals via bootstrapping over seeds or multiple independent ensembles, or otherwise quantify the variability of the ensemble performance.","section":"Section 4, Table 1 and Figures 2, 4-8"},{"comment":"The generality of the central claim is not supported by the evidence. All main experiments use MNIST with two synthetic shifts (color, crop), and the conclusion states that the method 'significantly enhances' OOD embedding quality. Given that the abstract and title refer to 'pre-trained encoders' generally, at least one additional dataset or a substantial tempering of the wording is needed before the claim can be accepted. The authors already acknowledge this in the future-work paragraph, but the abstract should reflect the MNIST-only scope.","section":"Section 4 and Section 5"}],"minor_comments":[{"comment":"Writing 'L_align = arg min_{R in R^{DxD}} ...' is formally incorrect: the left-hand side is a loss value but the right-hand side is a set of minimizers. Please write R* = argmin ... or define L_align as the objective function.","section":"Equations (1) and (2)"},{"comment":"The description 'randomly colored' is underspecified; please state how colors are sampled (e.g., per-channel uniform draws) and whether the background as well as the digit is colored, so that the OOD construction is reproducible.","section":"Section 4, OOD data description"},{"comment":"The WSE* description contains an unclear sentence about hyperparameters 'which were only provided for ResNet50' and a typo 'from0.1'; please rephrase and list the exact hyperparameter search ranges used for the MNIST architecture.","section":"Appendix A.5.4"},{"comment":"The claim of being the first embedding-space ensemble for self-supervised encoders would be strengthened by a brief discussion of prior unsupervised alignment approaches (e.g., Procrustes analysis or canonical correlation analysis) to make the novelty precise.","section":"Section 1, related work"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable workshop-level contribution, and the empirical comparison against unaligned and weight-space ensembles is informative. However, the theoretical propositions do not carry the weight assigned to them in the abstract, and the OOD claim rests on an untested transfer assumption for the alignment map. The MNIST-only evaluation and the absence of ensemble uncertainty estimates would need to be addressed before the paper is suitable for a journal. I recommend major revision with the concrete additions described in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical claim is credible as far as it goes, the theory is mostly decoration, and the real load-bearing premise—that an alignment learned on ID data transfers to OOD inputs—is never tested. Worth engaging, but it needs more evidence.\n\nWhat's new: the packaging of unsupervised orthogonal alignment plus Karcher mean for ensembling InfoNCE encoders. The objective in Eq. 2, with geodesic distance and a soft orthogonality constraint, is a reasonable, simple recipe. The controls are the right kind: the paper compares against unaligned embedding-space ensembles and weight-space ensembles, and it shows that aligned ensembling helps while unaligned ensembling hurts. The numbers in Tables 1 and 2 are internally consistent, and the appendix covers both supervised and unsupervised contrastive pre-training and two OOD shifts (color, crop). That is more than many workshop papers do.\n\nThe soft spots are real but not disqualifying. First, the theory is a direct corollary of the Zimmermann et al. recovery property. Proposition 1 is just composition of orthogonal maps; Proposition 2 says that under exact alignment the mean equals f1(x). So it does not explain why the ensemble should help—it only shows the ensemble does not destroy the recovered latents when alignment is perfect. Second, the OOD evaluation depends on the assumption that the orthogonal R learned on the training set remains valid on colored or cropped inputs. The paper gives no evidence for this: no OOD alignment loss, no comparison to re-learning R on OOD data. If the relative geometry of the two embedding spaces shifts nonlinearly, the method collapses to averaging misaligned vectors, which the paper's own results suggest is harmful. Third, the ensembles have no variance estimates, so we cannot tell whether the 1–7% gains are stable, and there is no significance testing. Fourth, no code is provided, which matters for a method whose value is mostly empirical. Fifth, it is MNIST only, and the gains are modest.\n\nWhat holds up: the method is simple, clearly described, and the qualitative story—alignment matters, ensembling helps when spaces are aligned—is plausible and worth testing on larger data. The paper is honest about its preliminary scope.\n\nThis is for anyone working on ensembling or merging self-supervised encoders without labels. A workshop audience will get value. It deserves a serious referee, but the authors should be pushed to add variance bars, test the transfer of R across shifts, and ideally include one larger dataset. My recommendation: send it to review with that feedback.","headline":"A clean, honest MNIST-scale demonstration that aligned embedding-space ensembles beat single encoders and unaligned averages, but the theory is a corollary of the recovery assumption and the OOD transfer of the alignment is untested.","tokens_in":10657,"tokens_out":2023,"would_cite":false,"duration_ms":21396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligned embedding-space ensembles improve OOD generalization of pre-trained encoders.","keywords":["ensemble learning","embedding-space alignment","out-of-distribution generalization","self-supervised learning","contrastive learning","hyperspherical embeddings","Karcher mean","orthogonal transformation"],"falsifier":"Measure the geodesic alignment error between $f_1(x)$ and $R f_2(x)$ on OOD data: if the error does not stay as low as on the training data, or if the best-fitting $R$ on OOD data differs substantially from the learned $R$, the transferred alignment claim fails. Concretely, train on MNIST and evaluate on a genuinely shifted distribution (e.g., rotated or corrupted digits from a different dataset), and check whether the aligned ensemble mean still outperforms the better single encoder.","tokens_in":9661,"feed_emoji":"🧩","tokens_out":5646,"duration_ms":48736,"temperature":0.7,"pith_summary":"The paper argues that the poor out-of-distribution (OOD) performance of self-supervised pre-trained encoders can be improved by ensembling several encoders in the embedding space, provided the individual hyperspherical spaces are first aligned. Its theoretical analysis extends an existing result that InfoNCE-trained encoders recover the true data latents up to an orthogonal transformation, showing that any two such encoders are related by an orthogonal map and that the mean of aligned embeddings recovers the correct latents. Based on this, the paper proposes Ensemble-InfoNCE, which learns an orthogonal alignment by minimizing geodesic distance with an orthogonality penalty, then aggregates aligned embeddings via the Karcher mean. Experiments on MNIST show that the aligned ensemble improves Recall@1 and MAP@R over single models in both in-distribution and out-of-distribution settings, with the largest gains on colored and cropped test inputs.","feed_headline":"Ensembling aligned embeddings lifts OOD recall by 7 percent","feed_subtitle":"Learning one rotation aligns InfoNCE encoders so their mean recovers true latents on shifted MNIST.","key_machinery":"The load-bearing identity is the orthogonal relationship between the embedding spaces of different InfoNCE-trained encoders: $f_1(x) = R f_2(x)$ for an orthogonal matrix $R$ (Proposition 1). The method learns this $R$ by minimizing the alignment loss $\\mathcal{L}_{\\mathrm{align}} = \\frac{1}{N} \\sum_{n=1}^{N} \\arccos\\big(f_i(x_n), R f_j(x_n)\\big) + \\lambda \\| R^T R - I_D \\|_F^2$ on in-distribution data, then aggregates the aligned embeddings with the Karcher mean on the hypersphere. This combination converts the ensemble mean into a quantity that provably recovers the true latent $z$ up to a rotation (Proposition 2).","core_discovery":"The central claim is that taking an ensemble mean in the embedding space of self-supervised InfoNCE pre-trained encoders yields better embeddings on both in-distribution and out-of-distribution data than any single encoder, but only if the embedding spaces are aligned first. The paper proves (Proposition 2) that if each encoder $f_i$ recovers the ground-truth latents $z$ up to a per-encoder orthogonal transformation $R_i$, then the mean of $f_1(x)$ and $R f_2(x)$ equals $R_1 z$, so the ensemble recovers the true latents up to an orthogonal transformation. It further establishes (Proposition 1) that any two encoders trained on the same data are related by a single orthogonal map $f_1 = R f_2$. This justifies an unsupervised alignment procedure that learns $R$ by minimizing the geodesic distance between paired embeddings plus a soft orthogonality constraint, and the paper demonstrates on MNIST that the aligned ensemble outperforms single encoders and misaligned ensembles, with a $6.99\\%$ improvement in Recall@1 and $17.38\\%$ in MAP@R on colored OOD data.","pith_inferences":["The paper's own results show smaller gains on cropped OOD data than on colored OOD data, hinting that the transferred rotation assumption may be sensitive to the type of distribution shift; a global orthogonal map may not be the right model for shifts that alter local geometry.","A testable extension is to replace the single learned rotation with an input-dependent alignment map (e.g., a small neural network), which could handle shifts that change embedding geometry nonlinearly while preserving the interpretability of the ensemble.","The OOD evaluation in the paper keeps the same label set; a stronger test would be evaluating on unseen classes or a different domain, where the claim that the aligned ensemble recovers correct latents is more consequential and less likely to hold."],"forward_implications":["The method offers a label-free way to ensemble pre-trained encoders, preserving the interpretability of output-space ensembles and the flexibility of weight-space ensembles.","If the orthogonal alignment transfers, the same pipeline applies to any InfoNCE pre-trained encoder family, not just MNIST-scale models, potentially improving zero-shot OOD transfer.","The theoretical guarantee that the ensemble mean recovers the true latents up to rotation suggests that properly aligned ensembles should consistently improve embedding fidelity over single encoders.","Misaligned ensembles are shown to hurt performance, establishing alignment as a necessary step for embedding-space ensembling rather than an optional refinement.","Since the approach operates on embeddings rather than logits, it can be applied to retrieval, classification, and any downstream task that consumes frozen features."],"supporting_citations":[{"why":"Supplies the theorem that InfoNCE-trained encoders recover latents up to an orthogonal transformation, which the paper extends to Propositions 1 and 2.","marker":"[25]"},{"why":"Defines the InfoNCE loss that the pre-trained encoders minimize.","marker":"[14]"},{"why":"Provides the Karcher Mean algorithm used to aggregate aligned embeddings on the hypersphere.","marker":"[18]"},{"why":"Introduces deep ensembles, the ensemble framework the paper adapts to the embedding space.","marker":"[9]"},{"why":"Shows that deep ensembles improve predictive performance on OOD data and informs the choice of an $M=5$ ensemble size.","marker":"[15]"},{"why":"Supplies the Recall@1 and MAP@R metrics used to evaluate embedding quality.","marker":"[12]"},{"why":"Motivates the OOD evaluation protocol and metrics through the URL representation-learning benchmark.","marker":"[7]"},{"why":"Provides the supervised contrastive pre-training setup and the cropped-image OOD evaluation procedure.","marker":"[6]"}],"fun_headline_variants":["Aligned embedding ensembles beat single encoders on OOD","Unsupervised rotation aligns InfoNCE encoders for strong OOD gains","One learned rotation makes embedding ensembles recover true latents","Mean of aligned encoders lifts OOD Recall@1 by 7% on MNIST","Simple ensemble trick: align embedding spaces first, then average"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a single orthogonal rotation $R$ learned on the in-distribution pre-training set also aligns the embedding spaces of the same encoders on out-of-distribution inputs; if distribution shift changes the relative geometry of the two embedding spaces in a way that is not a rigid rotation, the alignment and the ensemble mean will be incorrect.","fun_headline_variants_meta":{"raw":{"variants":["Aligned embedding ensembles beat single encoders on OOD","Unsupervised rotation aligns InfoNCE encoders for strong OOD gains","One learned rotation makes embedding ensembles recover true latents","Mean of aligned encoders lifts OOD Recall@1 by 7% on MNIST","Simple ensemble trick: align embedding spaces first, then average"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1664,"prompt_tokens":907,"completion_tokens":757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":666}},"tokens_in":523,"tokens_out":757,"duration_ms":6656,"temperature":1.0,"reasoning_tokens":666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:52:24.318253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the geodesic alignment error between $f_1(x)$ and $R f_2(x)$ on OOD data: if the error does not stay as low as on the training data, or if the best-fitting $R$ on OOD data differs substantially from the learned $R$, the transferred alignment claim fails. Concretely, train on MNIST and evaluate on a genuinely shifted distribution (e.g., rotated or corrupted digits from a different dataset), and check whether the aligned ensemble mean still outperforms the better single encoder.","supporting_citations":[{"cited_title":"Contrastive learning inverts the data generating process","cited_arxiv_id":null,"evidence_quote":"Supplies the theorem that InfoNCE-trained encoders recover latents up to an orthogonal transformation, which the paper extends to Propositions 1 and 2."},{"cited_title":"A dirichlet process mixture model for spherical data","cited_arxiv_id":null,"evidence_quote":"Provides the Karcher Mean algorithm used to aggregate aligned embeddings on the hypersphere."},{"cited_title":"A metric learning reality check","cited_arxiv_id":null,"evidence_quote":"Supplies the Recall@1 and MAP@R metrics used to evaluate embedding quality."},{"cited_title":"Url: A representation learning benchmark for transferable uncertainty estimates","cited_arxiv_id":null,"evidence_quote":"Motivates the OOD evaluation protocol and metrics through the URL representation-learning benchmark."},{"cited_title":"Probabilistic contrastive learning recovers the correct aleatoric uncertainty of ambiguous inputs","cited_arxiv_id":null,"evidence_quote":"Provides the supervised contrastive pre-training setup and the cropped-image OOD evaluation procedure."}],"review_version":1}