{"id":"9350132e-b3e6-4c6f-a787-c55d1ffa2eb8","arxiv_id":"1908.04130","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A deep congealing method using a fused spatial transformer network and a penalised auto-encoder aligns image ensembles of up to one million images with improved robustness over prior baselines.","lead":"This paper presents a deep learning method that aligns large collections of images, such as faces or digits, without any labels. The approach scales to millions of images and could reduce manual annotation work in computer vision tasks like face recognition and 3D reconstruction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reconstruction-loss proxy in Eq. (7) is asserted, not validated; because C alone shrinks and the autoencoder adapts jointly, reported APSNR may reward collapse to the reference rather than correct warps.","rationale":"The reader's weakest assumption points to the same core issue: the penalised reconstruction loss is used as a proxy for alignment quality without independent validation. My reading confirms and sharpens this concern. The paper gives no proof or controlled experiment establishing that minimising Eq. (7) at fixed autoencoder capacity is monotone in true alignment quality. In fact, the autoencoder is not at fixed capacity during training, so the proxy can be gamed by adapting the encoder/decoder to the current misalignment or by collapsing the code toward zero. The paper's own ablation (Sec. 3.2) shows that C alone produces shrinking, and the authors only claim that D prevents this degeneracy; but no experiment verifies that the combined loss recovers correct transformations. All quantitative evidence is based on appearance similarity to the mean or to a reference, which can be high for a collapsed solution. The strongest evidence for the central claim would be direct evaluation against known warp parameters, especially on affNIST and infiMNIST where such ground truth is available or constructible. The paper also lacks baseline comparisons on the largest datasets and error bars, but these are secondary to the missing validation of the core objective. Because the concern is about missing evidence rather than a demonstrated contradiction, the reader's CONDITIONAL verdict is appropriate; the paper should be accepted only after the proposed ground-truth-warp check is performed.","tokens_in":12928,"tokens_out":6107,"duration_ms":74305,"concrete_test":"On affNIST or a synthetic MNIST stack with known random affine/perspective warp parameters, run the full method (D+C), a D-only ablation, and a C-only ablation under identical initialisation. Evaluate the estimated warp parameters against the known ground-truth inverse transformations (e.g., mean angular and translation error per image) in addition to APSNR. If D+C does not significantly reduce parameter error relative to D-only, or if it improves APSNR while increasing parameter error, the auxiliary reconstruction loss is not a valid alignment objective and the central claim of a combined formulation fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the assumption that the low-capacity autoencoder's reconstruction loss C in Eq. (7) is a faithful surrogate for joint alignment quality: better-aligned stacks should have lower C at fixed capacity. This is never validated. Because Eθ and Dφ are optimized simultaneously with the warps, the 'capacity' is not held fixed: the network can lower C by adapting its weights to the current batch or by driving z toward zero, since f(z)=w^T z with positive weights penalises non-zero code. The authors themselves observe in Sec. 3.2 (footnote 4) that C alone causes 'shrinking'; they rely on the reference term D to prevent it, but no experiment checks whether D+C recovers correct warps rather than merely increasing similarity to the reference/mean. All reported quantities (APSNR, landmark errors normalised by eye-to-eye distance) measure appearance similarity, not correctness of the estimated transformation. On affNIST and infiMNIST, where ground-truth warp parameters are known by construction or can be recorded, no parameter-error evaluation is reported, and no baselines are run at these scales. The claimed 'high accuracy' therefore lacks a direct test of alignment correctness; the auxiliary loss could be inflating APSNR by collapsing appearance toward a template.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised joint image alignment (congealing) method that combines a spatial transformer network (DeSTNet) with a low-capacity auto-encoder. The total loss (Eq. 8) is a weighted sum of an l1 similarity term D against a reference image and a complexity term C that penalizes reconstruction error plus a positional penalty on the latent code. The method is trained end-to-end with SGD. Experiments on MNIST subsets, affNIST, infiMNIST, and LFW report APSNR and landmark errors, and claim robustness to perturbations and initialization as well as scalability to one million images per class, outperforming RASL, t-GRASTA, PSSV, and Deep Congealing.","tokens_in":13231,"tokens_out":5061,"duration_ms":55675,"significance":"If the central claim holds, the paper offers a practically scalable, fully differentiable congealing method that can handle far larger ensembles than existing low-rank or entropy-based approaches. The conceptual idea of using a learned, capacity-controlled auto-encoder as a differentiable proxy for stack complexity is appealing and could be reused in other alignment and co-processing tasks. The experimental scope is unusually large (up to 1M images per digit on infiMNIST), and the paper explicitly investigates robustness to initialization and perturbation type, which is a strength. However, the key theoretical premise—that the reconstruction loss C in Eq. (7) is a faithful and monotone proxy for alignment quality at fixed capacity—is asserted rather than validated, and several experimental comparisons are incomplete or confounded. The paper would be strengthened by directly measuring transformation-parameter accuracy on datasets with known ground-truth warp parameters and by controlling for hardware in the scaling comparison.","major_comments":[{"comment":"The complexity term C is proposed as a proxy for joint alignment quality, but the auto-encoder E_theta and D_phi are co-trained with the warps, so the capacity is not actually held fixed during optimization. The paper's own ablation (Sec. 3.2, footnote 4) shows that C alone causes shrinking, and the authors rely on the reference term D to prevent collapse. No experiment verifies that minimizing D+C recovers correct warps rather than merely making images more similar to the reference. On affNIST, where ground-truth affine transformation parameters are known by construction, the paper reports only APSNR and visual results, no parameter error; the same holds for infiMNIST. Because all reported metrics measure appearance similarity, and the auxiliary loss could in principle inflate APSNR by collapsing appearance toward a template, the claimed 'high accuracy' lacks a direct test of alignment correctness. I request a parameter-error evaluation on datasets with known transformations, or at least a quantitative comparison against the reference-only loss D on a held-out warp-parameter benchmark.","section":"Sec. 2.1, Eq. (7); Sec. 3.2"},{"comment":"The scalability comparison is confounded by hardware: the baselines (RASL, t-GRASTA) run on CPUs while the proposed method runs on a GPU, and the authors normalize processing times relative to the 1,000-image run for each method. This normalization removes only the constant factor at the reference size, not the hardware-dependent scaling in per-iteration cost, parallel efficiency, or implementation. Therefore, the conclusion 'the proposed method scales better than the baselines' is not established by Fig. 3. A same-hardware comparison, or a careful analysis of per-iteration complexity with controlled implementations, is needed to support the scalability claim.","section":"Sec. 3.1, Fig. 3"},{"comment":"The ablation study is qualitative only; no APSNR values are reported for the D-only, C-only, and combined conditions. Given the paper's central claim that C refines the alignment (digits '6' and '9') and acts as a regularizer (digit '3'), quantitative results with variance over runs are necessary. In addition, the hyperparameters lambda, gamma, k, and the latent code dimension b are set to single values (lambda=1, gamma=1, k=1) with no sensitivity analysis. Since the method's premise is that C is a faithful alignment surrogate at controlled capacity, the dependence on gamma (the capacity penalty) is load-bearing and should be studied explicitly.","section":"Sec. 3.2, Fig. 6"},{"comment":"The affNIST and infiMNIST experiments present results for the proposed method only, with no comparison to any baseline. The abstract and conclusion claim that the method 'outperforms state-of-the-art approaches,' but at these scales no competing method is evaluated. Even if existing methods cannot scale to 100k-1M images, a subsampled comparison or a comparison at the largest scale that baselines can handle would be needed to substantiate the claimed advantage.","section":"Sec. 3.3 and Sec. 3.4"},{"comment":"The choice of the reference image I_j is not described. The formulation in Eq. (8) aligns all images to a single reference, and the reported APSNR and landmark errors could be sensitive to which image is chosen as the reference. The score in Eq. (9) uses the average image after alignment, so a poor reference could bias the entire optimization and inflate or deflate the reported metrics. The paper should state how I_j is selected and test robustness to this choice.","section":"Sec. 2.1, Eq. (8)"}],"minor_comments":[{"comment":"The word 'tickening' appears in the description of infiMNIST deformations; this is a typo for 'thickening'.","section":"Sec. 3.4"},{"comment":"The latent code dimensionality b is denoted '|z|' but its actual value is never specified, which impedes reproducibility.","section":"Table 1"},{"comment":"The DeSTNet expansion rate kF=32 is used without definition; readers must consult reference [3] to interpret it. A one-sentence explanation would improve clarity.","section":"Sec. 3.1"},{"comment":"The notation in Eq. (8) is ambiguous: the sum over i includes the reference itself in the complexity term but the similarity term is written as I_{i≠j} ∘ p_{i≠j} − I_j. Clarify the index ranges and whether the reference image is also passed through the auto-encoder.","section":"Eq. (8)"},{"comment":"No confidence intervals or multiple-run statistics are reported for any of the APSNR or landmark-error numbers; given the stochastic optimization procedure, at least a few runs with means and standard deviations should be provided for the main comparisons.","section":"Figs. 4-10"}],"recommendation":"major_revision","confidential_remarks":"The manuscript makes a strong practical claim (millions of images) but the evidence is incomplete in several load-bearing places: the reconstruction-loss proxy is not validated against ground-truth warp parameters, the scaling comparison is confounded by CPU-vs-GPU execution, and the large-scale experiments lack baselines. These are addressable with additional experiments and analysis, so I recommend major revision rather than rejection. I also note that no code is released, and the method builds heavily on the authors' own DeSTNet; the novelty beyond DeSTNet plus the auxiliary auto-encoder loss is moderate, so the contribution should be framed as a system/algorithms paper. The citation of the authors' own work is appropriate but should not crowd out independent baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I looked at this paper with the stress-test note in hand. The note is right that the reconstruction-proxy assumption is the load-bearing piece, and the paper asserts it rather than demonstrates it. But the paper is also more than a stunt: it scales congealing to 100k–1M images per class, which prior low-rank methods can't touch, and the LFW landmark numbers give a real check beyond APSNR.\n\nWhat's new: replacing the rank term in congealing with a trainable low-capacity autoencoder plus a positional penalty on the code. That's a clean, differentiable surrogate, and the ablation shows the two loss terms are doing different work. The reference term in D prevents the shrinking that C alone produces, which the authors honestly acknowledge in the footnote.\n\nWhere it's soft: the monotonicity of C with alignment quality is never tested. The autoencoder is co-trained, so capacity isn't fixed; the penalty on z is a step, but the paper doesn't show that C pushes toward correct warps rather than template-like collapse. affNIST and infiMNIST are exactly the places where warp-parameter error could be measured, and the paper only reports APSNR. That's a real hole. Also: no error bars, no baselines on the large-scale sets, GPU-vs-CPU timing normalization is questionable, and no code release. The ablation is qualitative only.\n\nDespite that, the central idea is sound enough to be worth examining. If I were an editor, I'd send it to a knowledgeable referee, not desk-reject it. A revision that validates the proxy on synthetic data and adds parameter-error evaluation would strengthen it considerably.","headline":"A plausible scaling trick for unsupervised alignment, but the reconstruction-proxy assumption is asserted rather than shown, and the experiments don't fully support the headline claim.","tokens_in":13717,"tokens_out":3792,"would_cite":false,"duration_ms":43618,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised deep network can jointly align one million images with high accuracy.","keywords":["congealing","joint image alignment","spatial transformer network","auto-encoder","unsupervised learning","image registration","low-rank reconstruction","robust alignment"],"falsifier":"A direct test is to take a stack with known ground-truth warps and compare the proposed total loss at the true alignment with the loss at a deliberately degenerate alignment, such as shifting every image onto a uniform background. If any wrong alignment attains a lower objective value, the reconstruction proxy is not monotone in alignment quality and the method's central premise fails on that data.","tokens_in":12746,"feed_emoji":"🖼️","tokens_out":7061,"duration_ms":68701,"temperature":0.7,"pith_summary":"Joint image alignment, or congealing, is the task of finding a global warp for every image in an ensemble so that they become pixel-consistent. The paper proposes a fully unsupervised method that treats alignment as a differentiable optimisation over warp parameters, driven by two losses: an ℓ1 similarity to a reference image and the reconstruction error of a deliberately low-capacity auto-encoder. The core idea is that a well-aligned stack is more redundant, so a constrained auto-encoder can reconstruct it more accurately; minimising reconstruction error therefore pushes warps toward alignment. The authors report successful alignment on up to one million images at a time, across digits and faces, with robustness to initialisation and to affine and non-linear perturbations.","feed_headline":"Joint alignment scales to one million unsupervised images","feed_subtitle":"A low-capacity autoencoder's reconstruction error guides warping, so results stay sharp under noise and bad initialisation.","key_machinery":"The load-bearing mechanism is the penalised reconstruction auto-encoder used as an auxiliary alignment signal. The encoder $E_\\theta$ maps an aligned image to a non-negative code $z$, the decoder $D_\\varphi$ reconstructs the image, and the penalty $f(z)=w^\\top z$ with $w_l = l^k/\\sum_{l=1}^{b} l^k$ forces the code to concentrate its capacity on early components, so the auto-encoder's representational power is fixed and small. At parity of that capacity, a well-aligned stack contains more inter-image redundancy, so better alignment yields lower reconstruction error; back-propagating that error through the warping module updates the warp parameters. The warping module is the densely fused spatial transformer network, which predicts a global transformation per image and is differentiable, letting the whole pipeline be trained end-to-end by SGD.","core_discovery":"The discovery is a way to make congealing scalable and robust by combining a direct similarity measure with an indirect complexity measure in one differentiable loss. The proposed objective is\n$\\sum_{i\\ne j} \\|I_i\\circ p_i - I_j\\|_1 + \\lambda \\sum_i \\left(\\|D_\\varphi(E_\\theta(I_i\\circ p_i)) - I_i\\circ p_i\\|_1 + \\gamma f(E_\\theta(I_i\\circ p_i))\\right)$,\nwhere $p_i$ are warp parameters estimated by a densely fused spatial transformer network, $E_\\theta$ and $D_\\varphi$ form a low-capacity auto-encoder, and $f(z)=w^\\top z$ is a positional weighting penalty that pushes the code to use its first components. Minimising the second term rewards warps that make the image stack easier to reconstruct, the same redundancy principle behind low-rank congealing, while the first term anchors the ensemble to a reference and prevents the whole stack from collapsing. The authors show this objective can be optimised end-to-end with stochastic gradient descent and, in their experiments, aligns millions of digits and thousands of faces more sharply than existing rank-based and deep baselines.","pith_inferences":["One implication not spelled out in the paper is that the complexity loss could be reused as an unsupervised convergence check, stopping optimisation when reconstruction error plateaus.","Varying the penalty exponent $k$ or weight $\\gamma$ should interpolate between reference-anchored alignment and pure compression alignment, and could be tuned per dataset as a testable extension.","A natural extension is to replace the fixed reference $I_j$ with a learned template or batch mean, which would remove the need to select a reference image and might improve robustness to a bad initial reference."],"forward_implications":["Joint alignment can be applied to ensembles of a million images in a single optimisation, something rank-based methods cannot reach because of repeated SVD computations.","The same objective can be trained end-to-end on a GPU with standard SGD, so adding new images only requires a forward-backward pass rather than a fresh low-rank optimisation.","Alignment accuracy remains high under random perspective warps up to 30% displacement and under highly non-linear deformations as in infiMNIST, where baselines degrade.","On faces, the method keeps facial landmarks accurate even when the initial bounding box is enlarged by 30%, where rank-based methods blur."],"supporting_citations":[{"why":"Supplies the densely fused spatial transformer network used to predict and apply warp parameters in the aligner.","marker":"[3]"},{"why":"Defines the rank-based congealing objective and experimental protocol that the proposed formulation extends and compares against.","marker":"[35]"},{"why":"Provides the iterative Grassmannian optimisation baseline for large-scale robust alignment used as a comparison.","marker":"[15]"},{"why":"Provides the partial-sum minimisation baseline used as a comparison.","marker":"[32]"},{"why":"Provides the deep learning congealing baseline used as a comparison on LFW.","marker":"[16]"},{"why":"Introduces the lossy-compression view of joint alignment that motivates the auto-encoder complexity term.","marker":"[44]"},{"why":"Supplies the least-squares congealing method specialised for large numbers of images, the direct similarity-based predecessor.","marker":"[10]"},{"why":"Introduces the original entropy-based congealing framework and the evaluation practice of reporting mean and variance images.","marker":"[23]"}],"fun_headline_variants":["One loss to align them all: millions unsupervised","Congealing without labels: scale to millions","Align millions: deep penalised congealing","Deep loss fuses similarity and complexity for scale","Congealing millions: deep penalised reconstruction alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reconstruction error of a small auto-encoder, penalised to use little capacity, is a trustworthy and monotone proxy for alignment quality; if the auto-encoder instead learns to discard discriminative detail, minimising this loss will not move the images toward correct alignment.","fun_headline_variants_meta":{"raw":{"variants":["One loss to align them all: millions unsupervised","Congealing without labels: scale to millions","Align millions: deep penalised congealing","Deep loss fuses similarity and complexity for scale","Congealing millions: deep penalised reconstruction alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3521,"prompt_tokens":999,"completion_tokens":2522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2450}},"tokens_in":615,"tokens_out":2522,"duration_ms":20705,"temperature":1.0,"reasoning_tokens":2450,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:50:10.408364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take a stack with known ground-truth warps and compare the proposed total loss at the true alignment with the loss at a deliberately degenerate alignment, such as shifting every image onto a uniform background. If any wrong alignment attains a lower objective value, the reconstruction proxy is not monotone in alignment quality and the method's central premise fails on that data.","supporting_citations":[{"cited_title":"DeSTNet: Densely fused spatial transformer networks","cited_arxiv_id":null,"evidence_quote":"Supplies the densely fused spatial transformer network used to predict and apply warp parameters in the aligner."},{"cited_title":"Rasl: Robust alignment by sparse and low-rank de- composition for linearly correlated images","cited_arxiv_id":null,"evidence_quote":"Defines the rank-based congealing objective and experimental protocol that the proposed formulation extends and compares against."},{"cited_title":"Iterative grassmannian optimization for robust image alignment","cited_arxiv_id":null,"evidence_quote":"Provides the iterative Grassmannian optimisation baseline for large-scale robust alignment used as a comparison."},{"cited_title":"Partial sum minimization of singular values in robust pca: Algorithm and applications","cited_arxiv_id":null,"evidence_quote":"Provides the partial-sum minimisation baseline used as a comparison."},{"cited_title":"Learning to align from scratch","cited_arxiv_id":null,"evidence_quote":"Provides the deep learning congealing baseline used as a comparison on LFW."},{"cited_title":"Joint data alignment up to (lossy) transformations","cited_arxiv_id":null,"evidence_quote":"Introduces the lossy-compression view of joint alignment that motivates the auto-encoder complexity term."},{"cited_title":"Least-squares congealing for large numbers of im- ages","cited_arxiv_id":null,"evidence_quote":"Supplies the least-squares congealing method specialised for large numbers of images, the direct similarity-based predecessor."},{"cited_title":"Data driven image models through continuous joint alignment","cited_arxiv_id":null,"evidence_quote":"Introduces the original entropy-based congealing framework and the evaluation practice of reporting mean and variance images."}],"review_version":1}