{"id":"8ae25840-91e4-4239-ab3f-623c7978a5b9","arxiv_id":"2603.04114","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A unified latent-diffusion framework with per-modality autoencoders translates among RGB, SAR, NIR, MS, and PAN remote-sensing images, trained on a newly aggregated 1.2M-pair dataset.","lead":"This paper proposes a single latent-diffusion model that translates between any pair of five remote-sensing image modalities, backed by a new 1.2-million-pair dataset. It matters because it replaces the usual practice of training a separate translator for every modality pair with one shared model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'geometrically aligned' shared latent space is asserted but never established; independent per-modality VAEs cannot guarantee cross-modal alignment, so zero-shot composition has no grounded mechanism.","rationale":"The reader's weakest_assumption identifies the same core risk: the shared latent space may not be geometrically aligned, and the RGB pivot may be physically inconsistent. This is the most load-bearing concern because the paper's flagship claims — O(1) complexity and zero-shot generalization to unseen modality pairs — both rest on the existence of a coherent shared manifold. Without alignment, the single backbone is just a multi-task collection of pairwise mappings, and zero-shot predictions are ungrounded. The paper provides no direct evidence for alignment: stage I uses only reconstruction losses and a per-modality KL term, which cannot create cross-modal correspondences; the manual scaling factors are insufficient; and the zero-shot evaluation is qualitative only. The concrete test I propose directly measures latent-space alignment using the released encoders. If the test fails, the paper would still be valuable for seen-task multi-training, but the central 'any-to-any' claim would need to be substantially weakened. Since the reader already gave a CONDITIONAL verdict, my concern does not shift that label, but it sharpens the condition: the authors must demonstrate latent alignment, not merely assert it.","tokens_in":18456,"tokens_out":8727,"duration_ms":89267,"concrete_test":"Use the released code to extract latents from the frozen Stage-I encoders on the RST-1M test set. For each modality pair (i,j) with paired samples, compute L2 distance and cosine similarity between z_i=E_i(x_i) and z_j=E_j(x_j) for matched pairs, and compare to a null distribution from 10,000 random pairings. If matched-pair distances are not significantly lower than the null (e.g., Cohen's d < 0.5), the latent spaces are not aligned. Additionally, train a linear classifier on the RGB latents to distinguish Sentinel-2 RGB (from SEN12MS/CACo/SEN1-2) from WorldView pan-sharpened RGB (from SpaceNet); high accuracy (>90%) would show the RGB pivot is not a unified domain. Either outcome would falsify the geometric-alignment assumption and thus the grounding for zero-shot composition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Any2Any's central mechanism for O(1) scaling and zero-shot generalization is a 'geometrically aligned' shared latent manifold Z (§4.3). Yet Stage I trains N independent VAEs with per-modality KL losses toward N(0,I) (Eq. 3). This regularizes each marginal but imposes no cross-modal constraint — no paired latent loss, no shared encoder/decoder, no canonical coordinate alignment. The manual scaling factors in §4.3 only normalize latent magnitudes, not directions. Consequently, after Stage I the latent spaces are merely equal-dimensional normal-ish distributions, not a common manifold. The Stage II diffusion backbone learns a distinct mapping from each source latent space to each target latent space under paired supervision; these mappings need not be transitive. The residual adapters (Eq. 8) are per-target and only correct systematic mismatch for targets seen during training, so for an unseen pair like SAR→PAN the model is forced to compose SAR→RGB and RGB→PAN mappings without any basis for composition. Moreover, Appendix B.2/B.3 reveals the RGB pivot is physically heterogeneous: Sentinel-2 RGB at 10 m is pooled with WorldView pan-sharpened RGB at sub-meter resolution, so the 'RGB' encoder may span two different domains. Because the paper provides no quantitative evidence of latent alignment — only qualitative zero-shot results (Fig. 5) — the central claim of strong zero-shot generalization is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Any2Any, a unified latent diffusion framework for translating among five remote sensing modalities (RGB, SAR, NIR, MS, PAN). The method consists of: (i) independently trained per-modality VAEs that project each modality into a common latent dimension; (ii) a shared Diffusion Transformer backbone that, conditioned on source/target modality embeddings, directly regresses the target latent from a concatenation of the source latent and a noisy target latent; and (iii) lightweight target-specific residual adapters that correct latent mismatches. The authors also construct RST-1M, an aggregated dataset of about 1.2M pairwise aligned images drawn from SEN1-2, SEN12MS, CACo, SpaceNet-3, and SpaceNet-5, covering seven paired modality combinations and supporting 14 directed translation tasks. Experiments report improved PSNR/SSIM/RMSE over generic image-to-image baselines on these 14 tasks, and qualitative results on six unseen directions are presented as evidence of zero-shot generalization. The paper claims to reduce cross-modal translation complexity from O(N^2) to O(1).","tokens_in":18854,"tokens_out":4453,"duration_ms":48460,"significance":"If the empirical and architectural claims hold, the paper would be a useful step toward scalable multi-modal remote sensing translation: it provides a large benchmark dataset, a unified model that avoids per-pair training, and promising qualitative generalization. The dataset itself, if released, would be a valuable resource. The paper also incorporates sensible design choices: x0-prediction with a shared backbone, zero-initialized residual adapters, and a clear staged training protocol. The ablations, though limited, are informative. However, the central claims rest on the existence of a 'geometrically aligned shared latent space', which is asserted but not demonstrated, and the zero-shot evaluation is only qualitative. These gaps prevent the paper from being accepted in its current form.","major_comments":[{"comment":"The central claim of a 'geometrically aligned shared latent manifold Z' is not established. Stage I trains independent per-modality VAEs, each regularized only toward N(0,I) via its own KL loss. This constrains each marginal but imposes no cross-modal correspondence: there is no paired latent loss, no shared encoder/decoder, and no canonical coordinate alignment. The scaling factors in §5.1 normalize latent magnitudes, not directions. Consequently, after Stage I the latent spaces are merely equal-dimensional, approximately normal distributions, not a common semantic manifold. The Stage II backbone learns mappings from each source latent space to each target latent space under paired supervision; such mappings need not compose transitively. The residual adapters are trained only for seen target-source combinations, so for an unseen pair such as SAR→PAN there is no grounded basis for using","section":"§4.3, Eq. (3); §4.5, Eq. (8)"},{"comment":"The comparison omits recent remote-sensing-specific translation methods that are discussed in Related Work (§2.2), specifically S3OIL, RLIDM, and DOGAN. These are direct state-of-the-art baselines for SAR→optical translation, a central task in Table 2. Without them, the claim that Any2Any 'consistently outperforms existing methods' is not fully supported. Please add these methods (or explain why they are excluded, e.g., unavailability of code/results) and report their numbers on the same test sets.","section":"§5.1/Table 2"},{"comment":"Zero-shot generalization is a principal contribution, but it is evaluated only qualitatively. Figure 5 shows 'reasonable' outputs without any quantitative metric or comparison. Since the six unseen directions lack paired ground truth, a natural protocol is to hold out a subset of existing paired directions during training and evaluate quantitatively on those, or to compare against chaining pairwise models (e.g., SAR→RGB→PAN). As written, the claim of 'strong zero-shot generalization' is not quantitatively supported.","section":"§5.2, 'Zero-shot Experiments', Fig. 5"},{"comment":"The construction of RST-1M relies on 'RGB' as a pivot, but the physical content of this modality is heterogeneous: RGB from SEN12MS/SEN1-2/CACo is Sentinel-2 at ~10 m resolution, while RGB from SpaceNet is pan-sharpened WorldView at sub-meter resolution. These are treated as the same modality and encoded by a single RGB VAE. If the RGB encoder spans two different domains, the 'shared' latent space may contain a domain split that undermines the connectedness of the modality graph. The paper should analyze whether the RGB encoder is truly unified (e.g., by visualizing latent embeddings or evaluating RGB reconstruction on both sources) or at minimum discuss this limitation.","section":"§3, Appendix B.2/B.3"}],"minor_comments":[{"comment":"'First million-scale dataset with paired observations across five sensing modalities' is overstrong: RST-1M is an aggregation of pairwise datasets, not a collection of five-modality aligned tuples. Clarify that it provides connected pairwise supervision.","section":"Abstract"},{"comment":"No error bars, standard deviations, or significance tests are reported. Even if the improvements are large, a reader cannot assess run-to-run variability. Please report at least means and stds over multiple seeds.","section":"Table 2"},{"comment":"The layout is very hard to parse: the checkmarks and column headers are ambiguous, especially for Settings 5–6. Please present each setting on a separate row with explicit column names.","section":"Table 3"},{"comment":"Treating p(z|x_i) as a Dirac delta is a conceptual idealization; in practice the conditional distribution is not degenerate. This is acceptable as motivation but should be phrased as an approximation rather than a literal statement.","section":"Eq. (1)"},{"comment":"The text says SSIM is 'slightly lower than BBDM in a few cases' but the exact directions are not listed. Please specify which tasks and by how much; this would help readers interpret the trade-off.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising framework and a useful dataset, but the current version overclaims both the geometric alignment of the latent space and the zero-shot generalization. The lack of comparisons to RS-specific SOTA and the absence of any quantitative zero-shot evaluation are fixable with additional experiments. I do not see a fundamental flaw that would require rejection; the mechanism can be tested and the claims can be calibrated. The heterogeneity of the RGB pivot is a deeper issue that may require either additional analysis or a revised formulation of the dataset's connectivity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Had a look at Any2Any. The useful core is the RST-1M dataset—1.2M paired images across five modalities with a connected supervision graph—and a unified framework that replaces 14 separate pairwise models with one shared diffusion backbone plus per-modality VAEs and tiny residual adapters. On the 14 seen directions it beats the generic baselines on PSNR/RMSE across the board, with modest gains on most SSIM; the adapter and incremental training ablations are sensible. That part is a solid engineering contribution.\n\nThe soft spot is the claim about a 'geometrically aligned' shared latent space. Stage I trains five independent VAEs with a KL penalty to a standard normal; nothing forces cross-modal correspondence. The scaling factors normalize magnitudes, not directions. So after Stage I you have five equal-dimensional Gaussian-ish distributions, not one common manifold. The diffusion backbone then learns a separate mapping from each source latent space to each target under paired supervision; those mappings need not compose transitively. The residual adapters correct per-target mismatch only for seen targets. So the zero-shot results for SAR-PAN, NIR-PAN, etc., have no grounded mechanism—they might work empirically, but the paper gives only qualitative examples, no quantitative evaluation. The 'O(1)' framing is also loose: the shared backbone is constant, but you still need the source encoder and target decoder, and you store N VAE pairs.\n\nI'd also flag two smaller things. The comparison omits the RS-specific methods discussed in Related Work—S3OIL, RLIDM, DOGAN—so 'state of the art' isn't really demonstrated. And there are no error bars or significance tests, which matters more when the gaps are small (e.g., SSIM on some tasks).\n\nThe RGB pivot issue the stress-test raised is real but maybe less fatal than it sounds: even if Sentinel-2 RGB and WorldView RGB are different domains, the model sees both under the same label, so the encoder has to handle both; the question is whether the shared latent space is consistent enough for transitive composition. The paper doesn't show that, and it could with simple analyses—CCA or retrieval between latent spaces, or a small quantitative zero-shot test on the few held-out pairs.\n\nNet: worth giving a serious referee. The dataset alone justifies it, and the architectural critique is addressable with added experiments. I'd ask for evidence of alignment and the missing baselines, but I wouldn't desk-reject it.","headline":"Useful dataset and unified framework; the 'geometrically aligned latent space' claim is overreaching, but the paper deserves serious peer review.","tokens_in":19327,"tokens_out":2623,"would_cite":true,"duration_ms":26147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single latent-diffusion model can translate any remote-sensing modality into any other, beating per-pair models.","keywords":["remote sensing","cross-modal translation","latent diffusion","any-to-any translation","multimodal dataset","zero-shot generalization","shared latent space","missing-modality synthesis"],"falsifier":"Take a held-out set of co-registered SAR–PAN and PAN–MS pairs (or split the RST-1M RGB sources by satellite) and compare Any2Any's zero-shot outputs against ground truth with PSNR/SSIM; if the zero-shot pairs are no better than a constant-colour baseline, or if performance drops sharply when the RGB pivot comes from a different satellite than the one used at training, the shared-latent alignment claim fails.","tokens_in":18395,"feed_emoji":"🛰️","tokens_out":4495,"duration_ms":44594,"temperature":0.7,"pith_summary":"The paper claims that cross-modal translation in remote sensing need not be a separate model per modality pair. It proposes Any2Any, one latent diffusion model that treats each sensor image (RGB, SAR, NIR, MS, PAN) as a partial view of a shared scene representation, so any source can be mapped to any target through one shared backbone. To support this, the authors assemble RST-1M, a million-pair dataset spanning five modalities connected through a shared supervision graph, and report that Any2Any outperforms pairwise baselines on all 14 seen translation directions and produces reasonable outputs on six unseen pairs. If true, this collapses the cost of multi-sensor translation from quadratic in the number of modalities to constant, and makes missing-modality synthesis composable.","feed_headline":"One model translates any satellite-imaging modality into any other","feed_subtitle":"A shared latent backbone beats per-pair models on 14 tasks and handles 6 unseen modality pairs.","key_machinery":"The shared latent space Z: each of the five modalities is projected by a separately trained VAE into a common 4×64×64 tensor space, regularized toward a standard normal and scaled by hand-set per-modality factors. The Latent Anchor mechanism then treats the paired target's encoded latent zj as a delta-distribution supervision target, so the shared Diffusion Transformer (DiT) can be trained to regress the clean target latent directly (x0-prediction) instead of predicting noise. Target-indexed residual adapters, zero-initialized and trained with stop-gradient, add a learned correction outside the denoising loop; modality tokens enter through AdaLN conditioning. This trio—unified latent, anchor","core_discovery":"Any2Any's central claim is that once heterogeneous sensor images are embedded into a common latent space by per-modality autoencoders, a single diffusion transformer can predict the target modality's latent from any source modality's latent. The paired target latent acts as a deterministic 'anchor' that turns generation into anchored regression, while target-specific residual adapters, applied once after denoising, correct the systematic mismatches left by independently trained autoencoders. Trained on 14 directed tasks from the connected supervision graph of RST-1M, the model claims state-of-the-art PSNR/SSIM/RMSE on all evaluated pairs and zero-shot generalization to six unseen modality pa","pith_inferences":["If the shared latent is truly scene-aligned, zero-shot translations are compositions of learned pairwise paths; a direct test is whether SAR→PAN output accuracy on held-out co-registered pairs matches SAR→RGB→PAN composition quality.","The dataset pools RGB from Sentinel-based 10 m sources with WorldView pan-sharpened sub-metre RGB under one modality label; if these are not physically interchangeable, the 'shared' space may encode two clusters, and zero-shot results would degrade when source and pivot resolutions differ. This is testable by splitting the RGB test set by source satellite.","The KL weight is tiny and latent alignment relies heavily on manual scaling factors; a stronger alignment check would be whether nearest-neighbour latents across modalities correspond to the same geographic scene more often than chance.","A practical extension the paper leaves implicit: the learned latent anchors could provide synthetic paired supervision for modality pairs that are physically unavailable, effectively growing the supervision graph iteratively."],"forward_implications":["A single trained model can synthesize any missing modality for an observed scene, including pairs never seen together in training.","Adding a new sensor modality requires only training one new VAE and one small adapter; the existing backbone and all other adapters stay frozen.","Supervision can be sparse and transitive: training on a connected graph of modality pairs transfers along paths (e.g., SAR→RGB and RGB→PAN support SAR→PAN).","Inference cost is independent of the number of modalities: one forward pass of the shared backbone plus one adapter.","The O(N²) storage and training burden of direction-specific translators is replaced by O(N) modality encoders/decoders plus one shared network."],"fun_headline_variants":["Any2Any: One model, all satellite modality translations","Satellite image translation: Any2Any unifies all modalities","Any2Any: Zero-shot translation for any satellite sensor pair","Any2Any: Unified latent diffusion for multi-modal satellite data","From any sensor to any other: Any2Any satellite translation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that RGB images from very different satellites and resolutions are interchangeable in the dataset, and that independently trained VAE latents, nudged only by a small KL penalty and manual scaling factors, really do lie in one geometrically aligned shared space; if that alignment fails, the shared backbone is not learning one coherent scene map and the zero-shot results lose their grounding.","fun_headline_variants_meta":{"raw":{"variants":["Any2Any: One model, all satellite modality translations","Satellite image translation: Any2Any unifies all modalities","Any2Any: Zero-shot translation for any satellite sensor pair","Any2Any: Unified latent diffusion for multi-modal satellite data","From any sensor to any other: Any2Any satellite translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1254,"prompt_tokens":732,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":476,"tokens_out":522,"duration_ms":5377,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:54:49.465576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of co-registered SAR–PAN and PAN–MS pairs (or split the RST-1M RGB sources by satellite) and compare Any2Any's zero-shot outputs against ground truth with PSNR/SSIM; if the zero-shot pairs are no better than a constant-colour baseline, or if performance drops sharply when the RGB pivot comes from a different satellite than the one used at training, the shared-latent alignment claim fails.","supporting_citations":[],"review_version":1}