{"id":"c00bd74d-778f-42fa-843b-a0f624733a77","arxiv_id":"2504.19737","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CoDEx trains per-domain expert models on satellite imagery and uses an input-based selection head to combine their predictions, improving spatial generalization on four benchmarks.","lead":"The paper trains one satellite-image model per training region, then a small selection head picks which experts to trust for each new image. The method beats several domain-generalization and adaptation baselines on four Earth-observation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over D3G are 0.3 to 0.4 points with no error bars; without repeated seeds the central 'consistently outperforms' claim is not yet supported.","rationale":"I agree with the reader that the paper is a reasonable extension of multi-expert domain generalization and that the experiments are suggestive but not conclusive. The reader's weakest assumption was the transfer of the selection head's accuracy estimates to unseen target domains. That is a genuine concern, and the reader's evidence from the FMoW oracle row is well taken. However, I see an even more directly load-bearing issue with the central empirical claim: the headline numbers are separated from the strongest baseline by only 0.3 to 0.4 points, and no measure of variability is reported anywhere. Without multiple seeds or error bars, the paper cannot support the word 'consistently' in its own strongest claim. This is not an accusation of selective reporting; it is a gap in the evidence required to distinguish the method from its closest competitor. The D3G baseline being self-implemented makes the comparison especially fragile, since D3G is the only method that approaches CoDEx's performance. A rerun under varied seeds, with the D3G implementation released, would settle whether the gains are real. The reader's conditional verdict already accounts for this risk, so I recommend no change to the verdict, but I would elevate the missing variance to the primary required revision.","tokens_in":10406,"tokens_out":4981,"duration_ms":54987,"concrete_test":"Run the full CoDEx and D3G pipelines on all four datasets with at least 5 random seeds, keeping the same hyperparameters and data splits. Report the mean and 95% confidence interval for each metric, and perform a paired test between CoDEx and D3G. If the confidence intervals overlap and the paired test is not significant at p < 0.05, the central claim should be softened from 'consistently outperforms' to 'comparable within noise.' As part of the same check, release the D3G implementation and verify it reproduces the paper's reported D3G numbers when run independently.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is made in Section 4.2: 'Our method consistently outperforms competing methods across all metrics, except D3G, which is slightly better on OSCD-3ch.' The evidence for this is Table 2. The margins over the strongest baseline, D3G, are small: DynamicEarthNet 39.1 vs 38.7 mIoU (+0.4), MUDS 64.2 vs 63.9 mIoU (+0.3), FMoW 53.9 vs 53.5 accuracy (+0.4), and OSCD-3ch 47.8 vs 48.1 F1 in D3G's favor. No standard deviation, confidence interval, or number of seeds is reported in Section 3.3 or Section 4, so these differences cannot be distinguished from run-to-run noise. Compounding this, D3G is a self-implemented baseline: the paper states 'results are from our own implementation, which we will release.' If that implementation is mildly unfavorable to D3G, the claimed superiority on three of four datasets could vanish. This is load-bearing because the abstract and conclusion infer 'consistent gains' and 'outperforming ten state-of-the-art methods' directly from these sub-point margins. The reader's concern about the selection head's source-only training is also real: the FMoW oracle gap (91.1 vs 53.9) shows the selection mechanism is far from ideal. However, an imperfect selection mechanism could still leave the empirical ranking intact; the missing variance is what blocks any assessment of that ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoDEx, a domain-generalization framework for satellite imagery in which one expert is trained per training domain on a shared backbone, an affinity matrix between domains is learned rather than handcrafted, a consistency loss encourages similar experts to agree, and a second-stage selection head predicts per-expert accuracy from input features and uses a softmax mixture to aggregate expert predictions at test time without target-domain data. The method is evaluated on DynamicEarthNet, MUDS, OSCD-3ch, and FMoW across segmentation, change detection, and classification, with ablations over the affinity formulation, the two selection losses, temporal pooling, and backbone choice. The central claim is that CoDEx consistently outperforms ten prior domain-generalization and domain-adaptation methods.","tokens_in":10753,"tokens_out":3834,"duration_ms":38225,"significance":"If the empirical claims hold, CoDEx would be a useful contribution: it is a clearly specified, parameter-light mechanism for multi-expert spatial generalization that avoids geographic metadata at test time, and the paper covers four datasets, three tasks, and multiple backbones with public code. The ablations in Section 4.3 are informative and support the role of the consistency loss and the accuracy-prediction loss within the proposed pipeline. However, the headline result rests on sub-point margins over a self-implemented baseline, with no reported variance, and the transfer of the source-trained selection head to unseen domains is not analyzed; these issues currently limit the strength of the conclusions.","major_comments":[{"comment":"The central quantitative claim that CoDEx 'consistently outperforms competing methods' is supported only by differences of 0.3 to 0.4 points over D3G on DynamicEarthNet, MUDS, and FMoW, and by a 0.3-point deficit on OSCD-3ch, with no standard deviations, confidence intervals, number of seeds, or significance tests reported anywhere in Sections 3.3 or 4. Without this variance information, these margins cannot be distinguished from run-to-run noise, and the abstract's and conclusion's stronger wording is not justified. Please run the main comparison over multiple seeds and report means with confidence intervals or significance tests, and do the same for the ablations in Tables 3 and 4, where the learned-affinity advantage is only 0.1-0.2 points.","section":"Section 4.2, Table 2"},{"comment":"The selection head h_select is trained exclusively on source-domain samples (Section 3.2), and the paper's framework depends on the assumption that its accuracy estimates remain informative on unseen target domains. The FMoW result in Table 2—Oracle 91.1 versus CoDEx 53.9—shows a large gap between the best per-sample expert and the learned mixture, which sits uneasily with the Section 4.2 claim that the selection mechanism 'effectively identifies the best experts' on MUDS. Please provide a direct analysis of selection-head transfer: for example, report the correlation between predicted and true per-expert accuracy on target test samples, or the fraction of test samples where the top-weight expert is among the top few true experts, per dataset. This would clarify whether the source-only selection module is genuinely transferable.","section":"Section 3.2 and Section 4.2, FMoW row"},{"comment":"D3G, the strongest baseline, is a self-implementation ('results are from our own implementation, which we will release'), and the paper does not specify which components of the original D3G were used, what domain metadata was supplied, or how its hyperparameters were tuned. Since the reported CoDEx advantage over D3G is at most 0.4 points, a slightly unfavorable implementation could change the ranking. Please add implementation details for D3G, state the number of seeds used for all baselines, and, if at all possible, release the D3G implementation and checkpoints alongside the code so the comparison is reproducible.","section":"Section 4.2 and Section 3.3"}],"minor_comments":[{"comment":"The caption reads 'Combing Domain Experts'; this should be 'Combining Domain Experts'.","section":"Figure 1"},{"comment":"The text says Figure 4 shows 'DynamicEarthNet (i–iv), FMoW (v–vii), and OSCD-3ch. (viii)', while the figure caption says patches (v-vii) are from MUDS; the text then discusses MUDS in the following sentences. This inconsistency should be fixed.","section":"Section 4.2, Qualitative Results"},{"comment":"The dataset name is spelled 'DynEarthNet' in Table 1 and 'DynamicEarthNet' elsewhere; please use one consistent spelling.","section":"Table 1 and Table 2"},{"comment":"The vertical axes of the four panels start at 30, 55, 40, and 45 respectively, which visually exaggerates the improvements; consider starting all axes at zero or at least noting the truncated axes in the caption.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is potentially solid but the statistical evidence is currently insufficient for the strength of the claims. The missing error bars and the self-implemented D3G baseline are both fixable within a revision; I would look for multiple-seed results with significance information and a concrete analysis of the selection head's domain transfer before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely an improvement over D3G: instead of handcrafting an affinity between domains from geographic metadata, they learn it directly, and they replace metadata-based aggregation with an input-driven selection head trained via accuracy prediction and mixture supervision. That is a clean, sensible evolution of the multi-expert DG line, and the paper does a good job of explaining it. Second, the empirical evidence for the central claim is weaker than the abstract suggests. The reported margins over D3G are 0.4, 0.3, and 0.4 points on three datasets, with D3G up by 0.3 on the fourth, and no standard deviations, confidence intervals, or number of seeds anywhere. Those differences are within run-to-run noise for deep models, so the sentence 'consistently outperforms competing methods' is not yet supported.\n\nWhat the paper does well: the method is clearly specified; the ablations decompose the contributions of the consistency loss, the mixture loss, and the accuracy loss; the results across three backbones show the gains are not tied to one architecture; and they release code. The learned affinity matrix also shows a modest but real advantage over handcrafted and D3G-style affinities in their Table 4, which is the cleanest evidence the paper offers. The selection module is a nice contribution even if it is not as accurate as an oracle.\n\nThe soft spots are real but proportionate. The missing variance is the load-bearing issue; without it, the ranking against D3G could flip. The D3G baseline is self-implemented, which compounds the uncertainty. The selection head's source-only accuracy estimates are assumed to transfer to unseen domains, and the FMoW oracle gap (91.1 vs. 53.9) shows that transfer is far from perfect, though the authors at least acknowledge the oracle gap. The final claim of 'outperforming ten state-of-the-art methods' is an overreach given the margins. None of these are fatal to the method as a proposal, but they should be fixed before publication.\n\nWho this is for: anyone working on spatial domain generalization in Earth observation, or on multi-expert approaches generally. It is worth a serious referee, because the method is sound and the problem is important. I would recommend sending it to review with a request for multi-seed variance and significance testing, and for the authors to release their D3G implementation. A revised version with those additions would be a solid paper.","headline":"A well-built incremental extension of D3G with a learned affinity matrix and a learned expert-selection head, but the headline claim of consistent gains over D3G rests on sub-point margins without error bars.","tokens_in":11241,"tokens_out":1289,"would_cite":true,"duration_ms":13985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoDEx claims that training one expert per training region and learning to mix their predictions with a source-only selection head yields consistent spatial generalization gains over existing domain-generalization and adaptation methods on…","keywords":["domain generalization","satellite imagery","multi-expert models","expert selection","learned domain similarity","semantic segmentation","change detection","land-use classification"],"falsifier":"Evaluate the selection head on a held-out target domain and compute, per test sample, the rank correlation between the accuracy vector it predicts and the actual per-expert accuracy; if that correlation is near zero or negative on new domains, the mixture weights are not transferring and CoDEx would reduce to a fixed expert average.","tokens_in":10231,"feed_emoji":"🛰️","tokens_out":6358,"duration_ms":56537,"temperature":0.7,"pith_summary":"CoDEx is a framework for making satellite-image models generalize to locations never seen during training. Rather than learning a single model that must work everywhere, it trains one expert for each region in the training set, learns which regions are similar to one another, and uses that similarity to keep similar experts consistent. A lightweight selection module then examines a test image and weights the experts' predictions accordingly, without needing any labels, metadata, or retraining on the target region. The paper reports consistent gains over existing domain-generalization methods and even over domain-adaptation methods that do use unlabeled target images, across four satellite benchmarks and three tasks.","feed_headline":"One model per region beats single models on unseen satellite imagery","feed_subtitle":"CoDEx trains a specialist per training region and a selection head that mixes them at test time; no target data needed.","key_machinery":"The framework rests on three pieces: per-domain experts, a learned affinity matrix, and a selection head. Each expert is a shared backbone plus a small head for its own training domain. The affinity matrix is a row-wise softmax over a learnable D-by-D parameter matrix, and it feeds a consistency loss that makes an expert from a domain similar to d also predict correctly on samples from d. The selection head h_select takes backbone features from a test sample, outputs a predicted accuracy for each expert, and turns those values into mixture weights with a softmax; a mixture loss supervises the weighted combination directly. At inference only the weighted mixture is used, so the method needs no geographic metadata, no target labels, no target images for adaptation, and adds only a few thousand parameters (roughly $D^{2}$ for the affinity plus the head).","core_discovery":"The central claim is that spatial domain shift in satellite imagery is better handled by a multi-expert design than by any single domain-invariant model. CoDEx trains D experts, one per source domain, that share a backbone but keep their own output heads, and a fully learned affinity matrix that encourages experts of similar domains to agree. A selection head, trained only on source data, predicts each expert's accuracy on a given input and converts those estimates into mixture weights, with a second loss supervising the final mixture directly. The paper asserts that this beats all compared approaches on DynamicEarthNet, MUDS, and FMoW, and is slightly behind only the D3G method on OSCD-3ch, while also beating domain-adaptation methods that used target-domain inputs; on all four benchmarks only CoDEx and D3G exceed the single-model baseline.","pith_inferences":["Beyond the paper: the selection-head mechanism is backbone-agnostic, so the same expert-and-mixture recipe could plausibly extend to other sensor modalities, temporal sampling protocols, or foundation-model features; this specific claim is not tested in the paper.","Beyond the paper: because the affinity matrix is fully learned, it offers a natural probe for which properties drive domain similarity; comparing learned affinities to climate or land-cover categories would be a direct test of whether geographic proximity is being replaced by functional similarity.","Beyond the paper: the large oracle gap on FMoW (91.1 versus 53.9) suggests the selection head becomes the bottleneck as the number of experts grows, so a follow-up could test whether a learned router with larger capacity or a hierarchical grouping of experts closes part of that gap."],"forward_implications":["Spatial generalization in Earth observation can improve without target-domain data by replacing one generalist model with a set of specialists and a learned combiner.","The same recipe transfers across tasks: semantic segmentation, binary change detection, and 62-class land-use classification, and across three backbone architectures.","Because the selection head reads only the input image, the method applies directly to unseen locations with no fine-tuning and no metadata at test time.","The computational overhead is small: roughly 14% extra training time, about 213K additional parameters for 55 domains, and about 6% extra inference time."],"supporting_citations":[{"why":"provides the multi-expert aggregation baseline (D3G) that CoDEx extends with a learned affinity matrix and a selection head.","marker":"[40]"},{"why":"provides the baseline model and the MultiUTAE architecture that all compared methods, including CoDEx, are built on.","marker":"[36]"},{"why":"documents that single domain-invariant models often fail on in-the-wild shifts, the motivation for learning per-domain experts.","marker":"[14]"},{"why":"supplies the DynamicEarthNet benchmark with spatially separated domains used for the first segmentation evaluation.","marker":"[33]"},{"why":"supplies the MUDS benchmark used for the second segmentation evaluation.","marker":"[34]"},{"why":"supplies the OSCD change-detection benchmark used for the third evaluation.","marker":"[7]"},{"why":"supplies the FMoW land-use classification benchmark with country-level spatial domains for the fourth evaluation.","marker":"[5]"}],"fun_headline_variants":["Mix per-domain experts to beat single models on unseen satellite imagery","Instead of one model, CoDEx trains multiple experts and picks the right mix","Per-domain experts plus a selection head: CoDEx for satellite imagery","Forget one-size-fit-all: CoDEx trains an expert per region","Train an expert per region, then pick the best for each scene"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The selection head, trained only on source domains, keeps predicting which expert is best when it sees an image from a never-seen domain; the FMoW oracle gap (91.1 versus 53.9) shows it is far from perfect, so the whole gain depends on this transfer.","fun_headline_variants_meta":{"raw":{"variants":["Mix per-domain experts to beat single models on unseen satellite imagery","Instead of one model, CoDEx trains multiple experts and picks the right mix","Per-domain experts plus a selection head: CoDEx for satellite imagery","Forget one-size-fit-all: CoDEx trains an expert per region","Train an expert per region, then pick the best for each scene"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001693,"raw_usage":{"total_tokens":6662,"prompt_tokens":854,"completion_tokens":5808,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":5712}},"tokens_in":470,"tokens_out":5808,"duration_ms":36594,"temperature":1.0,"reasoning_tokens":5712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:44:29.085281+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the selection head on a held-out target domain and compute, per test sample, the rank correlation between the accuracy vector it predicts and the actual per-expert accuracy; if that correlation is near zero or negative on new domains, the mixture weights are not transferring and CoDEx would reduce to a fixed expert average.","supporting_citations":[{"cited_title":"Urban change detection for multispectral earth observation using convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"supplies the OSCD change-detection benchmark used for the third evaluation."},{"cited_title":"Improving domain generalization with domain relations","cited_arxiv_id":null,"evidence_quote":"provides the multi-expert aggregation baseline (D3G) that CoDEx extends with a learned affinity matrix and a selection head."},{"cited_title":"WILDS: A benchmark of in-the-wild distribution shifts","cited_arxiv_id":null,"evidence_quote":"documents that single domain-invariant models often fail on in-the-wild shifts, the motivation for learning per-domain experts."},{"cited_title":"DynamicEarthNet: Daily multi-spectral satellite dataset for semantic change segmentation","cited_arxiv_id":null,"evidence_quote":"supplies the DynamicEarthNet benchmark with spatially separated domains used for the first segmentation evaluation."},{"cited_title":"The multi-temporal urban development SpaceNet dataset","cited_arxiv_id":null,"evidence_quote":"supplies the MUDS benchmark used for the second segmentation evaluation."},{"cited_title":"Functional map of the world","cited_arxiv_id":null,"evidence_quote":"supplies the FMoW land-use classification benchmark with country-level spatial domains for the fourth evaluation."}],"review_version":1}