{"id":"c84d4cf3-0de5-4f03-9aed-a2d302e79bcd","arxiv_id":"2412.13394","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TARDIS detects out-of-distribution satellite images by clustering a model's internal activations to create surrogate labels, then training a binary classifier on those labels.","lead":"This paper introduces TARDIS, a method that flags satellite images which look unfamiliar to an already-trained model, without needing any labeled examples of unfamiliar images. A smart generalist might read it because it addresses a practical obstacle in deploying global Earth observation models: knowing when the model is operating outside its training distribution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's TARDIS numbers are inconsistent with the per-setup g* results in Table 2; the claim that TARDIS matches state-of-the-art detectors is unsupported until this discrepancy is resolved.","rationale":"The reader's weakest assumption concerned feature separability and transferability of k/T hyperparameters. Those are legitimate concerns, but the most load-bearing issue I find is the internal numerical inconsistency between Table 2 and Table 3. The abstract and Section 6 make two connected claims: g* approaches the oracle in 13 of 17 AUROC setups, and TARDIS matches or exceeds strong post-hoc baselines. Table 2 supports the first claim; Table 3 supports the second. But the arithmetic from Table 2 does not reproduce Table 3. Averaging the per-setup g* AUROC values from Table 2 gives roughly 0.875 for semantic shifts and 0.946 for covariate shifts, whereas Table 3 reports 0.95 and 0.98 for the same groups. The FPR95 averages from Table 2 (about 0.34 semantic and 0.21 covariate) are roughly twice Table 3's reported values (0.16 and 0.11). These are not small rounding effects; they indicate either a different evaluation protocol or a reporting error. Because the code is public, this is directly checkable, and the check would settle whether the headline claim is supported. I am not accusing anyone of dishonesty; I am pointing out that two central tables are mutually inconsistent as written. The verdict should remain conditional rather than outright rejection because the discrepancy might be a typo or a protocol difference that the code can clarify. However, acceptance must require the authors to reconcile the tables and state whether Table 2 metrics are computed against ground-truth or surrogate labels. If the discrepancy is not resolved, the paper's core comparison to existing methods is unsupported.","tokens_in":17976,"tokens_out":13784,"duration_ms":127048,"concrete_test":"Run the released code (github.com/microsoft/geospatial-ood-detection) to reproduce Table 2 and Table 3 from the same protocol. For each of the 17 setups, compute the g* AUROC and FPR95 on the same test split, then average within the semantic and covariate groups. Check whether the averages match the TARDIS rows in Table 3 (semantic AUROC 0.95 +/- 0.02, FPR95 0.16 +/- 0.03; covariate AUROC 0.98 +/- 0.01, FPR95 0.11 +/- 0.04). Also inspect the code to determine whether Table 2's AUROC/FPR95 are computed against ground-truth OOD labels or against surrogate labels; if the latter, the comparison to the oracle classifier is invalid and the near-oracle claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim has two pillars: g* approaches the oracle (Table 2) and TARDIS matches top post-hoc methods (Table 3). These two tables are presented as evaluating the same 17 setups, but their numbers cannot both be correct. Averaging the 10 semantic-shift g* AUROC values in Table 2 (0.97, 0.88, 0.56, 0.96, 0.95, 0.89, 0.82, 0.81, 1.00, 0.91) gives approximately 0.875, while Table 3 reports TARDIS semantic AUROC as 0.95 +/- 0.02. For the 7 covariate-shift setups, the Table 2 g* AUROC average is about 0.946, while Table 3 reports 0.98 +/- 0.01. The FPR95 discrepancies are larger: the semantic g* average from Table 2 is roughly 0.34 versus Table 3's 0.16 +/- 0.03, and the covariate average is roughly 0.21 versus 0.11 +/- 0.04. These differences are far outside the reported standard deviations. Unless Table 3 was produced with a different, unspecified protocol (for example, using the oracle classifier, pooling test sets, or evaluating against ground-truth rather than surrogate labels), the headline comparison to ReAct and NAP cannot hold simultaneously with Table 2. The paper must state explicitly which labels are used to compute g* AUROC/FPR95 in Table 2, and the released code should reproduce both tables from the same per-setup evaluations.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TARDIS is a post-hoc OOD detection method for Earth Observation. Given a pre-trained model f, an ID set, and unlabeled WILD data, it extracts activations from a chosen layer, clusters the combined ID+WILD feature space with k-means, assigns surrogate ID/OOD labels based on the ID fraction per cluster (threshold T), trains a logistic regression classifier g on these labels, and uses g at inference. The paper evaluates on EuroSAT and xBD across 17 covariate/semantic shift setups, comparing g* (surrogate labels) with an oracle classifier goracle (true labels), and compares TARDIS with MSP, Energy, Mahalanobis, ReAct, and NAP. It also reports a real-world deployment on the Fields of the World dataset. The central claims are that g* approaches the oracle in 13/17 AUROC and 17/17 FPR95 comparisons, that TARDIS matches or beats state-of-the-art post-hoc detectors, and that the method scales to large geographic regions.","tokens_in":18331,"tokens_out":9229,"duration_ms":77839,"significance":"The proposed idea is attractive: it avoids OOD labels, preserves ID task performance, and the deployment study is a genuine attempt to show scalability. The code release and use of public datasets are strengths. If the results are correct, TARDIS would be a useful tool for monitoring geospatial models. However, the current evidence is not yet convincing. The numerical inconsistency between Tables 2 and 3, the ambiguity about which labels are used for evaluation, and the selection of hyperparameters and layers on the evaluation setups all affect the central empirical claims. The significance is therefore conditional on resolving these issues.","major_comments":[{"comment":"Table 3 cannot be reconciled with Table 2. For the ten semantic-shift setups, the g* AUROC values in Table 2 average 0.875 and the FPR95 values average 0.34, while Table 3 reports TARDIS semantic AUROC 0.95 ± 0.02 and FPR95 0.16 ± 0.03; the covariate averages from Table 2 are 0.946 and 0.20 versus 0.98 ± 0.01 and 0.11 ± 0.04 in Table 3. These discrepancies are far outside the reported standard deviations. The paper must specify the exact protocol for Table 3 (which labels are used, whether test sets are pooled, and which classifier is evaluated) and provide code that reproduces both tables from the same per-setup evaluations.","section":"Tables 2 and 3"},{"comment":"Section 5 says g* is 'trained and evaluated using surrogate labels' while goracle is trained and evaluated with true labels. If Table 2 reports AUROC/FPR95 against surrogate labels, then the near-oracle comparison measures how well the classifier reproduces the clustering assignment, not how well TARDIS detects true OOD samples. The evaluation labels must be identical for a valid comparison. Please report g* metrics against ground-truth labels as well, and state explicitly whether Table 3 uses ground-truth labels.","section":"Section 5, Table 2"},{"comment":"The hyperparameters k and T are selected on the same 17 setups whose results are then reported: Section 10.4 says 'we run 20 independent experiments and select the best-performing (k, T) pair,' and Section 10.1 says 'we select the best-performing layer for each experiment.' The caption of Table 3 claims TARDIS requires 'no hyperparameter tuning,' but the reported numbers are obtained after tuning on the evaluation sets. This selection bias must be addressed, for example by fixing the layer/k/T protocol a priori or by using a separate held-out validation set, and the claim of no tuning should be revised or removed.","section":"Sections 6, 10.1, and 10.4"},{"comment":"There is a direct numerical contradiction for the FTW deployment: Section 7 fixes k = 0.3 × 1200 = 360, while Supplementary Section 9.3 states k = 150, 'calculated as 0.3 times the total number of WILD samples' (0.3 × 1200 would be 360). The authors should state the correct value and ensure the released code uses it.","section":"Section 7 vs. Supplementary Section 9.3"}],"minor_comments":[{"comment":"The reported best-fit line has R² = 0.16, which is a weak correlation; the claim that 'low skewness may signal the presence of OOD samples' should be tempered accordingly.","section":"Figure 4"},{"comment":"The composite objective H(S) + Pmis-ID − Pcorr-ID mixes an entropy term with proportions that require ground-truth ID/OOD labels to compute; the section should clarify that this objective is only used in the controlled experiments where labels are available, and that deployment uses the fixed values.","section":"Section 10.4, Eq. (1)"},{"comment":"The notation for M is inconsistent: Section 4 defines M as the number of ID samples, while Section 6 uses M for the total number of samples when setting k = 0.3 × M. Please define all symbols consistently.","section":"Sections 4 and 6"},{"comment":"The table reports 'mean ± SEM' while Section 6 says 'mean and standard deviation'; please specify which one is shown and use consistent terminology.","section":"Table 7"},{"comment":"The note 'All conditions are post-event except where specified' is unclear for the Portugal Wildfire row; please clarify what 'pre' and 'post' refer to in that setup.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The Table 2 versus Table 3 discrepancy is the most serious issue and should be the focus of the revision. If the authors cannot reconcile the two tables with a single, clearly specified evaluation protocol, the central comparison with existing methods is unsupported. I would also ask the authors to report g* metrics against ground-truth labels; otherwise the 'near-oracle' language in Table 2 is misleading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea: cluster ID + WILD activations, label clusters by ID fraction, train a logistic regression on those labels, and use it as a post-hoc OOD detector. That specific combination is new, and it is a sensible way to avoid OOD labels and retraining. The FTW deployment is the best part: fixed k/T, 1200 unlabeled Sentinel-2 scenes, 0.003 s per patch, and a geographic pattern (arid/polar regions flagged OOD) that is interpretable and useful. Code is released. I want the method to work.\n\nBut the evaluation has a load-bearing problem. Section 5 says g* is 'trained and evaluated using surrogate labels.' That means Table 2's g* AUROC/FPR95 are computed against the same clustering-derived labels used to train the classifier, not against the true OOD labels. Comparing that with goracle, which is evaluated on true labels, is not an upper-bound comparison. It mostly measures how well the classifier reproduces the clustering. The '13 of 17 not statistically significant' claim cannot support the conclusion that surrogate labels approach oracle performance.\n\nThe stress-test note is correct and easy to verify. Averaging the per-setup g* AUROC values in Table 2 gives roughly 0.88 for semantic shifts and 0.95 for covariate shifts; FPR95 averages are about 0.34 and 0.21. Table 3 reports TARDIS means of 0.95 and 0.98 AUROC and 0.16 and 0.11 FPR95 on the same 17 setups. The differences are far outside the stated error bars. Either Table 3 was produced with a different protocol (oracle labels? pooled test sets? different layer selection?) or one of the tables is wrong. The paper has to state exactly which labels and protocol were used for each table, and the released code should reproduce both.\n\nTwo smaller issues: Section 10.1 selects the best activation layer per experiment, and Section 10.4 describes running 20 searches and picking the best (k,T). That is tuning on the evaluation setups, so the 'no hyperparameter tuning' claim should be limited to the FTW deployment, where k and T are indeed fixed. And the comparison of g* to existing methods in Table 3 needs the same per-setup breakdown so readers can see the variance, not just means.\n\nWho should read it: geospatial ML practitioners who want a lightweight post-hoc OOD detector for a deployed model. They'll get a plausible recipe and a useful deployment case study, but they should not trust the headline benchmark numbers until the protocol is fixed. I don't think the idea is dead; the pipeline is plausible and the deployment is genuine. But the benchmark claims need a clean protocol: fix all choices on a held-out development set, report against ground-truth labels, and reconcile Tables 2 and 3. I'd send this to reviewers with a request for major revision rather than desk-reject it.","headline":"A plausible label-free OOD pipeline with a real deployment story, but the benchmark tables don't line up and g* is evaluated against its own surrogate labels.","tokens_in":18864,"tokens_out":4724,"would_cite":false,"duration_ms":40503,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a pre-trained geospatial model can detect distribution shifts without any OOD labels by clustering its own internal activations.","keywords":["out-of-distribution detection","distribution shift","Earth observation","surrogate labels","feature-space clustering","satellite imagery","post-hoc detection","geospatial deployment"],"falsifier":"Take a domain pair where ID and OOD images are known to overlap heavily in the chosen activation layer, apply TARDIS with the fixed hyperparameters k=0.3M and T=0.1, and measure surrogate label accuracy against true labels; if it nears chance, or if on a new deployment dataset with ground-truth OOD labels (e.g., manually labeling a subset of the FTW WILD pool) the AUROC gap between g* and the oracle exceeds about 0.1, the central claim would be refuted.","tokens_in":17768,"feed_emoji":"🛰️","tokens_out":6826,"duration_ms":52937,"temperature":0.7,"pith_summary":"TARDIS aims to make out-of-distribution (OOD) detection possible for geospatial models without any labels for shifted data. The paper's proposal is to take a pre-trained model, known in-distribution (ID) samples, and a mixed pool of unknown 'WILD' samples, cluster their internal activations, and label each cluster as surrogate-ID or surrogate-OOD based on the fraction of known ID samples it contains. A logistic regression classifier trained on these surrogate labels then acts as the deployed shift detector. Across 17 EuroSAT and xBD setups covering covariate and semantic shifts, the surrogate-trained classifier is statistically indistinguishable from an oracle trained on true labels in all FPR95 comparisons and in 13 of 17 AUROC comparisons, while the original model's task performance is untouched. If this holds, global satellite models can be audited for distribution shifts at scale without collecting OOD examples.","feed_headline":"Clustering activations finds distribution shifts with no shift labels","feed_subtitle":"TARDIS matches a ground-truth oracle on all 17 shift-detection benchmarks, with no shifted labels.","key_machinery":"The load-bearing mechanism is surrogate label assignment: max-pooled internal activations from known ID samples and unknown WILD samples are pooled together and clustered with k-means; a cluster is labeled surrogate-ID if the fraction of ID samples in it reaches a threshold T, and surrogate-OOD otherwise. The paper fixes k at 30% of the training sample count and T at 0.1, based on tuning curves on EuroSAT and xBD that show performance approaching the oracle as k grows. A logistic regression classifier is then fit on the surrogate-labeled activations and becomes the detector g; clustering alone is insufficient, since removing the classifier drops ROC AUC for the River experiment from 97% to 79%. The mechanism's work is to convert an unlabeled WILD pool into a labeled training set, so the OOD detector is trainable without true OOD labels.","core_discovery":"The central claim is that surrogate labels produced by k-means clustering of internal activations, with an ID-fraction threshold, are accurate enough that a binary classifier trained on them approaches an oracle classifier that uses ground-truth ID/OOD labels. The paper reports the gap is not statistically significant (p < 0.05) in 13 of 17 setups for AUROC and in all 17 setups for FPR95. This holds across covariate shifts (geographic, temporal, disaster type) and semantic shifts (withheld classes) on two very different tasks, patch classification and semantic segmentation. The method is post-hoc, requires no OOD data, and leaves the base model unchanged, so it is offered as a practical detector for real-world geospatial deployment rather than a maximally accurate one.","pith_inferences":["If the surrogate-labeling logic generalizes, the same recipe (cluster activations, threshold by ID fraction, fit a linear classifier) could serve as a label-free shift monitor for any pre-trained vision model, not only Earth observation.","The fixed k=0.3M suggests the method may be sensitive to the ID/OOD ratio within the WILD pool; a direct test would vary this ratio and measure the gap to the oracle.","Max pooling outperforming mean pooling implies the most salient activation per channel carries the shift signal, which could be probed with attention-based pooling to see if the oracle gap closes further.","The paper's assumption that activation clusters correspond to distribution distance could be cross-checked against calibrated uncertainty scores from the base model; where the two disagree, one of them is mislabeling."],"forward_implications":["Geospatial models can be screened for distribution shifts at inference time without any OOD labels, making monitoring feasible in low-data regions.","Because the base model is never modified, ID task performance is preserved, avoiding the trade-off that limits activation-manipulation methods.","The fixed hyperparameters (k=0.3M, T=0.1) appear to transfer across datasets and tasks, so per-deployment tuning is unnecessary.","The detector is fast enough (0.003 s per 256×256 patch) to screen continental-scale areas on a single GPU, enabling spatial diagnostics of model reliability.","Skewness of the detector's scores correlates with the base model's test performance, so the method can flag questionable test splits as well as deployment shifts."],"supporting_citations":[{"why":"Supplies the premise that OOD samples trigger internal activation patterns diverging from ID samples, which TARDIS exploits.","marker":"[25]"},{"why":"Supplies both that premise and a baseline activation-based method TARDIS is compared against.","marker":"[30]"},{"why":"Provides the EuroSAT benchmark dataset used for patch-level classification experiments.","marker":"[12]"},{"why":"Provides the xBD dataset used for semantic segmentation experiments.","marker":"[10]"},{"why":"Provides the Fields of the World dataset and pre-trained models used for the large-scale deployment study.","marker":"[14]"},{"why":"Baseline score-based OOD detection method (Mahalanobis distance) used in comparisons.","marker":"[19]"},{"why":"Baseline maximum softmax probability method used in comparisons.","marker":"[13]"}],"fun_headline_variants":["Clustering activations catches distribution shifts without shift labels","TARDIS: post-hoc OOD detection from in-distribution data alone","Activation clustering yields shift labels for geospatial OOD","Surrogate labels from activations match oracle OOD detection","No OOD data needed: cluster activations to spot shifts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on ID and OOD features being separable enough in the chosen activation layer that k-means with an ID-fraction threshold yields mostly correct surrogate labels; if clusters mix ID and OOD heavily, the surrogate-trained classifier cannot approach the oracle.","fun_headline_variants_meta":{"raw":{"variants":["Clustering activations catches distribution shifts without shift labels","TARDIS: post-hoc OOD detection from in-distribution data alone","Activation clustering yields shift labels for geospatial OOD","Surrogate labels from activations match oracle OOD detection","No OOD data needed: cluster activations to spot shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1318,"prompt_tokens":941,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":557,"tokens_out":377,"duration_ms":4036,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:09:54.408614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a domain pair where ID and OOD images are known to overlap heavily in the chosen activation layer, apply TARDIS with the fixed hyperparameters k=0.3M and T=0.1, and measure surrogate label accuracy against true labels; if it nears chance, or if on a new deployment dataset with ground-truth OOD labels (e.g., manually labeling a subset of the FTW WILD pool) the AUROC gap between g* and the oracle exceeds about 0.1, the central claim would be refuted.","supporting_citations":[{"cited_title":"Detection of out- of-distribution samples using binary neuron activation pat- terns, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that OOD samples trigger internal activation patterns diverging from ID samples, which TARDIS exploits."},{"cited_title":"React: Out-of- distribution detection with rectified activations, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies both that premise and a baseline activation-based method TARDIS is compared against."},{"cited_title":"Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the EuroSAT benchmark dataset used for patch-level classification experiments."},{"cited_title":"xbd: A dataset for assessing building dam- age from satellite imagery, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the xBD dataset used for semantic segmentation experiments."},{"cited_title":"Lavista Ferres, and Jennifer Marcus","cited_arxiv_id":null,"evidence_quote":"Provides the Fields of the World dataset and pre-trained models used for the large-scale deployment study."},{"cited_title":"A simple unified framework for detecting out-of-distribution samples and adversarial attacks, 2018","cited_arxiv_id":null,"evidence_quote":"Baseline score-based OOD detection method (Mahalanobis distance) used in comparisons."}],"review_version":1}