{"id":"8ac3c16d-487d-4d1c-8f10-c1df448ce177","arxiv_id":"2608.02959","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SABRE, a multi-agent system, selects and reliability-weights post-hoc OOD detectors per domain at deployment, converting a benchmark guess into a measurement from a small labeled sample.","lead":"A benchmark-selected out-of-distribution detector can invert on a new domain, scoring normal inputs as more anomalous than real outliers. The paper introduces SABRE, a three-agent system that selects and reliability-weights detectors at deployment from a small labeled sample, recovering reliable detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing baseline: a single detector chosen from the same calibration labels would beat SABRE on the key inverted domain (~79 vs 76.3) at one call per image, so 'tracks the strongest detector' is unsupported.","rationale":"The paper's empirical core is solid: the detector benchmark in Table 1 shows a real inversion of confidence and concept-matching detectors on histopathology across multiple encoders, and the component analysis honestly attributes the recovery to the Analyst's reliability calibration rather than to the selection order. Those parts deserve credit. My concern is with the method's headline claim that SABRE 'tracks the strongest detector in each domain' and does so under a budget. The paper's own Table 3 contains the ingredients for a much simpler baseline—use the same calibration labels to pick the single best detector—but that baseline is never run or reported. The oracle 'Best single detector' row shows the ceiling is 79.9 on NCT-CRC while SABRE is 76.3, and the calibration-argmax baseline is the practically implementable version of that row. If calibration-argmax indeed outperforms SABRE, then the multi-agent selection mechanism is not only unnecessary but suboptimal relative to a one-line alternative, and the abstract's 'tracks the strongest detector' is contradicted. The reader's conditional verdict is appropriate, but I would add this missing baseline as a required condition before the method claim is accepted.","tokens_in":21985,"tokens_out":10773,"duration_ms":96092,"concrete_test":"For every domain and encoder in Tables 1, 6, and 8-10, compute the calibration-argmax baseline: use the same labeled calibration split C that feeds Eq. (2), estimate A_m(C) for all nine detectors, select m* = argmax_m A_m(C), and apply that detector alone to the held-out test set. Report test AUROC and mean detector calls per image alongside SABRE in Table 3. If this baseline beats SABRE on the inverted histopathology domains, the claim that SABRE tracks the strongest detector under a budget fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's method-level claim is that SABRE 'tracks the strongest detector in each domain' under a bounded query budget. The evaluation never compares SABRE to the most natural calibration-based baseline: use the same labeled calibration set C that the Analyst uses to compute the weights w_m in Eq. (2), rank the nine detectors by their calibration AUROC A_m(C), select m* = argmax_m A_m(C), and deploy that single detector. This baseline is available in the paper's own setup, because C is labeled and disjoint from the scored set, and it costs one detector call per test image, fewer than SABRE's mean 2.6 calls (Table 3). On NCT-CRC, Table 3's oracle 'Best single detector' row is 79.9 while SABRE is 76.3; the calibration-argmax baseline should come close to the oracle whenever A_m(C) estimates the test AUROC well, and the paper gives no evidence that SABRE reaches that level. The claim that SABRE tracks or converges to the strongest detector is therefore not established; a trivial method using the same supervision could dominate SABRE in both AUROC and per-image budget. This does not question the well-supported empirical inversion finding, but it directly affects the central method contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether a post-hoc OOD detector selected on a benchmark remains reliable under domain shift for a frozen vision-language encoder. Using CLIP-family encoders and two domains (natural images and histopathology), the authors show that detectors such as MCM and Energy, which lead on natural images, invert on histopathology by scoring in-distribution tissue as more anomalous than outliers, and that the best detector changes by domain. They introduce SABRE, a multi-agent system with three LLM agents: a Selector chooses detectors under a bounded per-image budget, a Reporter consolidates evidence, and an Analyst computes reliability weights from a labeled calibration set disjoint from the test data. The paper proposes four multimodal density detectors and reports a component analysis showing that the Analyst's reliability weighting, rather than the selection order, drives recovery; random selection with the same weights matches SABRE, and SABRE reaches 76.3 AUROC on NCT-CRC with 2.6 mean detector calls.","tokens_in":22285,"tokens_out":7778,"duration_ms":68795,"significance":"The empirical finding that fixed detector reliability does not transfer across domains is clearly supported by the data and strengthened by careful controls: disjoint calibration and test splits, validation-split tracking that matches the test split, a random-selection control, and paired-bootstrap significance tests. However, the paper's central method claim, that SABRE tracks the strongest detector per domain, is not established because the most natural baseline using the same calibration labels, selecting the single detector with maximum calibration AUROC and deploying it, is never evaluated. Such a baseline costs one detector call per image and would likely approach the oracle 'Best single detector' entries that SABRE fails to reach on the main tissue datasets. The contribution of the LLM agents is therefore unclear beyond budget accounting, and the paper's method-level conclusions need to be re-examined.","major_comments":[{"comment":"The central claim that SABRE 'tracks the strongest detector in each domain' is not supported because the paper omits the calibration-argmax baseline. The labeled calibration set C used to compute w_m = max(0, 2A_m(C)-1) in Eq. (2) can equally be used to rank detectors by A_m(C), select m* = argmax_m A_m(C), and deploy that single detector. This baseline uses exactly the same supervision as SABRE, costs one detector call per image rather than SABRE's mean 2.6 calls (Table 3), and should approximately achieve the 'Best single detector' row: 79.9 on NCT-CRC where SABRE obtains 76.3, and 69.6 on CRC-VAL where SABRE obtains 64.4 (Table 2). The paper gives no evidence that SABRE reaches this baseline's level, so the method-level claim of tracking or converging to the strongest detector is unsubstantiated.","section":"Methodology, 'Analyst and the calibration mechanism' (Eq. 2); Table 3"},{"comment":"Table 3 shows that random selection under the Analyst's weights (76.6) matches SABRE (76.3) on NCT-CRC, and that unweighted selection is near chance. Combined with the component analysis (Table 2), this means the entire accuracy recovery comes from the calibration weights, not from the agent-based selection order. The paper should either demonstrate a domain or dataset where the Selector and Reporter improve over random selection under the same budget and weights, or reframe SABRE as a calibration-weighted ensemble with a budget cap. As written, the multi-agent selection mechanism is not shown to contribute to accuracy.","section":"Experiments, 'What the Recovery Depends On' (Table 3)"},{"comment":"There is an inconsistency in the reported best single detector for NCT-CRC: Table 2 gives 79.6, while Table 3 and supplementary Table 6 give 79.9; this must be reconciled. In addition, the natural-image SABRE results are reported only in prose and Figure 4, with no table of AUROC values and no comparison to the calibration-argmax baseline; such a table is needed to support the claim that SABRE 'matches the strongest fixed detector' on that domain, especially because there the reliability weights are also derived from labeled calibration data.","section":"Table 2 vs. Table 6; natural-image results"}],"minor_comments":[{"comment":"There are formatting artifacts in the abstract and introduction, such as 'SABRE (Selective Agentic Budgeted Reliability Ensemble,)' and 'S elective Agentic BudgetedR eliabilityE nsemble', with stray spaces and punctuation; please correct these.","section":"Abstract and main text"},{"comment":"The selective gate with threshold theta = 0.15 is described in the supplementary material but never mentioned in the main-text experimental protocol; please clarify whether the reported mean calls and AUROC figures include this gate.","section":"Supplementary D.6; main-text experiments"},{"comment":"Table 13 shows that MSP and MCM retain high selection shares even after their calibration weights become zero; since these calls do not affect the aggregate, please explain whether the Selector is aware of zero weights when spending budget, or report an effective weighted-call count.","section":"Table 13"},{"comment":"The four proposed detectors are strongly rank-correlated (Kendall tau between 0.72 and 0.99), which the main text should acknowledge when interpreting pool diversity and the Selector's choices.","section":"Supplementary C.5"},{"comment":"GCHTID is repeatedly called a 'difficulty ceiling', but the +Crit configuration reaches 58.0 against best single 60.7 on ViT-L/14 (Table 6); please define the term more carefully or weaken the ceiling language.","section":"GCHTID discussion"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the missing calibration-argmax baseline is the decisive issue. If the authors cannot show that SABRE at least matches this baseline, the agentic selection contribution will need to be substantially reframed; the empirical inversion finding and the calibration-weighted aggregation are, however, sound and potentially publishable as a measurement-oriented study. I would ask for this baseline and the resulting revision of the central claims before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the empirical inversion result is real and well-supported, while the method-level claim is oversold. On a frozen CLIP encoder, detectors like MCM and Energy genuinely invert on histopathology: near the top on natural images, below chance on tissue. That is a clean, reproducible finding, and the paper's controls are unusually careful: disjoint calibration and test splits, validation-split tracking that matches the test split, a random-selection control, and paired-bootstrap significance tests. The authors also honestly attribute the recovery to the Analyst's reliability weights rather than to the cleverness of the Selector. That is good science and should be credited.\n\nWhat I do not buy is the abstract's 'tracks the strongest detector in each domain.' The evaluation never compares SABRE to the simplest calibration-based baseline: take the same labeled calibration set the Analyst uses, rank the nine detectors by calibration AUROC, and deploy the single best one. That baseline costs one detector call per image, fewer than SABRE's mean 2.6, and on NCT-CRC it should land near the best single detector (79.9) rather than SABRE's 76.3. Their own Table 3 shows the weighted full pool at 78.7 and random selection with the same weights at 76.6, essentially tying SABRE. So selection order contributes little; the reliability weighting does the work, and a single detector chosen with the same supervision would likely do better at lower cost. The paper needs that baseline, and without it the central method claim is not established.\n\nOther soft spots are more minor: the main SABRE results in Table 2 are point estimates without confidence intervals, and no code or data is provided, which matters for a method built on LLM agents with greedy decoding and fixed seeds. The four proposed detectors are variants of a Mahalanobis-plus-caption idea and are fine additions to the pool, but not the core contribution.\n\nDespite the overclaim, this paper deserves a serious referee. The inversion finding alone is important for deployment practice, and the framework is a reasonable research direction. I would send it to review with a clear request: add the calibration-argmax single-detector baseline, report CIs on the main results, and soften the language unless the new baseline is actually beaten. As it stands, SABRE is a working reliability-weighted ensemble, not a detector-selection breakthrough.","headline":"The empirical finding that fixed OOD detectors invert across domains is solid and worth knowing; the SABRE framework is plausible but the paper overstates 'tracks the strongest detector' because it omits the obvious baseline of using the same calibration labels to simply pick the best single detector, which likely beats SABRE at lower cost.","tokens_in":22840,"tokens_out":1897,"would_cite":true,"duration_ms":20434,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An out-of-distribution detector that leads on natural images can invert on a specialized domain, and SABRE replaces that fixed choice with per-regime selection.","keywords":["out-of-distribution detection","domain shift","detector inversion","reliability calibration","vision-language models","multi-agent selection","budgeted inference","post-hoc detectors"],"falsifier":"If a domain pair is found on a frozen encoder where no post-hoc detector inverts and the benchmark-preferred detector remains best, the central claim would collapse; concretely, compute AUROC of MSP, Energy, and MCM on natural images versus any specialized image domain, and if none of them falls below 50 while scoring above 50 on natural images, the inversion premise fails.","tokens_in":21722,"feed_emoji":"🎯","tokens_out":9411,"duration_ms":82256,"temperature":0.7,"pith_summary":"The paper claims that the standard practice of choosing one post-hoc out-of-distribution (OOD) detector on a benchmark and fixing it at deployment is unsafe: on a single frozen encoder, a detector that leads on natural images can invert on a specialized domain, scoring normal inputs as more anomalous than genuine outliers, and the best detector changes from one domain to the next. This matters because a deployed system cannot know in advance which domain it faces, so a fixed choice can be not merely suboptimal but systematically reversed. SABRE addresses this by using three language-model agents at inference: a Selector chooses which detectors to consult under a bounded budget, a Reporter summarizes evidence per image, and an Analyst weights detectors by reliability measured on a small labeled calibration set drawn from the deployment domain. The paper argues that reliability is therefore something to establish at deployment, not assume from a benchmark, and that SABRE does this automatically, recovering reliable detection where a fixed detector inverts and matching the best fixed detector where one already works.","feed_headline":"Benchmark-best detectors can fall below chance after a domain shift","feed_subtitle":"SABRE replaces the fixed choice with per-domain selection, recovering detection where fixed detectors invert.","key_machinery":"The load-bearing identity is the Analyst's reliability weight $w_m = \\max(0, 2A_m(\\mathcal{C})-1)$, computed from each detector's AUROC $A_m$ on a small labeled calibration set $\\mathcal{C}$ drawn from the deployment domain and disjoint from the scored data. Detectors at or below chance ($A_m \\le 0.5$) receive zero weight and are ruled out, preventing an inverted detector from cancelling sound ones in the aggregate. The final score is the weight-adjusted mean of calibrated positions, $s_{\\text{SABRE}}(x) = \\frac{\\sum_{m \\in R(x)} w_m \\tilde{s}_m(x)}{\\sum_{m \\in R(x)} w_m}$; each raw score is mapped to the scale of positions among calibration in-distribution scores. This calibration loop, not the order in which the Selector consults detectors, carries the recovery, with the Selector and Reporter contributing budget efficiency and modest gains.","core_discovery":"On CLIP ViT-B/16, MCM scores 97.7 AUROC on iNaturalist and 31.2 on NCT-CRC; Energy drops from 98.5 to 43.6; MSP drops from 87.9 to 29.9, so the detector that ranks highest on natural-image benchmarks ranks below chance on histopathology. The paper establishes that no fixed post-hoc detector is reliable throughout, and the same pattern recurs on all five encoders, including a domain-specialized biomedical encoder that lifts every detector but leaves the per-domain ordering intact. SABRE's central claim is that reliability can be measured at deployment rather than assumed: three language-model agents select and weight detectors per regime, and the decisive component is the Analyst's reliability calibration, which assigns zero weight to detectors that are at or below chance on a small held-out labeled sample. The result is that SABRE reaches or approaches the best fixed detector in each domain without knowing in advance which detector that is, and a component analysis isolates the calibration step as the source of the recovery.","pith_inferences":["Editorial inference: the inversion phenomenon is likely broader than the detectors tested here; any fixed scoring rule reads a particular structure of a frozen representation, and a domain shift can reorganize which structure separates novelty, so the same failure could appear for other score families.","Editorial inference: the calibration weight $2A_m(\\mathcal{C})-1$ is a monotone transform of per-domain AUROC, so the same logic could be used to choose among feature extractors or captioning models, not only among post-hoc detectors.","Editorial inference: the paper's random-selection control shows the Selector's ordering contributes cost savings rather than accuracy, suggesting the agent layer could be replaced by a cheaper deterministic scheduling rule whenever latency matters, leaving the Analyst's weights as the core mechanism."],"forward_implications":["A practitioner entering a new domain with a frozen encoder can measure which detectors to trust with roughly 50 to 150 labeled examples and can rule out detectors that invert there.","Benchmark comparison should not be used to lock in a single detector for deployment; the paper's evidence implies reliability must be re-established per domain.","Under a budget of three detector calls per image, SABRE reaches about 97% of full-pool quality while spending 2.6 calls per image on average.","SABRE tracks the best fixed detector rather than surpassing it, so gains concentrate where a fixed detector would fail; on the hardest tissue source, where no detector separates the classes, selection cannot create signal.","Using a domain-matched encoder and per-regime selection are complementary: a specialized encoder raises every detector's floor but does not remove the need to choose among them."],"supporting_citations":[{"why":"Supplies the MSP confidence baseline, one of the detectors the paper shows inverts on histopathology.","marker":"Hendrycks and Gimpel 2017"},{"why":"Supplies the Energy baseline, which leads on natural images and drops to 43.6 on NCT-CRC.","marker":"Liu et al. 2020"},{"why":"Supplies the MCM concept-matching baseline, which inverts from 97.7 on iNaturalist to 31.2 on NCT-CRC.","marker":"Ming et al. 2022"},{"why":"Supplies the Mahalanobis density baseline, the family that remains reliable on tissue.","marker":"Lee et al. 2018"},{"why":"Defines the CLIP encoder family used frozen throughout the experiments.","marker":"Radford et al. 2021"},{"why":"Provides the NCT-CRC histopathology dataset whose domain exposes the inversion.","marker":"Kather, Halama, and Marx 2018"},{"why":"Provides the CRC-VAL covariate-shift companion that tests generalization across scanners and sites.","marker":"Kather et al. 2019"}],"fun_headline_variants":["Fixed OOD detectors invert across domains; SABRE adapts per domain","Detector rankings flip per domain; SABRE picks at inference","SABRE recovers reliable OOD detection by calibrating at deployment","Per-domain selection beats any single OOD detector","Reliability must be measured at deployment, not assumed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes a small labeled calibration set from the deployment domain is available at deployment; without those labels the reliability weights cannot be computed and SABRE degrades to an unweighted selector that is near chance where detectors invert.","fun_headline_variants_meta":{"raw":{"variants":["Fixed OOD detectors invert across domains; SABRE adapts per domain","Detector rankings flip per domain; SABRE picks at inference","SABRE recovers reliable OOD detection by calibrating at deployment","Per-domain selection beats any single OOD detector","Reliability must be measured at deployment, not assumed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001182,"raw_usage":{"total_tokens":4934,"prompt_tokens":1051,"completion_tokens":3883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":3796}},"tokens_in":667,"tokens_out":3883,"duration_ms":22404,"temperature":1.0,"reasoning_tokens":3796,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:53:53.489889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a domain pair is found on a frozen encoder where no post-hoc detector inverts and the benchmark-preferred detector remains best, the central claim would collapse; concretely, compute AUROC of MSP, Energy, and MCM on natural images versus any specialized image domain, and if none of them falls below 50 while scoring above 50 on natural images, the inversion premise fails.","supporting_citations":[],"review_version":2}