{"id":"2da70ee9-476f-4856-8e55-8a9da2abbea5","arxiv_id":"2607.18451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A chance-calibrated discordance measure shows frozen encoders encode fine-grained clinical findings weakly rather than blindly, with collapse rates ranging from 4.5% (bird species) to 50.0% (glaucoma).","lead":"CANDOR is a corrected nearest-neighbor measurement of how often a frozen AI image encoder places a positive case nearer an opposite-label example than its own kind, with the chance level fixed at exactly 50% so results are comparable across tasks. Across 605,443 images and 22 encoders it finds frozen encoders are weak, not blind, and the weakness is mappable by finding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Residual confounders within matched contexts can drive CANDOR's below-chance rates; Ablation 4 is inconclusive, so the measurement may not isolate the finding.","rationale":"The reader's CONDITIONAL verdict is appropriate. Our stress-test converges on the same load-bearing assumption: the sufficiency of the acquisition-context match. The mathematical concern about Lemma 1 is real (the proof as written swaps label-conditioned blocks, which violates the conditioning), but the intended H0 statement is recoverable and does not undermine the empirical below-chance pattern. The selection-oracle overclaim is secondary. The context-match assumption, however, is the one that determines whether every reported rate is a property of the finding at all. The paper's own Ablation 4 is inconclusive, and the data cannot check residual confounders. The proposed inpainting test would directly assess this. If it shows the rates are finding-driven, the paper's central claim stands; if not, the map and 'weak not blind' conclusion would need substantial reinterpretation. Until then, CONDITIONAL is the right verdict.","tokens_in":15595,"tokens_out":9905,"duration_ms":145607,"concrete_test":"Use the MS-CXR expert boxes to construct perfectly matched pairs: for each box-verified positive, create a counterfactual negative by inpainting the annotated finding region (e.g., with a diffusion model or mean-fill), leaving the rest of the image unchanged. Run the CANDOR pipeline with the negative bank composed of these synthetic negatives and the positive bank of real positives from the same context, with k=5 and equal bank sizes. If D on these pairs remains materially below 50, the below-chance rates reflect the finding itself; if D moves to near 50, the original rates are inflated by within-context confounders. This tests the context-match assumption directly because the only difference between the two label groups is the finding region.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central interpretation—that a CANDOR rate measures whether a frozen encoder separates a specific finding—depends on the context match removing all non-finding differences between the positive and negative banks. Within a single site and projection, films that differ in the finding can still differ systematically in illness severity, patient habitus, image processing, or co-occurring devices (e.g., chest tubes in pneumothorax). If any such covariate is correlated with the label, DM>0 can be driven by that covariate rather than the finding, and the below-chance collapse rate would not mean 'encodes the finding weakly.' The manuscript itself flags this as 'the one assumption the data cannot check,' and its Ablation 4 (pooling banks across sites and views) is reported as inconclusive—the matched rate is the lower one in only 9 of 22 cells. The erasure-retention correlation is suggestive but correlational and does not establish that the separation is finding-specific. Because every headline number (132 cells below chance, the domain map, the 'weak not blind' conclusion) inherits this assumption, it is the most load-bearing point in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CANDOR, a nearest-neighbor discordance operator D(g;F,s)=Pr[DM_g(x)>0 | F(x)=1] built from equal-size positive/negative reference banks matched on acquisition context (site-by-view for chest, source dataset elsewhere). It claims an exact chance level of 1/2 under the null, proves a Lipschitz margin bound on any head, and reports a large empirical study (22 encoders, 20 datasets, 605,443 images) in which collapse rates fall below 50% almost everywhere, leading to the conclusion that frozen encoders encode fine-grained findings weakly but are not blind. The paper also analyzes blind-set overlap across encoders, correlates erasure retention with collapse, and shows that an oracle per-image selector among 11 heads achieves 2.8% error where a single head misses 35.9%.","tokens_in":15798,"tokens_out":14580,"duration_ms":127615,"significance":"If correct, CANDOR provides a head-free, chance-calibrated measurement for evaluating frozen encoders, potentially before training any readout. The study's strengths include a careful empirical design: paired hashed rows shared across encoders, bootstrap CIs, FDR correction, synthetic planted-encoder calibration, and a released code repository. The Lipschitz margin bound is a useful theoretical contribution, as is the demonstration that several standard uncertainty scores collapse into one detector. However, the central chance-level theorem as stated is imprecise, and the interpretation of the rates as finding-specific rests on an explicitly unverifiable no-confounder assumption.","major_comments":[{"comment":"Lemma 1 states that for any encoder g, Pr[DM_g(x)>0]=1/2 when B+ and B- are drawn as two equal-sized blocks uniformly from the context. This is not the sampling distribution of the operator in Eq. (2), where B+ and B- are label-conditioned banks. Exchangeability of the two blocks holds only under H0 (g(X)⊥F|s) or under a permutation null, not for an arbitrary informative encoder. As written, the lemma is contradicted by the planted 'sees it' encoder scoring 0.0 (Ablation 1, Fig. 2a) and by every below-50 rate in Table 2. Please restate the claim as 'under H0, the expected discordance is 1/2' and make the probability space explicit. This is the central chance-calibration result and needs correction.","section":"CANDOR Framework, Lemma 1 ('The chance level is exactly one half')"},{"comment":"The interpretation of a CANDOR rate as measuring whether an encoder separates a specific finding assumes that matching within an acquisition context removes all non-finding confounders. The paper itself calls this 'the one assumption the data cannot check.' Ablation 4 reports that pooling banks across sites/views is inconclusive: the matched rate is the lower one in only 9 of 22 cells. Unrecorded covariates (severity, patient habitus, image processing, co-occurring devices) could therefore drive the discordance and produce the 'weak encoding' conclusion even if the encoder is not encoding the finding. The headline claims in the abstract and conclusion should be qualified as measuring label separation within the recorded context, or supplemented with analyses that directly address specific plausible confounders.","section":"CANDOR Framework, Problem formulation; Ablation 4"}],"minor_comments":[{"comment":"The manuscript contains many missing spaces and odd capitalizations (e.g., 'Chance-cAlibrated NeighborhoodDiscORdance'), likely from a rendering artifact. Please ensure a cleanly formatted manuscript, especially in the abstract and throughout the text.","section":"Throughout"},{"comment":"The column labeled 'Best' is the per-image oracle over the 11 heads. Please clarify this in the caption, since the reader might otherwise interpret it as a single trained head.","section":"Table 4"},{"comment":"The planted-blind and planted-sees encoders are described with inconsistent hyphenation ('planted blind' vs 'planted-blind'). Please standardize.","section":"Ablation 1 / Figure 2a"}],"recommendation":"major_revision","confidential_remarks":"The Lemma 1 ambiguity is likely to attract statistical criticism; the authors must make the null-hypothesis conditioning explicit. The unmeasured-confounder issue is the more scientifically serious risk and should be addressed by either new analyses or careful caveats in the abstract and conclusion. The oracle selection result is suggestive but should not be oversold as evidence that a practical selection rule exists."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper is worth reading, and its central measurement — that frozen encoders sit below chance on discordant twins almost everywhere — holds up, but the authors overstate their own math and one key assumption is unchecked.\n\nWhat's actually new: the equal-size reference bank trick fixes the chance level at exactly 1/2, and the paper shows convincingly that the uncorrected k-NN discordance measure manufactures collapse in proportion to class rarity. The empirical scope is real: 22 encoders, 7 domains, 605,443 images, with paired hashed rows, bootstrap CIs, FDR correction, and explicit disclosure of small samples. The planted-blind calibration landing at 49.5 with a CI covering 50 is a nice verification. The paper also ships code and data, and the Lipschitz margin bound is correctly borrowed from certified robustness — that part is fine.\n\nThe soft spots are real but manageable. Lemma 1 as stated claims Pr[DM>0] = 1/2 for any encoder, but that's only true under the null that the representation is independent of the finding within context. The paper's own planted sees-it encoder scores 0.0, which is a direct contradiction. The proof works as a symmetry argument under H0, but the statement needs to be fixed. This is a mathematical overstatement, not a fatal flaw, because the later text does introduce H0 — but it should be front and center.\n\nThe bigger concern is the context-match assumption. The paper says matching on site and view forces the encoder onto the finding, and then admits this is 'the one assumption the data cannot check.' Ablation 4, which pools across sites and views, is inconclusive. So the interpretation of every collapse rate as a property of the finding inherits this risk. Residual confounders correlated with the label could push rates in either direction. That doesn't kill the 'weak not blind' conclusion, but it weakens the finding-specific map.\n\nThe 'deficit is selection, not information' claim also rests on a label-revealing oracle over only 263 positives, and the abstract drops that qualification. The paper's own label-free selector fails to beat confidence, which suggests the headline claim is about a theoretical upper bound, not a practical result.\n\nStill, the core idea is sound, the empirical apparatus is careful, and the authors are honest about many limitations. This deserves a serious referee. I'd send it to peer review with a request for major revision: fix Lemma 1, soften the selection claim, and add a sensitivity analysis that actually tests confounder robustness beyond Ablation 4.","headline":"CANDOR's equal-bank chance calibration is a genuinely useful fix and the empirical map is impressive, but Lemma 1 is overstated and the context-match assumption is load-bearing, so the paper needs revision before the strong claims can stand.","tokens_in":16422,"tokens_out":2741,"would_cite":true,"duration_ms":27236,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen encoders are not blind: a chance-calibrated discordance measure shows they encode clinical findings weakly.","keywords":["frozen encoders","discordance metric","chance level calibration","nearest-neighbor geometry","foundation models","medical image analysis","prevalence bias","representation evaluation"],"falsifier":"Build a single-site, single-projection context with no real finding difference but with a systematic difference in an unrecorded covariate—for example, label one subset 'positive' and another 'negative' purely by scanner protocol or image brightness. If CANDOR on this pseudo-task returns a collapse rate clearly below 50%, the within-context matching has not removed the confounder and the operator misattributes it to a finding; if it stays at 50%, the matching assumption holds.","tokens_in":15352,"feed_emoji":"🩻","tokens_out":9708,"duration_ms":74571,"temperature":0.7,"pith_summary":"Frozen encoders are usually judged by how well a small trained head reads a finding from their features, but a high AUROC can hide the fact that many positive images sit closer to opposite-label images in the encoder's geometry. CANDOR measures this discordance directly and fixes its chance level at exactly one half by using equal-size positive and negative reference banks, a symmetry correction that removes the prevalence confound. Across 22 encoders and 605,443 images, the paper finds that discordance falls below chance in every cell, so no encoder is blind to fine-grained clinical findings, but rates stay high for chest findings and sit at chance for glaucoma and mammographic malignancy, so the encoders are uniformly weak. A Lipschitz margin bound shows that a discordant twin caps the normalized margin any head can achieve on that encoder, yet a per-image selection rule across eleven encoders recovers almost all of the missed cases. The paper's central conclusion is that the deficit is selection, not information: the findings are in the geometry, but no single frozen encoder separates them strongly.","feed_headline":"Chance-calibrated discordance flips the verdict on frozen encoders","feed_subtitle":"No encoder is blind, yet the best chest model still places one positive in five beside its opposite.","key_machinery":"The central mechanism is the operator D(g;F,s) = Pr[DM_g(x)>0 | F(x)=1], with DM_g(x) comparing the mean cosine similarity of a positive query to its k nearest neighbors in an equal-size negative bank against the mean to its k nearest neighbors in an equal-size positive bank, both banks drawn from the same acquisition context. The design choice that carries the argument is equal-size banks: they make the two label groups exchangeable under relabeling, so the chance level is fixed at exactly one half by a one-line symmetry argument (Lemma 1) with no empirical null. Behind the operator stands Proposition 1, which converts any discordant twin into a geometric cap on the normalized margin of any","core_discovery":"The paper's central claim is that the discordance operator D(g;F,s), defined as the share of positive images whose nearest opposite-label neighbors are closer on average than their own kind within a matched acquisition context, has an exact chance level of one half when the two reference banks are equal in size, and that this chance-calibrated measurement reverses the standard reading of frozen encoders. Over 132 encoder-finding cells, every bootstrap interval for the collapse rate sits above 50%, proving no encoder is blind; yet the best chest model, RAD-DINO, still places 18.4% of pneumothorax positives nearer an opposite-label film, and the same encoder that collapses only 4.5% of bird-sp","pith_inferences":["The equal-size bank symmetry is a general correction for any nearest-neighbor-based similarity measure with imbalanced groups; published 'blindness' or 'shortcut' results using imbalanced nearest-neighbor statistics may need to be re-read as prevalence artifacts.","The per-image selection result suggests a practical routing system that picks an encoder per image, but the paper shows a label-free proxy for 'which encoder to trust' remains open; an unsupervised version of that selector would be a direct extension.","If erasure retention is causally related to collapse, a training objective that penalizes retention when evidence is removed might directly reduce discordance; this is a testable hypothesis the paper does not pursue.","The finding that general-purpose encoders collapse less on natural fine-grained tasks (birds, aircraft) than on chest findings suggests that the weakness is task-specific, possibly tied to the subtlety and low-level nature of medical findings; this could guide the design of medical-specific encoders."],"forward_implications":["If CANDOR is correct, model selection for medical image encoders should be done by measuring geometry directly, not only by head AUROC, since a strong AUROC can coexist with a fifth of positives being placed on the wrong side of the geometry.","Because the chance level is fixed, CANDOR can be computed before any head is trained, giving a head-free way to flag which findings a frozen encoder supports poorly.","The Lipschitz bound implies that on cases flagged as discordant, increasing head width, depth, or norm cannot increase the normalized margin; the only route to improvement is a different encoder geometry.","The near-total recovery of missed cases by per-image selection across the panel implies that ensembling or selection strategies can tap information that no single encoder holds, provided one can solve the label-free selection problem.","The association between erasure retention and collapse suggests that encoders that barely move their representation when the evidence is occluded are the ones that place positives beside their opposite, offering a cheap diagnostic for collapse."],"fun_headline_variants":["Chance-fixed discordance flips the blind-encoder verdict","No encoder is blind, yet all are weak: CANDOR's fix","CANDOR: prevalence can't fake blindness at chance 0.5","Frozen encoders: none blind, but best chest model loses 18.4%","Discordance at chance 1/2 overturns frozen-encoder collapse"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Every discordance rate is interpreted as a property of the finding only if matching images by recorded acquisition covariates (site, projection, and so on) removes all non-finding confounders, and the paper's own ablation pooling across sites and views is inconclusive, so a systematic difference between matched images that correlates with the label—such as an unrecorded scanner protocol or patient habitus—would be misattributed to the finding.","fun_headline_variants_meta":{"raw":{"variants":["Chance-fixed discordance flips the blind-encoder verdict","No encoder is blind, yet all are weak: CANDOR's fix","CANDOR: prevalence can't fake blindness at chance 0.5","Frozen encoders: none blind, but best chest model loses 18.4%","Discordance at chance 1/2 overturns frozen-encoder collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3417,"prompt_tokens":824,"completion_tokens":2593,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2491}},"tokens_in":568,"tokens_out":2593,"duration_ms":22283,"temperature":1.0,"reasoning_tokens":2491,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:22:39.145355+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a single-site, single-projection context with no real finding difference but with a systematic difference in an unrecorded covariate—for example, label one subset 'positive' and another 'negative' purely by scanner protocol or image brightness. If CANDOR on this pseudo-task returns a collapse rate clearly below 50%, the within-context matching has not removed the confounder and the operator misattributes it to a finding; if it stays at 50%, the matching assumption holds.","supporting_citations":[],"review_version":1}