{"id":"27e3d6ad-dda9-4969-bb47-b2c3fa02ef38","arxiv_id":"2608.09101","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A reference-free metric that compares keep and erase views of competing masks with a frozen vision-language judge can audit annotation quality and reveal class-dependent ground-truth distortion in remote sensing segmentation.","lead":"This paper introduces a reference-free metric, Contrastive Mask Fidelity (CMF), that decides which of two competing segmentation masks better matches the image by comparing keep and erase views through a frozen vision-language judge. Applied to over ten thousand image-class pairs across ten remote sensing benchmarks, it finds that model masks often beat human annotations for man-made classes, and using its verdicts as supervision improves cross-domain transfer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CMF's keep/erase contrast systematically rewards mask enlargement; Table 2's low dilation detection (0-61%) means the audit's class-structured win rates may reflect area/completeness bias rather than true annotation fidelity.","rationale":"The reader identified the shared CLIP paradigm as the weakest assumption; my concern is different and, to my mind, more load-bearing because it is not speculative. The paper's own data in Table 2 show that CMF detects dilation poorly or not at all, while its audit conclusions depend on detecting exactly the boundary over-inclusion that is common in remote-sensing annotations. This is an internally demonstrated limitation of the core metric, not an external assumption about pretraining distributions. If CMF systematically favors larger masks, then the observed class-dependent win rates (buildings 75-78%, roads 76%, cars 85%) could be explained by Seg-Probe's tendency to produce more expansive masks for structurally well-defined objects, independent of whether those masks are more faithful. The paper's area-sensitivity analysis reports a low correlation rather than a directional analysis, so it does not address this confound. The expert study is the strongest counter-evidence: CMF agrees with human consensus on 81% of pairs, including 89.3% on man-made classes. However, with only 279 decidable pairs and no public protocol or mask-area controls, the expert study cannot fully exclude a shared completeness preference. The concern is serious enough to require a targeted analysis, but it does not overturn the paper's overall evidence chain, which includes controlled corruptions, a large audit, and matched transfer experiments. The reader's CONDITIONAL verdict already reflects significant residual risk, and adding this concrete check reinforces that condition rather than changing it.","tokens_in":14658,"tokens_out":10416,"duration_ms":133587,"concrete_test":"On a stratified subset of audited pairs, split by whether the candidate mask is a superset of the human mask (e.g., area ratio ≥ 1.2 and mask IoU < 0.8) versus a subset or comparable-area condition. Fit a logistic regression predicting the CMF verdict (ΔF > 0) from class, relative mask area, and their interaction. If the man-made class coefficient shrinks toward zero after controlling for relative area, the audit's class structure is driven by CMF's dilation insensitivity. As a synthetic check, generate images with known ground-truth masks and present two equal-area alternatives, one correctly bounded and one dilated/eroded; if CMF does not rank the correct mask systematically higher, it is not a general fidelity measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CMF measures semantic mask fidelity, but Equations (3)-(5) reveal a directional bias. For a fixed true class region T, enlarging M_c to T∪E has two simultaneous effects: the keep view gains true-class pixels, raising P(c|x_k), and the erase view loses true-class pixels, lowering P(c|x_e). Both changes increase F(M_c). The paper claims in §3.3 that the two terms respond to enlargement in opposite directions, but algebraically they move F in the same direction. The empirical validity controls in Table 2 confirm this: dilation detection is 0-61% across datasets and 0% on Massachusetts Buildings, whereas erosion detection is 52-86%. Thus CMF is largely blind to over-inclusion, one of the documented annotation defects (polygonized boundaries, over-smoothed edges) that motivated the audit. Seg-Probe, built on SAM3, tends to produce expansive masks for crisp man-made classes, so the reported 62-85% prediction win rates for buildings, roads, and cars may be an artifact of mask-area inflation rather than evidence of annotation distortion. The reported area correlation ρ=0.08 is averaged over the full score range and does not rule out a monotone directional preference for larger masks. This directly threatens the interpretation of the audit's headline class structure and, ultimately, the claim that CMF is a reference-free fidelity measure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Contrastive Mask Fidelity (CMF), a training-free and reference-free metric that compares two competing class masks by scoring keep and erase counterfactual views of each mask with a frozen vision-language judge (RemoteCLIP). The fidelity score F(M_c) = P(c|x_k) - P(c|x_e) in Eq. (5) is used to decide, per class, whether the human annotation or a candidate prediction is more faithful to the image. The authors validate CMF on controlled corruptions, audit 10,731 image-class pairs across ten remote-sensing datasets, report class-structured win rates (man-made classes favor the candidate, amorphous land cover favors the human label), compare CMF against a blinded three-annotator consensus (81.0% agreement), and show that training a segmenter on CMF-arbitrated labels improves cross-domain transfer over raw annotations and matched controls.","tokens_in":14963,"tokens_out":10413,"duration_ms":117546,"significance":"The paper is well executed and unusually transparent: it validates on controlled corruptions, includes a blinded expert study, swaps judges, sweeps the arbitration margin, and uses matched controls to isolate the arbitration effect from pseudo-label distillation. If the metric's limitations can be resolved, CMF would be a genuinely useful screening tool for segmentation datasets. The main threat is that the metric appears to be primarily a completeness/omission detector: it detects erosion and shift much better than dilation (Table 2), and its mathematical formulation rewards enlarging an under-inclusive mask through both terms of Eq. (5). This threatens the interpretation of the audit's class structure. The shared CLIP-style representation between judge and candidate is a second, acknowledged confound. Both are addressable with additional targeted analyses rather than being fundamental impossibilities.","major_comments":[{"comment":"The assertion in §3.3 that the two terms of Eq. (5) 'respond to mask enlargement in opposite directions' is not correct for the regime relevant to the audit. If a mask is under-inclusive with respect to the true class extent T (the direction of the documented defects: incomplete tracing, polygonized simplification), enlarging it to include more of T adds true-class pixels to the keep view, increasing P(c|x_k), and removes those same pixels from the erase view, decreasing P(c|x_e). Both effects increase F(M). Conversely, when a mask is over-inclusive, the metric has limited ability to penalize the enlargement: Table 2 reports dilation detection rates of only 0-61% (0% on Massachusetts Buildings) versus 52-86% for erosion. CMF therefore does not implement the symmetric purity/completeness test it claims; it is primarily a completeness and displacement detector. Because Seg-Probe's masks for man-made classes are described as expansive, the class-structured win rates in Table 2 (e.g., buildings 75-78%, roads 76%, cars 85%) may reflect mask-enlargement bias rather than annotation distortion. Please add a formal analysis of F for under-inclusion versus over-inclusion, per-class dilation detection rates, and an area-matched or area-stratified version of the audit.","section":"§3.3, Eq. (5), Table 2"},{"comment":"The area-sensitivity evidence is insufficient to rule out a directional area bias. The reported correlation ρ=0.08 is a single Pearson coefficient pooled over the full score range and does not test the specific failure mode suggested by the dilation results, namely a monotone preference for larger masks at fixed semantic content. I request a stratified analysis: report the prediction win rate by deciles of the candidate/GT area ratio, and repeat the audit on the subset of pairs where the candidate and GT masks are area-matched (e.g., by clipping the larger mask to the smaller mask's area). If the man-made class wins persist under area matching, the class-structure claim is secure; otherwise it must be reinterpreted as a completeness preference rather than a fidelity judgment.","section":"§4.2, Table 2"},{"comment":"The confound between the judge and the candidate generator is acknowledged but not resolved. All four judges tested in Table 3(d) share the CLIP pretraining paradigm, and Seg-Probe is built on SegEarth-OV3, which itself uses CLIP-style text-visual alignment; §4.7 states that 'all four judges we test share the CLIP pretraining paradigm.' The expert study is an independent anchor, but it covers only 279 decidable pairs and its aggregate 81% agreement does not rule out a shared CLIP bias on the man-made classes where the audit reports the largest effects. Please test at least one non-CLIP judge (e.g., a supervised segmentation model or a pixel-level classifier trained on a disjoint dataset) or generate candidate masks with a non-CLIP method and verify that the class-structured win rates persist. If this is not feasible, the audit's conclusion should be reframed as 'distortion as perceived by CLIP-family judges' rather than as a property of the annotations.","section":"§4.6/§4.7, Table 3(d)"}],"minor_comments":[{"comment":"There is a typo in §4.6: 'does nothingeonthe specific judge' should read 'does not hinge on the specific judge.'","section":"§4.6"},{"comment":"The expert study excludes 21 split-vote pairs and then reports agreement on 279 pairs; please report the agreement when split votes are counted as disagreements or resolved by a pre-registered rule, and give the per-dataset and per-class breakdown of the 300 pairs, since the class imbalance of the expert sample may affect the aggregate 81%.","section":"§4.4"},{"comment":"The validity controls in Table 2 report only detection rates; please also report false-positive rates on clean pairs and bootstrap confidence intervals, especially for Massachusetts Buildings where n=130 and the dilation detection rate is 0%.","section":"§4.2, Table 2"},{"comment":"The 'All' row pools pairs across datasets with different class distributions and class vocabularies; report dataset-macro averages for the validity controls as well as the pooled values.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid paper with honest limitations. The main risk is the metric's enlargement bias; the authors should be asked for additional analyses rather than rejection. The shared-CLIP confound is real but partly addressed by the expert validation; a non-CLIP judge experiment would strengthen it. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. The core idea is new and useful: instead of trusting one mask as reference, score two competing masks by asking a frozen VLM whether class evidence sits inside or outside each mask. The keep/erase construction is clever, and the authors back it with a serious evidence chain: controlled corruptions, a 10,731-pair audit over ten datasets, a blinded expert study, and matched transfer experiments that separate arbitration from pseudo-label distillation. The matched controls in §4.5 are particularly good—random replacement does nothing, confidence replacement does less than CMF, so the effect is attributable to the arbitration signal, not just to injecting strong pseudo-labels.\n\nThe stress-test flag is real, though I'd put a softer spin on it. In §3.3 the paper claims the two terms of Eq. (5) respond to mask enlargement in opposite directions. That is wrong: if the mask is inside the true region, enlarging it raises the keep-view probability and lowers the erase-view probability, so both changes push F up. The empirical controls agree—dilation detection is 0–61% across datasets, with 0% on Massachusetts Buildings—so CMF is partially blind to over-inclusion, one of the annotation defects that motivated the audit. The paper acknowledges this in passing but doesn't link it to the class-structured win rates. That's a genuine weakness, and it needs a direct fix, not just a sentence in the limitations.\n\nBut I don't think the weakness is load-bearing. The expert validation is a real check: CMF matches the expert majority on 81% of pairs overall and 89% on man-made classes. If the verdicts were just a mask-area artifact, independent human judges would not agree with them that often. The expert study is small (279 decidable pairs), and the shared CLIP paradigm between judge and candidate generator is a legitimate worry, but the judge-swap stability across four CLIP-family models and the expert agreement provide enough support to take the empirical claim seriously. The authors are also honest about these limits in §4.7.\n\nMain fixes I'd want before fully trusting it: correct the erroneous sentence in §3.3, report win rates split by whether the preferred mask is larger or smaller to quantify area bias, and release code and data. The supplementary material is described but not available to me.\n\nThis paper deserves a serious referee. The idea is novel, the execution is mostly careful, and the central claim is testable. I'd accept it conditionally, with the area-sensitivity analysis as a required revision.","headline":"Fresh, well-executed reference-free mask audit with a real area-sensitivity flaw and a shared-CLIMB confound, but the evidence chain is strong enough to warrant a serious referee.","tokens_in":15470,"tokens_out":5204,"would_cite":true,"duration_ms":62031,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A contrastive keep–erase score lets a frozen vision-language judge decide which of two competing segmentation masks better matches the image, without designating either as ground truth.","keywords":["semantic segmentation","remote sensing","mask fidelity","reference-free evaluation","vision-language model","annotation audit","ground-truth quality","open-vocabulary segmentation"],"falsifier":"Take a set of images with independently verified high-resolution ground truth, produce deliberately distorted masks, and check whether CMF ranks the correct mask first on a large majority of pairs; also replace the CLIP-style judge with a judge trained under a different alignment objective and see whether the per-class win-rate profile and the 81% expert agreement survive. If a non-CLIP judge reverses the verdicts, or if CMF prefers geometrically loose but semantically plausible masks over boundary-correct ones in the verified set, the reference-free fidelity claim would fail.","tokens_in":14474,"feed_emoji":"🛰️","tokens_out":7275,"duration_ms":72099,"temperature":0.7,"pith_summary":"Semantic segmentation benchmarks are scored against human-drawn masks, but in remote sensing those masks are often coarse or misaligned, so a high overlap score can mean agreement with a flawed label rather than fidelity to the image. This paper tries to break that evaluation paradox with Contrastive Mask Fidelity (CMF): a training-free, reference-free score that asks a frozen vision-language judge whether class evidence is concentrated inside a mask and absent outside it. The paper claims CMF can decide which of two competing masks is more faithful without trusting either, and supports that claim by matching a blinded three-annotator expert consensus on 81% of adjudicated pairs and by exposing systematic annotation distortion, with man-made classes like buildings, roads, and cars favoring the model mask on most pairs while ambiguous land cover favors human annotations. If the claim holds, overlap-based evaluation can be complemented by an image-grounded audit that reports annotation-noise headroom per class and supplies safer supervision for cross-domain training.","feed_headline":"Image-based mask audit matches experts 81% of the time","feed_subtitle":"CMF's keep/erase contrast exposes class-structured annotation bias and improves cross-domain transfer.","key_machinery":"The load-bearing mechanism is the keep–erase contrast: for each binary class mask $M_c$, the paper composites a keep view $x_k^c=M_c\\odot x+(1-M_c)\\odot B(x)$ and an erase view $x_e^c=(1-M_c)\\odot x+M_c\\odot B(x)$, with $B(\\cdot)$ a Gaussian blur that preserves local color statistics, then scores both views with a frozen remote-sensing vision-language judge through a temperature-scaled softmax over the dataset's class texts. The fidelity score $F(M_c)$ is the difference of the judge's class probability on the two views, so purity and completeness are tested in one number, and the contrast is designed to reduce area sensitivity, with the paper reporting the area correlation dropping from $\\rho=0.61$ to $\\rho=0.08$. The same frozen judge scores the human mask and the candidate mask symmetrically, which is what makes the arbitration reference-free.","core_discovery":"CMF is defined per class as $F(M_c)=\\mathbb{P}(c\\mid x_k)-\\mathbb{P}(c\\mid x_e)$, where $x_k$ keeps only the region inside mask $M_c$ with the rest blurred and $x_e$ erases it by blurring the mask area, and $\\mathbb{P}$ is a frozen remote-sensing vision-language judge's class probability over the dataset's class texts. A faithful mask must score high on both purity, meaning the kept region depicts the class, and completeness, meaning no class evidence remains outside, and the contrast between the two views cancels trivial area inflation because enlarging a mask pushes the two terms in opposite directions. The paper's empirical claim is that this score arbitrates real disagreements: on 10,731 image-class pairs across ten remote-sensing benchmarks, CMF's verdicts match expert consensus more often than keep-only scoring, model confidence, or a trained label-quality baseline, and conservative class-wise arbitration of its verdicts improves cross-domain transfer over raw annotations, pure predictions, and matched random- or confidence-based replacements.","pith_inferences":["If the keep–erase contrast really measures fidelity, the same construction should transfer beyond closed-set segmentation to instance- or panoptic-level claims and to open-vocabulary classes that no annotation scheme ever named, since the judge's vocabulary need not match the dataset's.","The class-structured win rates imply that annotation-production conventions, such as polygonization, minimum mapping units, and registration, drive more of the observed distortion than class difficulty or object size; a testable prediction is that re-annotating the same imagery under stricter conventions would shrink the prediction-win gap for man-made classes.","The authors' caveat that all four judges tested share the CLIP-style pretraining paradigm suggests a stress test the paper does not run: if a judge trained under a different alignment objective produced a different class structure, part of the 81% expert agreement would be representation bias rather than image fidelity.","Because trained label-quality baselines inherit the very label distribution they audit, CMF could be used as a pre-filter to clean training sets before robust training, decoupling the audit from the model under test."],"forward_implications":["Benchmarks can be audited without a trusted reference: any two competing masks, including a human annotation and an open-vocabulary prediction, can be scored directly against the image.","Annotation noise becomes reportable per class, so the audit can show on which classes human labels are systematically distorted and on which they carry information the image alone does not.","CMF verdicts provide usable supervision: conservatively replacing the human label on pairs with $\\Delta F>0$ improves cross-domain mIoU by about 1.2 to 1.6 points over raw labels and beats coverage-matched confidence replacement, indicating the gain comes from selection rather than pseudo-label distillation.","Because CMF is training-free and needs only four judge passes per image-class pair, roughly 16 minutes for the full 10,731-pair audit on one GPU, the audit can scale to large datasets.","The arbitration margin acts as a precision–coverage dial: raising $\\delta$ trades replacement coverage for per-replacement expert agreement without losing most of the transfer gain, so CMF can serve as a triage layer in annotation pipelines."],"supporting_citations":[{"why":"Establishes the reference-free image-text scoring paradigm (CLIPScore) that CMF adapts to mask regions.","marker":"(Hessel et al. 2021)"},{"why":"Supplies the frozen RemoteCLIP judge whose class probabilities define the keep-erase fidelity score.","marker":"(Liu et al. 2024b)"},{"why":"Provides the SAM3 encoder that Seg-Probe uses as its frozen backbone for generating candidate masks.","marker":"(Carion et al. 2026)"},{"why":"SegEarth-OV3 is the base open-vocabulary segmenter that Seg-Probe extends with synonym pooling, multi-scale evidence, and flip test-time augmentation.","marker":"(Li et al. 2025b)"},{"why":"Defines the trained label-quality baseline that CMF must beat and exemplifies the label-distribution bias that reference-free scoring avoids.","marker":"(Lad and Mueller 2023)"},{"why":"Documents pervasive label errors in test sets, motivating the audit of ground-truth masks.","marker":"(Northcutt, Athalye, and Mueller 2021)"},{"why":"CLIP is the shared pretraining paradigm of all judges tested, grounding the stability analysis and the principal limitation.","marker":"(Radford et al. 2021)"}],"fun_headline_variants":["Keep-or-erase test audits masks, matches experts 81%","Reference-free mask audit exposes annotation bias","Image-based mask audit: 81% expert agreement","CMF keep/erase contrast arbitrates mask disputes","Mask quality metric beats baselines, matches experts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the frozen vision-language judge's preference for one mask over another reflects genuine visual evidence in the image rather than shared pretraining biases, since the judge and the candidate generator both come from the CLIP-style model family and could therefore agree for reasons unrelated to true fidelity.","fun_headline_variants_meta":{"raw":{"variants":["Keep-or-erase test audits masks, matches experts 81%","Reference-free mask audit exposes annotation bias","Image-based mask audit: 81% expert agreement","CMF keep/erase contrast arbitrates mask disputes","Mask quality metric beats baselines, matches experts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3428,"prompt_tokens":1031,"completion_tokens":2397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":2322}},"tokens_in":647,"tokens_out":2397,"duration_ms":21458,"temperature":1.0,"reasoning_tokens":2322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:30:25.045017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of images with independently verified high-resolution ground truth, produce deliberately distorted masks, and check whether CMF ranks the correct mask first on a large majority of pairs; also replace the CLIP-style judge with a judge trained under a different alignment objective and see whether the per-class win-rate profile and the 81% expert agreement survive. If a non-CLIP judge reverses the verdicts, or if CMF prefers geometrically loose but semantically plausible masks over boundary-correct ones in the verified set, the reference-free fidelity claim would fail.","supporting_citations":[{"cited_title":"2021 , doi =","cited_arxiv_id":null,"evidence_quote":"Establishes the reference-free image-text scoring paradigm (CLIPScore) that CMF adapts to mask regions."}],"review_version":1}