{"id":"62095c98-b5d4-4349-b8ff-2a80e47df3d0","arxiv_id":"2508.04050","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DOMR jointly matches and refines multiple object masks across ego and exo views, reaching 49.7% and 55.2% mean IoU on Ego-Exo4D.","lead":"A new two-stage system, DOMR, matches objects between first-person and third-person videos by comparing many objects at once instead of searching for one. It reports the best scores on the Ego-Exo4D benchmark, beating prior methods by roughly 4 to 6 points of mask overlap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposal recall is unmeasured; if any GT object receives no proposal, DOMR cannot recover it, capping the reported SOTA IoU.","rationale":"The reader's weakest_assumption identifies proposal recall as the key risk, and I agree. The central claim—SOTA IoU on Ego-Exo4D—depends on every ground-truth target object having a usable proposal. Table 1 reports average best IoU, which is an aggregate that can be high even when a nontrivial fraction of objects have zero proposal overlap. Since the training loss masks out objects with IoU ≤ 0.5, the model may not learn robust behavior for missing proposals. At inference, the default matching strategy will still select some proposal, likely yielding near-zero IoU for missed objects. This is not a refutation, but it is a missing measurement that directly affects the interpretation of the headline numbers. The paper's internal ablations (SOS vs. DOM, mix vs. top-1) are consistent and support the value of dense matching, but they do not resolve the coverage question. I also note the ambiguous inference-stage sentence in §3.7; if read as using target GT boxes, it would be a protocol violation, but the more natural reading is that it refers to source GT masks, which are available in the benchmark. Because this concern is addressable by reporting recall statistics, the reader's CONDITIONAL verdict is appropriate; no change is needed.","tokens_in":14811,"tokens_out":7493,"duration_ms":85417,"concrete_test":"On the Ego-Exo4D validation split, compute recall@0.5 and recall@0.1 for YOLO-UniOW+SAM2 proposals in ego and exo views (fraction of GT objects with at least one proposal exceeding the IoU threshold), and recompute the upper-bound mIoU assuming perfect matching of covered objects. If recall@0.5 is above ~95%, the missing-object concern is minor; if it is materially lower, the SOTA improvement over ObjectRelator may be largely due to proposal coverage rather than the dense matching architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 (Table 1) reports only the average IoU of the best available proposal for each ground-truth object, not recall: the fraction of objects receiving no proposal above any threshold. If an object is missed by YOLO-UniOW/SAM2, DOM has nothing to match, and the reported mIoU is bounded by the fraction of objects with usable proposals. The training loss (Eq. 15) explicitly filters out objects with IoU ≤ 0.5 in either view (v(i)=0), so the model is never trained on missing-proposal cases. The paper's claim that 'the proposal generator successfully localizes most objects' is not supported: an average best mask IoU of 67.1% (exo) can coexist with a non-trivial zero-coverage subset. Additionally, the §3.7 sentence 'During inference, the ground truth masks do not directly participate... their bounding box is served as box prompts for SAM 2' is ambiguous; if it refers to target GT boxes, the protocol would leak target annotations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DOMR, a two-stage framework for ego-exo cross-view object correspondence on the Ego-Exo4D benchmark. The first stage, Dense Object Matcher (DOM), generates object proposals with YOLO-UniOW and SAM 2, builds per-proposal embeddings from visual, positional, and label cues, and matches proposals across views using a bidirectional 'mix' strategy over a similarity matrix. The second stage, Mask Refinement (MR), refines the matched masks with a SAM-2-style decoder. The authors report state-of-the-art mean IoUs of 49.7% (Ego→Exo) and 55.2% (Exo→Ego), outperforming ObjectRelator by 5.8 and 4.3 points. Ablations indicate that dense object matching, the mix strategy, and mask refinement each contribute to the final result.","tokens_in":15124,"tokens_out":6358,"duration_ms":77240,"significance":"If the reported numbers are taken at face value, DOMR is a substantial empirical improvement over prior work on a challenging cross-view benchmark, and the paper provides a clear and sensible explanation for why joint multi-object matching outperforms single-object searching. The ablations are internally consistent and the design choices are well motivated. The work does not rely on circular reasoning: the training signal comes from ground-truth correspondences and the evaluation is on an external validation split. However, the manuscript currently leaves two load-bearing questions open: (i) whether the inference protocol inadvertently uses ground-truth bounding boxes in the target view, and (ii) whether the proposal generator's recall is high enough to support the headline numbers. The absence of error bars, code, and checkpoints further limits confidence in the claimed state-of-the-art result.","major_comments":[{"comment":"The sentence 'During inference, the ground truth masks do not directly participate in DOM ... Instead, their bounding box is served as box prompts for SAM 2' is critically ambiguous. If 'their' refers to ground-truth masks in the target view, then the evaluation uses target annotations at test time, which would invalidate every reported number. If it refers only to the source-view masks, which are legitimate inputs, then the text must state this explicitly and describe how target proposals are generated. Please clarify and, if target boxes are used, rerun the experiments without them.","section":"§3.7, Loss Functions"},{"comment":"The proposal coverage table reports only the average IoU of the best proposal per ground-truth object. This does not measure recall: the fraction of objects that receive no proposal above any threshold. Since an object absent from the proposal set can never be matched, and since the training loss in Eq. (15) discards objects with IoU ≤ 0.5, the reported mIoU may be optimistically biased if the evaluation ignores or down-weights such objects. Please report recall at several IoU thresholds (e.g., IoU > 0, > 0.5), the full IoU distribution, and the mean IoU upper bound including zero-coverage objects. Also specify how a ground-truth object with no usable target proposal is treated in the final metric.","section":"§3.3, Table 1"},{"comment":"The claim that the proposal generator 'successfully localizes most objects' is not supported by the presented statistic. An average best-mask IoU of 67.1% (exo view) can coexist with a non-trivial fraction of objects having zero proposal overlap. Because the method's upper bound is set by proposal coverage, the paper needs to quantify the fraction of ground-truth objects that are unreachable by the proposal stream, and ideally report performance separately for the reachable subset.","section":"§3.3 and §4.1"}],"minor_comments":[{"comment":"All reported numbers appear to be single-run validation scores without error bars or multiple seeds. Please report means and standard deviations over at least three runs, or justify why single-run evaluation is standard for this benchmark.","section":"§4.1"},{"comment":"No code, checkpoints, or public implementation details are provided. Given that the method builds on several external components (YOLO-UniOW, SAM 2.1), releasing code would be important for reproducibility.","section":"General"},{"comment":"The formatting of Table 8 is confusing: some entries are dashes, and it is unclear what the two non-frozen rows represent (two independent trainings? one per direction?). Please clarify the experimental setup and complete the table.","section":"Table 8"},{"comment":"The phrase 'The maximum coverage also indicates the upper bound precision of the proposed DOM' is unclear; Table 1 reports an average, not a maximum. Rephrase to avoid confusion.","section":"§3.3"},{"comment":"The sentence mentioned in the major comments should be rewritten to remove ambiguity. As written, it is difficult to tell whether the ground-truth boxes are used in the source or target view.","section":"§3.7"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is potentially very strong, but the ambiguity in §3.7 is serious. If the authors confirm that target-view ground-truth boxes are used at inference, the paper should be rejected. If they are only using source-view boxes (which are part of the task input), the paper is publishable after adding recall analysis and clarifying the protocol. The lack of code and error bars also makes it hard to assess robustness; I would encourage the editor to request these as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DOMR is a solid engineering contribution that sets a new SOTA on the Ego-Exo4D ego-exo correspondence benchmark, and unlike a lot of papers in this area, the main idea is actually novel: matching all proposals jointly rather than doing single-object retrieval. The dense matching module with positional, semantic, and visual embeddings, plus the bidirectional mix strategy, is a clean idea, and the ablations back it up—SOS -> DOM gains 2-7 points, and mix beats top-1. The mask refinement stage is a standard SAM decoder but it adds a consistent boost. I believe the headline numbers are plausible, but I'd want a few things before citing them as fact.\n\nThe biggest gap is in reporting proposal coverage. Table 1 gives the mean best IoU per ground-truth object, but not the recall—how many objects get zero usable proposals. If a target object is missed by YOLO-UniOW, DOM has nothing to match, so the final mIoU is capped by detector recall. The mean best IoU of 67% in exo view sounds okay, but it can hide a non-trivial fraction of zero-coverage objects, and since the training loss (Eq. 15) ignores proposals with IoU <=0.5, the model never sees that failure mode. That's not a fatal flaw, but it's a missing number.\n\nAlso, no code or checkpoints, and every result is a single validation run without error bars. That's common for this kind of paper, but with a 5.8-point SOTA claim, it'd be nice to see some stability.\n\nOne passage in Sec. 3.7 is worded badly: 'During inference, the ground truth masks do not directly participate... their bounding box is served as box prompts for SAM 2.' In context I think this means the source masks you're given at inference are converted to box prompts, which is fine. But as written, it could be misread as using target ground-truth boxes, which would leak. That needs a rewrite.\n\nOverall, the central argument holds: joint dense matching with context beats single-object matching. The paper is worth a serious referee. I'd ask for code, error bars, proposal recall, and a clarification of the inference protocol, then I'd be comfortable accepting it.","headline":"Solid SOTA on Ego-Exo4D correspondence with a genuinely new dense-matching idea, but the missing recall numbers and an ambiguous inference sentence need tightening before I'd fully trust the numbers.","tokens_in":15576,"tokens_out":4863,"would_cite":true,"duration_ms":51898,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dense object matcher that compares all proposals in both views at once sets new state-of-the-art scores on the Ego-Exo4D cross-view correspondence benchmark.","keywords":["cross-view object correspondence","ego-exo correspondence","dense object matching","mask refinement","Ego-Exo4D","SAM 2","open-vocabulary detection","segmentation"],"falsifier":"Compute, on the Ego-Exo4D validation set, the proportion of ground-truth objects whose best proposal from the DOMR pipeline has IoU below 0.5 (or no proposal at all). If that proportion is non-negligible and DOMR never matches those objects, the reported mean IoU of 49.7/55.2 cannot be reproduced without changing the proposal source or adding a recovery mechanism.","tokens_in":14779,"feed_emoji":"🎯","tokens_out":8574,"duration_ms":90599,"temperature":0.7,"pith_summary":"This paper tries to establish that cross-view object correspondence between egocentric and exocentric video is better solved by dense, joint matching among all candidate objects than by searching for each query mask independently. The proposed DOMR pipeline first turns detector-plus-SAM 2 proposals into object embeddings that combine visual, positional, and semantic cues, lets all embeddings from both views interact in a transformer, and enforces bidirectional symmetric consistency when selecting matches. On the Ego-Exo4D benchmark it reports mean IoUs of 49.7% for Ego→Exo and 55.2% for Exo→Ego, beating the previous best method by 5.8 and 4.3 points. A sympathetic reader would care because this is a step toward reliable first-person/third-person alignment for embodied systems, where repeated similar objects currently cause ambiguity.","feed_headline":"All-at-once matching lifts ego-exo object IoU to 49.7%","feed_subtitle":"Jointly comparing every proposal in both views plus mask refinement beats prior ego-exo methods by 5.8 points.","key_machinery":"Central machinery is the Dense Object Matcher (DOM) plus a Mask Refinement (MR) head. DOM builds per-proposal embeddings as $e=\\mathrm{Linear}(e_{\\mathrm{token}})+\\mathrm{Linear}(e_{\\mathrm{pos}})+\\mathrm{Linear}(e_{\\mathrm{label}})$ and processes the concatenated $2N$ proposal embeddings through repeated self-attention and cross-attention blocks against SAM 2 image features. The mix matching strategy multiplies row-normalized and column-normalized softmax similarities, so a pair is selected only when it is the best match in both directions. MR then injects the source and matched target embeddings into learnable output tokens that iteratively update against target image features to predict t","core_discovery":"The paper's central claim is that the ego-exo correspondence task should be treated as dense many-to-many matching, not one-to-one retrieval. DOMR jointly models all proposals from both views: each proposal is represented by the sum of a SAM 2 token embedding, a positional embedding of its bounding-box corners, and a CLIP label embedding from the open-vocabulary detector, then updated by self-attention and cross-attention against the fused SAM 2 image features. Matching scores are computed in both directions, and the mix strategy selects the pair whose Ego→Exo and Exo→Ego softmax similarities multiply to the largest value, so a match is accepted only if it is mutually most plausible. A separ","pith_inferences":["Beyond the paper's claims: the coverage table reports average best-proposal IoU but not the recall of ground-truth objects that receive no proposal at all; if such coverage gaps exist, DOMR inherits them and the headline IoU depends on an unmeasured proposal-recall rate.","Beyond the paper's claims: because more proposals monotonically improve performance, DOMR could be pushed further by exhaustive or category-aware proposal generation rather than the LVIS-vocabulary YOLO-UniOW detections.","Beyond the paper's claims: the mix strategy's symmetric-consistency product is task-agnostic and could transfer to other bidirectional dense-correspondence settings such as multi-view tracking or video object re-identification."],"forward_implications":["If the reported numbers hold, DOMR is the best published method on the Ego-Exo4D ego-exo correspondence benchmark, improving on ObjectRelator by 5.8 points Ego→Exo and 4.3 points Exo→Ego.","Spatial-only dense matching with refinement outperforms a spatio-temporal baseline (XView-Xmem + XSegTx) by roughly 12.8 and 19.1 points, suggesting temporal cues are not required for this benchmark's accuracy.","Increasing the number of proposals from 80 to 160 raises DOM's IoU, so the pipeline should continue to benefit from better or denser proposal generators.","Ablations show the mix strategy, the mask refinement head, and freezing DOM during refinement training each contribute; removing any of them lowers both task scores."],"supporting_citations":[{"why":"Supplies the Ego-Exo4D dataset, the ego-exo correspondence benchmark, the train/val split, and the XView-Xmem/XSegTx baselines.","marker":"[18]"},{"why":"ObjectRelator is the previous state-of-the-art method whose per-task IoU DOMR must beat.","marker":"[17]"},{"why":"SAM 2 generates mask proposals from box prompts and provides image features and the mask decoder used for refinement.","marker":"[36]"},{"why":"YOLO-UniOW is the open-vocabulary detector whose bounding-box proposals seed the DOM pipeline and whose LVIS labels provide semantic embeddings.","marker":"[27]"},{"why":"CLIP's text encoder produces the label embeddings that carry category-level semantics into the matching representation.","marker":"[35]"},{"why":"SAM supplies the promptable mask-decoding design and loss weighting that the mask refinement stage is initialized from.","marker":"[22]"}],"fun_headline_variants":["Dense cross-view matching lifts ego-exo IoU by 5.8%","Joint dense matching and refinement for ego-exo views","Mutual object matching beats prior ego-exo methods","DOMR: dense match and refine for cross-view segmentation","Many-to-many matching improves ego-exo object IoU"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that the YOLO-UniOW + SAM 2 proposal pipeline returns at least one mask with real overlap for every ground-truth object; the paper reports only average best-proposal IoU, never the fraction of objects that receive no usable proposal, and any such object cannot be rescued by matching or refinement.","fun_headline_variants_meta":{"raw":{"variants":["Dense cross-view matching lifts ego-exo IoU by 5.8%","Joint dense matching and refinement for ego-exo views","Mutual object matching beats prior ego-exo methods","DOMR: dense match and refine for cross-view segmentation","Many-to-many matching improves ego-exo object IoU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000342,"raw_usage":{"total_tokens":1735,"prompt_tokens":774,"completion_tokens":961,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":875}},"tokens_in":518,"tokens_out":961,"duration_ms":10698,"temperature":1.0,"reasoning_tokens":875,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:54:59.806713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on the Ego-Exo4D validation set, the proportion of ground-truth objects whose best proposal from the DOMR pipeline has IoU below 0.5 (or no proposal at all). If that proportion is non-negligible and DOMR never matches those objects, the reported mean IoU of 49.7/55.2 cannot be reproduced without changing the proposal source or adding a recovery mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Ego-Exo4D dataset, the ego-exo correspondence benchmark, the train/val split, and the XView-Xmem/XSegTx baselines."}],"review_version":1}