{"id":"fcef09e8-1fbf-4fbd-9e40-5bb729dc9b5c","arxiv_id":"2506.22062","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MDC-R adds expert annotations of anaphoric and deictic reference, with block-level IDs and bounding boxes, to 101 Minecraft building dialogues, and shows that current referring-expression models struggle on this dynamic benchmark.","lead":"This paper introduces MDC-R, a version of the Minecraft Dialogue Corpus annotated for anaphoric and deictic reference by expert linguists, with each referring expression mapped to the specific blocks it points to. The resource lets researchers study how people refer to objects in a world that changes while they talk, and gives multimodal models a new, hard benchmark for referring expression comprehension.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Annotation reliability on block-ID sets is the load-bearing assumption; kappa 0.43 does not demonstrate it, and the reported lower-bound interpretation conflicts with the corpus's central claim.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: expert annotations of block-ID sets must be reliable enough to serve as gold labels. I find no independent issue that supersedes this. The paper itself provides only a small double-coded sample with kappa 0.43, and the authors' rationale for treating it as a lower bound actually targets the exact granularity the resource is meant to provide. The REC experiments do not add independent evidence of label quality because model performance is measured against the same annotations. The proposed concrete test would settle whether single-annotator labels are consistent at scale: if exact block-set agreement is high, the corpus is a credible resource; if not, the central claim about constituent parts is undermined and any downstream benchmark would be noisy. The reader's CONDITIONAL verdict already reflects this condition, so no change to the verdict is warranted. I also note the REC subset selection (7,600 markables down to 423 expressions) is underspecified, but this affects only the demonstration of usefulness, not the core resource claim, so it is secondary. No concerns about authorship or internal consistency arose; the paper is transparent about its limitations. The non-finding option is not appropriate because the reliability gap is concrete and testable, and the paper itself flags no equivalent caveat about annotation consistency.","tokens_in":13919,"tokens_out":2514,"duration_ms":30143,"concrete_test":"Select a random sample of 20 MDC-R dialogues (beyond the original 5), have two experienced annotators independently apply the released guidelines to the same markables, and compute: (i) Cohen's kappa on mention-span detection; (ii) per-markable exact-match agreement on the block-ID set; and (iii) relaxed agreement where two sets count as agreeing if their block-ID IoU >= 0.5, reported overall and by NP category (NP1 vs NPn). If exact-set agreement is materially lower than relaxed agreement, the 'lower bound' interpretation is misleading and the constituent-part labels are not reliable for benchmarking; if exact-set agreement is high (e.g., > 0.7), the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MDC-R reliably contributes 'information about the constituent parts of the referenced objects' (Section 8) as gold labels for situated reference modelling. The constituent-part information is precisely the annotator-entered block-ID sets. The sole reliability evidence (Section 5.1) is a preliminary kappa of 0.43 on 5 double-coded dialogues (~400 markables), with the remaining 96 dialogues singly annotated. The authors call this a lower bound because disagreements often involve slightly different sets of blocks. However, those set-level differences are exactly the granular object-set distinctions that make the corpus novel. If single-annotator labels are inconsistent at a similar rate, then the 7,600 markables, the 1,150 bounding boxes, and the REC baselines computed against them all inherit noise. The paper's Limitations section admits partial coverage and single-perspective annotation but does not quantify the risk of label inconsistency at scale. The kappa value as reported conflates mention detection with object-set assignment, so the reliability of the critical constituent-part labels is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MDC-R, an annotation layer on top of the Minecraft Dialogue Corpus (MDC). The authors extend the CODI/CRAC-ARRAU annotation scheme with block-level grounding: each referring expression is linked to unique block IDs and to bounding boxes derived from screenshots of the dynamic Minecraft world. The paper reports corpus statistics, qualitative observations on dynamically changing scenes, perspective, grounding, and intensional objects, and a small referring expression comprehension (REC) experiment using Qwen2-VL and MDETR. The central claim is that MDC-R is a new resource for studying anaphoric and deictic reference in a situated, dynamic setting and is, in the authors' words, 'the first dataset which provides information about the constituent parts of the referenced objects' (Section 8).","tokens_in":13993,"tokens_out":5658,"duration_ms":61758,"significance":"If the reliability of the block-ID annotations can be established, MDC-R is a potentially valuable resource for situated reference, anaphora, and REC in dynamic visual environments. The paper openly releases the corpus and is transparent about its pilot reliability study and limitations. The annotation scheme is a documented extension of existing guidelines, and the baseline experiments provide a first probe of task difficulty. The low baseline performance itself signals that the corpus is challenging and likely useful for future work. However, the reliability evidence is currently too thin to treat the gold labels as authoritative, and the REC experiment is underspecified in several ways. The resource is significant in potential, but the paper needs additional annotation-quality analysis before the central claims can be fully supported.","major_comments":[{"comment":"The reliability evidence for the central annotation decision is insufficient. The only inter-annotator agreement reported is a preliminary kappa of 0.43 on 5 dialogues containing about 400 markables, while Table 1 reports 7,600 markables across 101 documents and Section 5.3 states that the data were annotated by two professional linguists without indicating that the remaining 96 dialogues were double-coded. The 'lower bound' interpretation is not convincing: disagreements about slightly different sets of blocks are not a side issue but concern exactly the constituent-part information that the corpus claims to provide. The reported kappa also conflates mention detection with object-set assignment. The authors should report agreement separately for markable detection, for block-set membership, and for exact set equality, ideally with a larger double-coded sample and post-adjudication agreement. Without this, the gold labels, the bounding boxes, and the REC baselines built on them inherit an unquantified level of noise.","section":"Section 5.1, Table 1, Section 8"},{"comment":"The REC subset is underspecified, which weakens the claim that the experiment demonstrates the corpus's usefulness. The paper states that the dataset comprises 1,150 bounding boxes, 423 referring expressions, and 101 distinct scenarios, but it does not say how the 423 expressions were sampled from the 7,600 markables, whether they are all deictic referents with associated screenshots, or how the ground-truth merged bounding box is derived from the block coordinates. No train/test split, few-shot setup, or evaluation protocol is described. The results in Tables 2 and 3 are reported without confidence intervals or significance tests, and Table 4 contains several NP categories with only one or two examples, making those rows uninformative. In addition, the Qwen2-VL evaluation uses the merge-box protocol while the MDETR evaluation uses per-block boxes, so the mIoU values in Tables 2 and 3 are not directly comparable. The authors should provide the full dataset-construction details, report variance, and clearly separate the two evaluation protocols.","section":"Section 7.1 and Section 7.5"},{"comment":"The claim that MDC-R is 'the first dataset which provides information about the constituent parts of the referenced objects' is too strong without a systematic comparison to existing resources. Several simulated environments (e.g., CLEVR, CAESAR, REVERIE) provide object-level or part-level annotations, and the novelty of MDC-R appears to be the combination of dynamic dialogue, block-level grounding, and bounding boxes, not the mere existence of constituent-part information. The claim should be scoped to the specific combination of features or supported with a comparison table. This is not a fatal issue, but it affects how the contribution is positioned.","section":"Section 8"}],"minor_comments":[{"comment":"There is a numerical inconsistency: Section 5.3 says a random subset of 100 dialogues was selected, while Table 1 reports 101 documents and Section 7.1 reports 101 distinct scenarios. This should be reconciled.","section":"Section 5.3 and Table 1"},{"comment":"The reliability statistic is reported only as 'kappa = 0.43' without specifying whether it is Cohen's kappa or Krippendorff's alpha, or what the annotation unit and chance-correction model were. This information is needed to interpret the value.","section":"Section 5.1"},{"comment":"The footnote says 'random bounding box limits (mIoU = 5.1) and limits over the entire image (mIoU = 10.8)'; this appears to contain a typo, and the intended meaning of 'limits' is unclear. The sentence should be rewritten to describe the random baseline procedures.","section":"Section 7.5"},{"comment":"The NP category table omits several values (NP11, NP17, NP21-NP27) without explanation; the authors should state that these categories have zero quantity, or otherwise explain the gaps. Rows with one or two instances should not be interpreted as meaningful performance estimates.","section":"Table 4"},{"comment":"The sentence 'we discuss in this Section how a selection of interesting linguistic phenomena in this Section end up being captured' contains a redundancy and should be rephrased.","section":"Section 6.1"},{"comment":"The caption contains a typo: 'thee bell' should be 'the bell'.","section":"Figure 6 caption"},{"comment":"There are several small reference errors, including 'V on Ahn' (should be 'von Ahn') and duplicate entries for He et al. 2023a/2023b; these should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a resource paper from a well-known group, and the dataset itself is likely to be useful to the community. The main risk is that the reliability evidence is too thin for the corpus to serve as gold standard. I recommend major revision rather than rejection because the issues can be addressed with additional agreement analysis, adjudication, and a clearer description of the REC subset. I would also ask the authors to double-check the 'first dataset' claim against related work before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MDC-R is worth knowing about. It is the first resource that couples reference annotations in a dynamic, multi-turn situated dialogue with constituent-block grounding: every markable can be linked to the specific blocks it denotes through unique IDs and 2D bounding boxes. That combination does not exist in MDC, MSDC, Spatial AMR, or the CODI-CRAC dialogue corpora. The annotation scheme extends the authors' own ARRAU/CODI approach, which is a natural move rather than a defect, and the qualitative analysis of perspective errors, conceptual pacts, and intensional objects is genuinely informative. The paper also ships the corpus and runs two baselines (Qwen2-VL and MDETR) that make the task look hard, which is useful for the field.\n\nThe load-bearing issue is annotation reliability. The only double-coding is five dialogues (~400 markables) with kappa = 0.43; the other 96 dialogues are singly annotated. The authors call 0.43 a lower bound because disagreements often involve slightly different block sets, but that is exactly the layer of information the corpus is novel for. The lower-bound framing does not follow: a single annotator may be more consistent than two coders in pairwise agreement, or less; we have no data. So the central artifact, the block-ID sets, carries unquantified risk. The paper's Limitations section is honest about partial coverage and single perspective, but it does not confront this reliability question directly. In addition, the REC subset of 423 expressions is underspecified (how were plurals, zeros, or uncertain mentions filtered?) and the baseline results are single runs without confidence intervals or a human ceiling.\n\nThese are fair referee questions, not fatal flaws. The resource is available, the methodology is described in enough detail to critique, and the task is clearly motivated. I would send it to peer review with a request for a second round of reliability coding or at least an explicit statement about how much label noise the baselines can tolerate. If the authors can show that single-annotator labels are stable at scale, this becomes a strong resource; if not, it remains a useful pilot. The citation pattern is appropriate; the ARRAU/CODI lineage is handled transparently. Overall: a solid, honest resource paper that needs one more piece of evidence before I would trust its gold labels.","headline":"A genuinely new situated-reference resource with block-level grounding, but the reliability evidence for its central annotation layer is thinner than the paper's own lower-bound framing suggests.","tokens_in":14665,"tokens_out":3015,"would_cite":true,"duration_ms":34032,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MDC-R, a Minecraft dialogue corpus in which every anaphoric and deictic reference is annotated and linked through block IDs and bounding boxes to the exact blocks it denotes.","keywords":["Minecraft Dialogue Corpus","reference annotation","anaphoric reference","deictic reference","referring expression comprehension","situated dialogue","multimodal benchmark","voxel grounding"],"falsifier":"Double-annotate a random sample of the singly annotated dialogues and measure agreement on block-set assignment; if Cohen's kappa on which blocks a referring expression denotes falls well below 0.43, or if disagreements systematically concern whether an expression refers to one block versus a multi-block structure, the claim that MDC-R provides reliable gold labels for constituent-part reference is undercut.","tokens_in":13637,"feed_emoji":"🧱","tokens_out":7107,"duration_ms":65081,"temperature":0.7,"pith_summary":"The paper introduces MDC-R, a new language resource built on the Minecraft Dialogue Corpus: 101 dialogues in which an Architect instructs a Builder to construct a structure, with expert annotations of anaphoric and deictic reference. Each referring expression is linked through unique block IDs and 2D bounding boxes to the specific blocks it denotes, making this, in the authors' account, the first dataset that provides information about the constituent parts of referenced objects. The paper argues this resource enables studying reference in a dynamic, multi-turn situated setting, where the visual scene changes as the dialogue unfolds, and it backs this with a referring-expression comprehension experiment in which two strong baselines perform markedly worse than on static-image benchmarks, indicating a challenging new testbed for multimodal reasoning.","feed_headline":"Minecraft chat corpus grounds every reference down to the exact blocks","feed_subtitle":"Per-block IDs and bounding boxes turn 101 builder dialogues into a hard benchmark for multimodal reference.","key_machinery":"The load-bearing mechanism is the annotation infrastructure: each voxel block in the world is assigned a unique alphanumeric tuple label, tied to its 3D Cartesian coordinates, and annotators using a modified MMAX2 interface see the object-labelled image of the world before each utterance and enter the labels of the blocks a noun phrase refers to in an 'Object' slot. The coordinates then yield 2D bounding boxes in the perspective-based screenshot, so a referring expression is grounded as a set of blocks. This lets the corpus support both classic REC with a merged single bounding box and generalized REC with multiple bounding boxes, and it makes reference to abstract shapes and composites, such as towers, beams, and bells, transparently decomposable into their constituent voxels.","core_discovery":"MDC-R is an expert-annotated extension of the Minecraft Dialogue Corpus in which every referring expression in 101 dialogues (7,600 markables) is annotated for anaphoric and deictic reference and grounded to the set of voxel blocks it denotes. Each block carries a unique alphanumeric label tied to its 3D coordinates, and each reference is associated with the object-labelled image of the world state at that utterance, from which a 2D bounding box is derived; references thus reduce to references to sets of blocks, which the authors align with generalized referring expression comprehension. The authors claim MDC-R is the first dataset to provide information about the constituent parts of referenced objects, and they position it as a resource for studying reference in a dynamically changing visual world, where grounding, perspective mismatches, and conceptual pacts produce complex referential phenomena.","pith_inferences":["If annotation reliability can be raised beyond the reported kappa of 0.43 on the double-coded subset, MDC-R could also serve as a testbed for models that track object identity across state changes, since block IDs persist while the scene is edited.","The object-set granularity invites a move beyond bounding-box evaluation: the corpus could support voxel-level grounding metrics that penalize partial-set predictions more precisely than IoU on a merged box.","The apparent fuzziness of set boundaries, where 'the green block' can mean one block or a whole tower, suggests referent sets might be better modelled as distributions, and the corpus's disagreement data could be used to study that ambiguity."],"forward_implications":["MDC-R provides a benchmark for referring expression comprehension in a dynamic, multi-turn situated setting, where the scene changes as the dialogue progresses and the two interlocutors hold different perspectives.","Because references are grounded as block sets, the corpus supports generalized REC with multiple targets, going beyond the single-object assumption of classic datasets.","The low baseline scores (mIoU 30.4 for Qwen2-VL in classic REC; F1@0.5 of 9 for MDETR in GREC) indicate that state-of-the-art multimodal models still struggle with embodied, dialogue-grounded reference in voxel worlds.","The annotations capture rare linguistic phenomena, including intensional references to objects not yet built, bridging references, and negotiated conceptual pacts, that static-image REC datasets do not exhibit."],"supporting_citations":[{"why":"Supplies the original Minecraft Dialogue Corpus and Collaborative Building Task that MDC-R annotates.","marker":"Narayan-Chen et al., 2019"},{"why":"Defines the CODI/CRAC annotation scheme for anaphora, bridging, and discourse deixis in dialogue that MDC-R extends.","marker":"Yu et al., 2022a"},{"why":"The ARRAU 3 annotation manual, the underlying scheme for the CODI/CRAC extension used here.","marker":"Poesio et al., 2024a"},{"why":"Provides the method for annotating references to the visual situation on demand, adapted to block labels.","marker":"Loáiciga et al., 2022"},{"why":"The MMAX2 annotation tool that was modified to show object-labelled images and record block IDs.","marker":"Müller and Strube, 2006"},{"why":"RefCOCO, the classic REC dataset used as the baseline benchmark context for comparison.","marker":"Kazemzadeh et al., 2014"},{"why":"Qwen2-VL, the multimodal large language model evaluated in the classic REC experiment.","marker":"Wang et al., 2024"},{"why":"GREC and gRefCOCO, the generalized REC framework used to train and adapt the MDETR baseline.","marker":"He et al., 2023a"}],"fun_headline_variants":["Minecraft chat corpus grounds every reference to blocks","MDC-R: Expert reference annotations for Minecraft dialogue","101 dialogues, 7600 references, all grounded in Minecraft","New Minecraft corpus pins references to exact voxel blocks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expert annotations are reliable enough to be gold labels, even though only 5 of 101 dialogues were double-coded, with a kappa of 0.43, and the rest were annotated by a single annotator.","fun_headline_variants_meta":{"raw":{"variants":["Minecraft chat corpus grounds every reference to blocks","MDC-R: Expert reference annotations for Minecraft dialogue","101 dialogues, 7600 references, all grounded in Minecraft","New Minecraft corpus pins references to exact voxel blocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1104,"prompt_tokens":823,"completion_tokens":281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":217}},"tokens_in":439,"tokens_out":281,"duration_ms":3933,"temperature":1.0,"reasoning_tokens":217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:12:08.804975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Double-annotate a random sample of the singly annotated dialogues and measure agreement on block-set assignment; if Cohen's kappa on which blocks a referring expression denotes falls well below 0.43, or if disagreements systematically concern whether an expression refers to one block versus a multi-block structure, the claim that MDC-R provides reliable gold labels for constituent-part reference is undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The MMAX2 annotation tool that was modified to show object-labelled images and record block IDs."}],"review_version":1}