{"id":"93189dc3-775d-425c-aea1-017b0be77075","arxiv_id":"2509.04844","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"REMOTE is a unified multimodal relation extraction framework and dataset covering intra-modal and inter-modal relation triplets, reporting state-of-the-art F1 scores on three datasets.","lead":"This paper introduces REMOTE, a unified multimodal relation extraction framework that extracts text-text, object-object, and text-object relations in a single model, and a new UMRE dataset with 55,021 triplets. Generalists should read it because unified multimodal relation extraction could make multimodal knowledge graph construction simpler and because the new dataset gives the field a common benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MOT module's OT equations are dimensionally inconsistent; since Table 3 credits this module with the largest ablation gain on UMRE, the central novelty and its measured contribution cannot be independently reproduced until the marginal constraints and normalization are specified.","rationale":"The reader's weakest assumption correctly identifies the multilevel optimal transport module as the least secure part of the paper. The central claim of the paper has two components: a new unified task and dataset, and a framework whose novelty is the MOT module plus MMoE. The dataset contribution is plausible, but the framework contribution is not independently checkable because the mathematical specification of MOT is internally inconsistent. The dimension mismatch between the matrix-valued distributions and the vector-valued marginal constraints, together with the swapped source/target marginals and inconsistent cost-matrix indexing, means the equations as printed do not define a valid optimal transport problem. Since Table 3 attributes a large part of the reported performance to this module, the empirical claim inherits the uncertainty: if the released code uses a different normalization or a different alignment objective, then the paper is not actually demonstrating that multilevel optimal transport is the cause of the improvement. This is a load-bearing concern because it affects both reproducibility and the attribution of the central result, not merely a typo in an auxiliary equation. I agree with the reader that the issue is correctable and that the overall empirical direction is reasonable, so the CONDITIONAL verdict is appropriate. The requested concrete check directly tests whether the shipped implementation matches the intended OT formulation and whether the ablation gain is robust to a plausible alternative alignment mechanism.","tokens_in":16440,"tokens_out":5808,"duration_ms":55282,"concrete_test":"Inspect the released implementation of Eqs. 5-8: print the shapes of mu, nu, and Pi after any normalization and verify whether Pi*1_b = nu and Pi^T*1_a = mu hold with a=2u and b=l*2u. Then re-run the UMRE ablation with MOT replaced by a row-normalized cosine-attention baseline using the same source and target features; if the reported F1 gap (69.17 vs 64.82) shrinks or reverses, the OT module is not the source of the gain. Independently re-deriving Eq. 6 from Eq. 5 should also confirm that the stated feasible set is empty without the unstated normalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 is not merely imprecise; as written, Eqs. 5-8 cannot define the claimed optimal-transport fusion. Eq. 5 sets the source mu as a matrix in R^{(l*2u)*d} and the target nu as a matrix in R^{2u*d}. Eq. 6 then requires Pi in R^{a*b}_+ with Pi*1_b = mu and Pi^T*1_a = nu, where a=2u and b=l*2u. The left-hand sides are vectors of lengths a and b, while mu and nu are matrices with d columns; even after flattening, the marginals are swapped, since the row sum of a b-column plan is a length-a vector and should equal nu, not mu. The cost matrix C_ij in Eq. 7 is also indexed so that C would be b x a while the objective in Eq. 6 sums over an a x b plan. The only way to execute this module is to add an unstated step that turns feature rows into probability weights and to correct the marginal constraints. The ablation in Table 3 credits MOT with 78.47 vs 75.64 accuracy (69.17 vs 64.82 F1) on UMRE, so the central claimed contribution depends on exactly this unspecified operation. Without the normalization rule and the corrected plan shape, a reader cannot verify that the module performs optimal transport rather than a reweighted cosine attention, and the measured ablation gain cannot be attributed to OT.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes REMOTE, a unified multimodal relation extraction framework that jointly extracts intra-modal (text-text, object-object) and inter-modal (text-object) relational triplets. The method combines a Multilevel Optimal Transport (MOT) fusion module, intended to preserve low-level encoder features, with a Multimodal Mixture-of-Experts (MMoE) router that selects per-triplet interaction features. The authors also introduce the UMRE dataset, containing 55,021 triplets from 12,737 text-image pairs, built by extending MNRE and MORE with MLLM-assisted candidate extraction and manual verification. Experiments on UMRE, MORE, and MNRE report state-of-the-art or competitive results, with ablations attributing gains to the MOT and MMoE modules.","tokens_in":16742,"tokens_out":4548,"duration_ms":39527,"significance":"If the technical description is corrected and the results verified, the paper would make a useful contribution: it is the first to define and evaluate a unified task covering all three triplet types, it contributes a sizable human-verified benchmark, and it reports extensive comparisons with many recent baselines, including honest reproductions of FocalMRE on MNRE. The authors release their resources, which supports reproducibility. The main weakness is that the central MOT module is specified with dimensionally inconsistent equations, and the empirical claims on the more established datasets rest on small margins without variance reporting.","major_comments":[{"comment":"The optimal transport formulation is dimensionally inconsistent and therefore under-specified. The source is defined as μ∈R^{(l·2u)×d_v} and the target as ν∈R^{2u×d_v}; the text sets b=l·2u and a=2u. With these definitions, a plan transporting μ onto ν should be in R^{b×a}, with row-sum μ and column-sum ν. Instead, Eq. (6) states Π∈R^{a×b}_+ with constraints Π1_b=μ and Π^T1_a=ν. The left-hand sides are vectors of lengths a and b, while μ and ν are matrices with d_v columns, so the constraints are not well-formed. Equation (8), F_V^{l'}=Π^*·μ, is dimensionally valid only if Π^*∈R^{a×b} and μ∈R^{b×d_v}, which contradicts the stated a,b and the stated constraints. The cost matrix in Eq. (7) is also indexed as C_{ij} for i∈[a], j∈[b] but then described as C∈R^{m×n} without specifying m,n. Moreover, the paper does not explain how feature matrices are converted into probability distributions; an OT plan requires probability vectors, not raw feature rows. Because Table 3 attributes the largest ablation gain to MOT (78.47 vs 75.64 accuracy and 69.17 vs 64.82 F1 on UMRE), the reader cannot independently determine that the module performs optimal transport rather than a reweighted similarity operation. Please correct the plan dimensions, marginal constraints, cost indexing, and specify the normalization (or learnable weighting) used to obtain histograms from feature rows.","section":"Section 6.1 and Table 2"},{"comment":"The empirical claim that REMOTE outperforms all baselines on almost all metrics is not fully supported by the reported statistics because Section 6.1 only states that experiments are averaged over 3 runs, but no standard deviations or significance tests are provided. On the MORE dataset, the reported improvements over the best baseline are small (e.g., +0.14 accuracy and +1.16 F1), so without variance information these gains may be within run-to-run variability. The larger gains on the new UMRE dataset are more convincing, but the general claim of superiority across all three datasets would be stronger with error bars or significance tests.","section":"Section 6.4 and Table 2"}],"minor_comments":[{"comment":"The abstract contains a typo: 'performanc' should be 'performance'. In Section 5, 'as shown in in Fig. 3' duplicates the word 'in'.","section":"Abstract and Section 5"},{"comment":"The formatting of the FocalMRE row (e.g., '88.85/86.96*' and '88.01/86.21*') and of the ΔSOTA row is confusing; please split reported and reproduced results into separate columns or clearly label them.","section":"Table 2"},{"comment":"References [15] and [16] are duplicates of the same paper (Hu et al., ACL 2023); please merge them.","section":"References"},{"comment":"The annotation section mentions a Weighted Cohen's Kappa of 0.7325, but it does not state the number of samples that were double-annotated or the exact scoring procedure; please clarify the protocol.","section":"Section 4.2"},{"comment":"The notation 'H_{T,V}^{⟨s⟩}' and 'H_{V,T}^{o_i}' in Eq. (13) is not explicitly defined; please clarify which of the previously introduced interaction features are used for textual entities versus visual objects.","section":"Eq. (13)"}],"recommendation":"major_revision","confidential_remarks":"The dimensional inconsistency in Section 5.2 is the main concern; it is serious because the MOT ablation is the largest single contributor in Table 3, but it appears fixable in revision by providing the correct plan shape, marginal constraints, and normalization. I recommend asking the authors to also report standard deviations, and to clarify the cross-attention ablation variant. Given that the code is released, the authors should be able to address these points concretely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing about this paper is that the dataset is the contribution, not the model. UMRE unifies three triplet types in one benchmark and is larger than MNRE/MORE; that alone justifies attention from anyone in multimodal information extraction. The model (REMOTE) is a sensible combination of known pieces—MoE routing, hierarchical cross-attention, and a multilevel fusion module—and the evaluation is extensive: three datasets, adapted baselines, ablations, subtask analyses, and a case study. I also give them credit for reporting the reproduced FocalMRE numbers and letting it show that they do not beat the original MNRE F1; that is honest.\n\nThe soft spots are real. The multilevel optimal transport section, which is the main methodological novelty, is under-specified to the point of not being implementable as written. The source mu is a matrix with l*2u rows and d columns, the target nu has 2u rows, and the constraints Pi 1_b = mu and Pi^T 1_a = nu cannot hold because the left sides are vectors and the right sides are matrices. The cost matrix indices are swapped relative to the plan. The authors must have an unstated normalization that turns feature rows into probability weights, but it is not in the paper. This matters because Table 3 credits MOT with the largest ablation gain on UMRE (69.17 vs 64.82 F1). If the module is actually a reweighted attention, the attribution changes. This is correctable, but it is not a cosmetic typo.\n\nAlso, no variance is reported even though they say they ran 3 averages. The SOTA claim on MNRE holds only against their reproduced FocalMRE baseline, not the original published numbers; they do flag it with the asterisk, but the abstract's 'state-of-the-art performance' phrasing overstates it. The dataset construction relies on MLLMs plus human review; fine for a benchmark, though the kappa of 0.7325 is moderate.\n\nIf this crossed my desk as a submission, I would send it out. The dataset is valuable, the experiments are broad, and the OT flaw is fixable. For a reader in MRE, the UMRE dataset is worth citing now; the model is worth a careful look only after the OT equations are corrected and code is released.","headline":"The UMRE dataset is a genuine contribution; the model's optimal transport module is not reproducible as written, and the ablation that credits it most cannot be checked until the authors fix the math and release code.","tokens_in":17286,"tokens_out":2235,"would_cite":true,"duration_ms":19820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims unified multimodal relation extraction is feasible, and that its REMOTE framework with multilevel optimal transport and mixture-of-experts outperforms all compared baselines on almost every metric across the new UMRE…","keywords":["unified multimodal relation extraction","mixture-of-experts","optimal transport","multilevel feature fusion","dataset construction","multimodal knowledge graph","relation extraction"],"falsifier":"Run the released REMOTE code on UMRE with the MOT module replaced by a dimensionally correct transport plan, for example a Sinkhorn plan with feature rows as marginals, and compare against Table 3 (F1 69.17 with OT, 64.82 without). If the corrected OT plan does not reproduce the gain, the claimed mechanism is falsified. Also verify whether Eq. 6's constraints hold for the shapes in the implementation: with $a=2u$ and $b=lu$, $\\Pi \\mathbf{1}_b = \\mu$ requires $\\mu$ to be a vector of length $a$, not a $(b \\times d)$ matrix.","tokens_in":1598,"feed_emoji":"🔗","tokens_out":3614,"duration_ms":58989,"temperature":0.7,"pith_summary":"The paper argues that prior multimodal relation extraction is artificially split into tasks that each handle only one kind of triplet, and that a single framework can extract all three kinds—text-to-text, object-to-object, and text-to-object—at once. To make that case, it introduces the UMRE dataset with 55,021 triplets across 12,737 image-text pairs, and the REMOTE model, which fuses encoders' low-level and high-level features through optimal transport and routes triplet-specific features through a mixture-of-experts module. On UMRE and on the existing MNRE and MORE benchmarks, REMOTE reports higher F1 than all compared baselines, including multimodal LLMs, which the paper reads as evidence that unified extraction with dynamic feature selection works. A sympathetic reader cares because this collapses three separate research lines into one task and one method, and supplies a benchmark for that task.","feed_headline":"One model extracts all three kinds of multimodal relations.","feed_subtitle":"REMOTE plus a new 55k-triplet benchmark beats single-type MRE models on F1 across three datasets.","key_machinery":"The central object is the multilevel optimal transport (MOT) fusion module. It takes low-level features from encoder layers $0$ to $l-1$ as the source distribution $\\mu$ and the high-level feature at layer $l$ as the target distribution $\\nu$, solves a Sinkhorn-regularized transport problem to obtain a plan $\\Pi^*$, and fuses the transported low-level features with the high-level features through a learned weight $\\alpha$. Together with the multimodal mixture-of-experts (MMoE) router, which weight-combines textual-only, visual-only, and cross-modal hierarchical attention features per triplet, MOT is what the paper claims carries the improvement. The UMRE dataset itself is also machinery: it provides the three triplet types and the 28-relation label space that lets the method be trained and evaluated as a single task.","core_discovery":"The central discovery claim is that unified multimodal relation extraction—predicting relations among textual entities, among visual objects, and across text and images in one model—is feasible and beats task-specific designs. The evidence is the UMRE dataset and the REMOTE framework: multilevel optimal transport preserves low-level visual and textual detail that single-layer fusion loses, and the multimodal mixture-of-experts router assigns each relational triplet the interaction features best suited to it, giving a 5.3-point F1 gain over the previous best on UMRE and consistent gains on MORE and MNRE. The paper further claims that this is the first unified formulation of the task, going beyond both MNRE-style text-entity extraction and MORE-style text-object extraction.","pith_inferences":["The OT component may not be load-bearing: the ablation that removes MOT also removes multilevel feature depth, so the 4.35-point F1 drop on UMRE could be due to losing low-level features altogether rather than to transport itself.","A dimensionally faithful reimplementation of Eqs. 5-8 would clarify whether the transport plan is computed over feature rows or over a per-example probability vector; the released code should settle this.","The same router-plus-multilevel-fusion idea could transfer to other multimodal tasks such as entity normalization or visual question answering, where fine-grained object detail and text semantics interact.","The dataset construction relies on MLLM candidates and human adjudication, and the reported Kappa of 0.7325 suggests the task is harder than typical relation extraction, so future work may need more refined annotation protocols."],"forward_implications":["Unified MRE becomes a single benchmark problem, so future methods no longer need separate designs for MNRE-style and MORE-style extraction.","MLLM-generated captions plus low-level features become a reusable recipe: the paper shows that stronger captioning MLLMs improve extraction without retraining the core model.","Mixture-of-experts routing gives a per-triplet choice of modality evidence, and the weight visualizations suggest that spatial-temporal relations favor lower-layer features while role relations favor visual and bidirectional features.","The UMRE dataset gives the community a large testbed with 55,021 triplets, 28 relations, and three triplet types, enabling direct comparison of unified approaches.","The framework's state-of-the-art numbers on three datasets imply that task-specific modality filtering can be replaced by a single dynamic router without losing performance."],"supporting_citations":[{"why":"Supplies the MNRE dataset and the text-entity relation extraction task that UMRE generalizes.","marker":"[40]"},{"why":"Defines the MORE task and dataset, and the visual encoding with position and depth that REMOTE extends.","marker":"[12]"},{"why":"Provides the entropy-regularized Sinkhorn algorithm used by the multilevel optimal transport fusion module.","marker":"[6]"},{"why":"Supplies the MEGA cross-modal alignment baseline that REMOTE must beat.","marker":"[41]"},{"why":"Supplies the MKGformer hybrid transformer baseline and motivation for multilevel fusion.","marker":"[4]"},{"why":"Supplies the HVFormer visual mixture-of-experts baseline that REMOTE compares against.","marker":"[26]"},{"why":"Supplies the FocalMRE baseline and the focal attention idea that motivates image-region emphasis.","marker":"[13]"}],"fun_headline_variants":["REMOTE: all multimodal relations in one unified model","Text, image, cross-modal: one extraction framework","Mixture-of-experts and optimal transport unify MRE","Beyond single-type: unified multimodal relation extraction","REMOTE framework sets new MRE SOTA with multimodal fusion"],"cache_read_input_tokens":19328,"weakest_assumption_plain":"The load-bearing premise is that the multilevel optimal transport module performs a genuine, well-defined feature alignment; as printed, the equations are dimensionally inconsistent, so if the implementation does not match the described transport problem, the attribution of the gains to OT is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["REMOTE: all multimodal relations in one unified model","Text, image, cross-modal: one extraction framework","Mixture-of-experts and optimal transport unify MRE","Beyond single-type: unified multimodal relation extraction","REMOTE framework sets new MRE SOTA with multimodal fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1564,"prompt_tokens":967,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":583,"tokens_out":597,"duration_ms":5474,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:25:48.863557+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released REMOTE code on UMRE with the MOT module replaced by a dimensionally correct transport plan, for example a Sinkhorn plan with feature rows as marginals, and compare against Table 3 (F1 69.17 with OT, 64.82 without). If the corrected OT plan does not reproduce the gain, the claimed mechanism is falsified. Also verify whether Eq. 6's constraints hold for the shapes in the implementation: with $a=2u$ and $b=lu$, $\\Pi \\mathbf{1}_b = \\mu$ requires $\\mu$ to be a vector of length $a$, not a $(b \\times d)$ matrix.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MNRE dataset and the text-entity relation extraction task that UMRE generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the entropy-regularized Sinkhorn algorithm used by the multilevel optimal transport fusion module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MEGA cross-modal alignment baseline that REMOTE must beat."}],"review_version":2}