Pith. sign in

REVIEW 4 major objections 6 minor 62 references

M$^{3}$D: A Multimodal, Multilingual and Multitask Dataset for Grounded Document-level Information Extraction

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper constructs M3D, a 4,093-document video-and-text dataset in English and Chinese annotated for entity recognition, coreference chains, relation extraction, and visual grounding, and reports benchmark F1 scores of 53.80 and 53.77.

desk verdict The dataset is genuinely new and the construction is careful, but the benchmark numbers hinge on split hygiene that the paper never demonstrates, and the data are not released. read the letter →

arxiv 2412.04026 v2 pith:P5CB4QID submitted 2024-12-05 cs.CL

Jiang Liu , Bobo Li , Xinran Yang , Na Yang , Hao Fei , Mingyao Zhang , Fei Li , Donghong Ji This is my paper · ORCID
classification cs.CL
keywords multimodalinformationextractiondocument-levelvisualgroundingcoreferencechainsrelationmultilingualdatasetvideo-textbenchmarkmissingmodalityreconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper constructs M3D, a dataset of 4,093 biography video-and-subtitle pairs in English and Chinese, annotated for four information extraction tasks: named entity recognition, coreference chain extraction, relation extraction, and visual grounding. It argues that existing multimodal IE datasets are mostly sentence-level, image-based, and English-only, whereas M3D offers document-level video-text grounding in two languages. To make the dataset usable as a benchmark, it also designs a hierarchical model that fuses denoised text and video features and that can reconstruct a missing modality; the model reaches average F1 of 53.80 on English and 53.77 on Chinese, slightly above the MDocRE-HN baseline. The intended contribution is a reusable resource and a reference standard for studying grounded, document-level multimodal IE.

What carries the argument

The central mechanism is a hierarchical cross-modal fusion pipeline. Encoders (Longformer for text, ViT for images) produce layer-wise features that are grouped into low, middle, and high levels; the DFFM passes each level through a VAE encoder, applies multi-head attention with the other modality's VAE-reduced features, and reconstructs the fused feature through the VAE decoder. The MMCM generates absent modality features by concatenating a randomly initialized prompt with the available modality's convolved features and reconstructing through a convolutional layer. For visual grounding, the YOLO-format bounding boxes are predicted by classification plus mean absolute error regression on the fused image features.

What would settle it

Look at the released dataset's metadata for each clip's source video identifier and check whether clips from one source video appear in both training and test; alternatively, compute transcript n-gram overlap between splits and see whether any test document's text is nearly identical to a training document. If either check finds leakage, the reported numbers, including the advantage over MDocRE-HN, should be treated as inflated.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that M3D is a valid multimodal, multilingual, multitask dataset and that its hierarchical model is a reliable benchmark, achieving average F1 53.80 on English and 53.77 on Chinese across the four tasks. The dataset contains two modalities (video frames and subtitle text), two languages (English and Chinese), four tasks, and a biography domain. The paper further claims that the Denoised Feature Fusion Module (DFFM), which uses a variational autoencoder to denoise low-, mid-, and high-level features before cross-modal attention, and the Missing Modality Construction Module (MMCM), which generates missing modality features from prompt and convolutional features, each contribute measurable gains in ablation experiments.

Load-bearing premise

The benchmark's reported scores assume that no two clips from the same original biography video end up in different splits, so near-duplicate footage and subtitles cannot leak between training and testing.

Editorial extensions

If this is right

  • Any model trained on M3D is evaluated on four tasks in the same document, so document-level cross-task interactions (e.g., chains feeding relations) can be studied jointly.
  • The reported baselines give later work concrete numbers to beat, including the observation that zero-shot large language models perform far below supervised baselines.
  • The ablation results imply that both denoised fusion and missing-modality construction are needed for the best robustness when video or text is absent.
  • The 8:1:1 splits and published annotation statistics support direct comparison of entity, chain, relation, and grounding performance in English and Chinese.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state whether the 8:1:1 split was done on clips or on whole source videos; if a source video contributes clips to both training and test, the reported F1 numbers, including the margins over MDocRE-HN, could be inflated by near-duplicate footage and transcripts.
  • Because the multilingual component is two independently collected corpora rather than aligned translations, the dataset supports per-language benchmarking but not direct cross-lingual transfer experiments.
  • The visual grounding evaluation uses IoU > 0.5, a coarse localization standard; a stricter threshold might change the ranking of models on the grounding task.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces M3D, a new multimodal, multilingual, multitask dataset for document-level information extraction. The dataset contains 4,093 video-text documents in English and Chinese, annotated for named entity recognition, entity chain (coreference) extraction, relation extraction, and visual grounding. The authors also propose a hierarchical multimodal model with a denoised feature fusion module (DFFM) and a missing modality construction module (MMCM), reporting average F1 scores of 53.80 on English and 53.77 on Chinese, with ablations and error analyses.

Significance. If the dataset is released and its construction is validated, M3D would be a useful resource: it is among the first document-level video-text IE datasets, covers two languages and four tasks in one benchmark, and introduces a biography domain with fine-grained visual grounding annotations. The annotation specifications are detailed, inter-annotator agreement is reported, and the model baseline is accompanied by ablations and error analysis. However, the central artifact is not currently released, and the benchmark's validity depends on split hygiene and the evaluation protocol, both of which have unresolved issues.

major comments (4)
  1. [Section III (Step 2 and Dataset Statistics; Table II)] The paper does not state whether clips cut from the same source video are kept within the same train/dev/test split. Step 2 splits 30+ minute videos into one-to-two-minute clips, so multiple documents originate from one source video; the statistics paragraph describes only an 8:1:1 document-level split. If clips from one biography video straddle splits, near-duplicate frames and subtitles leak across train and test, which would inflate all F1 scores in Table III, including the 1.15% and 1.35% margins over MDocRE-HN. Please document the split at the source-video level, report the number of source videos per split, and verify that no two documents from the same source video appear in different splits; if document-level splitting was used, re-run the experiments with a leakage-free split.
  2. [Section VI.C (Implementation Details)] The sentence 'For entity chain extraction and relation extraction tasks, their entity (chain) pairs are both derived from golden entity (chain) pairs' means the reported Cha. and Rel. F1 scores are conditional on gold entities and chains, not end-to-end extraction performance. This is inconsistent with the joint task definitions in Eqs. (1)-(3), and it weakens the claim that Table III establishes a benchmark for the four tasks as defined. Please report end-to-end scores as well, and in any case disclose the conditional protocol in the table and text and discuss how each baseline is affected by the gold-pair assumption.
  3. [Section VI.C (Implementation Details) and Table III] The missing-modality setup is under-specified: 'Our main experiments will divide the train/dev/test sets into three equal parts, which are set to full modality, missing text modality, and missing visual modality.' Table III reports only a single average per task and language, with no per-condition breakdown, and the text does not state how each baseline encodes a missing modality (e.g., zeroing, blanking, or prompt construction). Because two-thirds of every split is corrupted by construction, the reported 'full modality' numbers are not full-modality numbers, and the comparison across systems depends on the unspecified corruption encoding. Please report condition-wise results and specify the missing-modality representation used for every model.
  4. [Section I (Contributions) and general] The central artifact of the paper is the dataset, but the manuscript only says 'Our code will be released later' and gives no data URL, source-video identifiers, or release timeline. Without the data or at least a detailed datasheet and split metadata, the split-hygiene and evaluation-protocol concerns above cannot be checked by readers. For a dataset paper, releasing the dataset, along with annotation guidelines and source-video-level metadata, should be a condition of publication.
minor comments (6)
  1. [Section VIII (Conclusion)] The sentence 'establish a The model contains' is truncated and should be completed.
  2. [Equation (14)] Equation (14) contains a stray closing bracket: it reads 'ReLU(Convλ(Ĥgλ))]'.
  3. [Figure 8] The horizontal axis of Figure 8 is labeled np, although the prompt length is denoted lp in Section V.E.
  4. [References] Reference [19] duplicates reference [15] (the same grounded multimodal NER paper appears twice); please deduplicate.
  5. [Equations (16)-(17)] The text uses 'Blinear' where 'Bilinear' is intended.
  6. [Section III (Step 6)] A single Cohen's Kappa of 78.18% is reported without specifying the annotation unit (document, mention, chain, or relation) or per-task agreement; please report per-task inter-annotator agreement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: M3D is a human-annotated dataset and the reported F1 scores are measured against held-out test splits, with model components ablated rather than derived from the targets they predict.

full rationale

The paper's central claim is empirical: it constructs a new multimodal, multilingual, multitask dataset and reports benchmark F1 on held-out test data. No reported result is equivalent to a fitted input by construction. The dataset is built from crawled videos, ASR-generated and manually corrected subtitles, sampled frames, and human annotations under explicit guidelines (Section III, Section IV); none of the four target tasks' labels enter the construction rule for the input features. The proposed model (Sections V-D and V-E) is a concrete architecture whose components are evaluated by ablation (Table IV) against external baselines (MDocRE-HN, Video-LLaMA-2, Video-ChatGPT) on the same split, which is standard benchmark practice rather than a self-referential derivation. The paper's self-citations ([5], [6], [7], [31], [62]) appear in related-work and error-analysis contexts and do not carry the argument: no uniqueness theorem, ansatz, or fitted parameter is imported from the authors' prior work to force the dataset or the architecture. Two statistical risks identified in review, namely possible clip-level overlap across the 8:1:1 split and the use of golden entity/chain pairs when evaluating CR/RE, are validity concerns and should be checked by the authors, but neither makes a prediction equal to its input by definition. Consequently no circular step can be exhibited under the required standard.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on data collection choices and evaluation protocol, not on physical parameters. The hand-chosen quantities listed above affect dataset difficulty and benchmark numbers. No new physical or theoretical entities are introduced.

free parameters (5)
  • Total-loss weights alpha_gamma = not reported
    Equation (24) sums task losses with tunable weights; the values are not given, so the exact training objective cannot be reproduced.
  • Missing modality prompt length l_p = not stated; Figure 8 sweeps 24, 48, 64, 96, 120
    The prompt length is a hand-chosen hyperparameter and Section VII-C shows average F1 varies with it, but the final value is not reported.
  • Frame sampling rate = 1 frame per 24 frames
    Section III Step 3 fixes the sampling rate, which defines which visual targets can be annotated and affects visual grounding difficulty.
  • Video clip duration = one to two minutes
    Section III Step 2 fixes clip length, which bounds document length and defines the document-level nature of the benchmark.
  • Missing-modality corruption split = three equal parts: full, missing text, missing visual
    Section VI-C describes this artificial corruption scheme, which determines how MMCM is evaluated, but the exact mechanism for removing a modality is not specified.
assumptions (5)
  • domain assumption Automatic subtitle generation with manual correction yields transcripts accurate enough for IE annotation and text-video alignment.
    Section III Step 4 uses a subtitle generation tool and says errors are checked and modified, but no quantitative accuracy measure is reported. All text annotations and visual grounding depend on transcript fidelity.
  • domain assumption Clips from the same source video do not cross train/dev/test splits.
    Section III Step 2 creates clips, and the statistics paragraph applies an 8:1:1 split without stating any per-video grouping, so leakage is possible.
  • domain assumption Sampling one frame every 24 frames captures the visual targets relevant to entity mentions.
    Section III Step 3 fixes the sampling rate; entities appearing only between sampled frames cannot be grounded, which changes the visual grounding task definition.
  • domain assumption Biography videos show a visible image of an entity when that entity is mentioned in the narration.
    Section III Step 1 justifies the video domain by saying corresponding images are usually shown when a specific entity is mentioned; visual grounding annotation depends on this heuristic holding.
  • domain assumption An entity chain can be treated as a coreference relation, and relations can be annotated at the chain level.
    Section IV-B defines entity chains this way; this is a modeling choice that shapes the relation extraction and coreference tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of M$^{3}$D: A Multimodal, Multilingual and Multitask Dataset for Grounded Document-level Information Extraction." pith.science (2026). https://pith.science/paper/P5CB4QID

@misc{pith2026241204026,
  author       = {Pith},
  title        = {Pith review of: M$^3$D: A Multimodal, Multilingual and Multitask Dataset for Grounded Document-level Information Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P5CB4QID}},
  note         = {Machine review of arXiv:2412.04026}
}
abstract

Multimodal information extraction (IE) tasks have attracted increasing attention because many studies have shown that multimodal information benefits text information extraction. However, existing multimodal IE datasets mainly focus on sentence-level image-facilitated IE in English text, and pay little attention to video-based multimodal IE and fine-grained visual grounding. Therefore, in order to promote the development of multimodal IE, we constructed a multimodal multilingual multitask dataset, named M$^{3}$D, which has the following features: (1) It contains paired document-level text and video to enrich multimodal information; (2) It supports two widely-used languages, namely English and Chinese; (3) It includes more multimodal IE tasks such as entity recognition, entity chain extraction, relation extraction and visual grounding. In addition, our dataset introduces an unexplored theme, i.e., biography, enriching the domains of multimodal IE resources. To establish a benchmark for our dataset, we propose an innovative hierarchical multimodal IE model. This model effectively leverages and integrates multimodal information through a Denoised Feature Fusion Module (DFFM). Furthermore, in non-ideal scenarios, modal information is often incomplete. Thus, we designed a Missing Modality Construction Module (MMCM) to alleviate the issues caused by missing modalities. Our model achieved an average performance of 53.80% and 53.77% on four tasks in English and Chinese datasets, respectively, which set a reasonable standard for subsequent research. In addition, we conducted more analytical experiments to verify the effectiveness of our proposed module. We believe that our work can promote the development of the field of multimodal IE.

Figures

Figures reproduced from arXiv: 2412.04026 by the authors.

Figure 1
Figure 1. A sample in the M3D dataset. The left part is the input example, and the right part is the output example of four tasks. usually includes the following widely-studied tasks, such as named entity recognition (NER) [1], coreference resolution (CR) [2], relation extraction (RE) [3] and event extraction (EE) [4]. Various methods have been investigated for solving these tasks. For example, sequence-to-sequence (Seq2Seq) … view at source ↗
Figure 2
Figure 2. The overall construction process of the M [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Quantity statistics for each relation type. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Some examples of visual grounding annotations. The red entity in the text corresponds to the visual target in the image. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The overall architecture of our model. The dashed line indicates execution when the modality is missing. For a detailed introduction to DFFM and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The architecture diagram of our denoised feature fusion module (DFFM). We take image integration into text as an example. QQ [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The architecture diagram of our missing modality construction module [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The effect of prompt length on the model. The horizontal axis and vertical axis represent the prompt length [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: The effect of missing modality ratio on the model. The horizontal axis and vertical axis represent the missing modality ratio and the F1 value of the [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 49 canonical work pages

  1. [1]

    DiffusionNER: Boundary diffusion for named entity recognition,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “DiffusionNER: Boundary diffusion for named entity recognition,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 3875–3890

    work page 2023

  2. [2]

    Dual cache for long document neural coreference resolution,

    Q. Guo, X. Hu, Y . Zhang, X. Qiu, and Z. Zhang, “Dual cache for long document neural coreference resolution,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 15 272–15 285

    work page 2023

  3. [3]

    An autoregressive text-to-graph framework for joint entity and relation extraction,

    U. Zaratiana, N. Tomeh, P. Holat, and T. Charnois, “An autoregressive text-to-graph framework for joint entity and relation extraction,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2024, pp. 19 477–19 487

    work page 2024

  4. [4]

    Event extraction as question generation and answering,

    D. Lu, S. Ran, J. Tetreault, and A. Jaimes, “Event extraction as question generation and answering,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2023, pp. 1666–1688

    work page 2023

  5. [5]

    Rethinking boundaries: End-to-end recognition of discontinuous mentions with pointer networks,

    H. Fei, D. Ji, B. Li, Y . Liu, Y . Ren, and F. Li, “Rethinking boundaries: End-to-end recognition of discontinuous mentions with pointer networks,” in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 14, 2021, pp. 12 785–12 793

    work page 2021

  6. [6]

    A span-based model for joint overlapped and discontinuous named entity recognition,

    F. Li, Z. Lin, M. Zhang, and D. Ji, “A span-based model for joint overlapped and discontinuous named entity recognition,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021, pp. 4814–4828

    work page 2021

  7. [7]

    Unified named entity recognition as word-word relation classification,

    J. Li, H. Fei, J. Liu, S. Wu, M. Zhang, C. Teng, D. Ji, and F. Li, “Unified named entity recognition as word-word relation classification,” in proceedings of the AAAI conference on artificial intelligence , 2022, pp. 10 965–10 973

    work page 2022

  8. [8]

    Knowledge enhanced coreference resolution via gated attention,

    K. He, B. Mao, X. Zhou, Y . Li, T. Gong, C. Li, and J. Wu, “Knowledge enhanced coreference resolution via gated attention,” in 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . IEEE, 2022, pp. 2287–2293

    work page 2022

Show all 62 references
  1. [9]

    Double graph based reasoning for document-level relation extraction,

    S. Zeng, R. Xu, B. Chang, and L. Li, “Double graph based reasoning for document-level relation extraction,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 1630–1640

    2020

  2. [10]

    Coreference resolution without span representations,

    Y . Kirstain, O. Ram, and O. Levy, “Coreference resolution without span representations,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pape...

    2021

  3. [11]

    A sequence-to-sequence approach for document-level relation extraction,

    J. Giorgi, G. Bader, and B. Wang, “A sequence-to-sequence approach for document-level relation extraction,” in Proceedings of the 21st Workshop on Biomedical Language Processing , 2022, pp. 10–25

    2022

  4. [12]

    Visual attention model for name tagging in multimodal social media,

    D. Lu, L. Neves, V . Carvalho, N. Zhang, and H. Ji, “Visual attention model for name tagging in multimodal social media,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 1990–1999

    2018

  5. [13]

    Adaptive co-attention network for named entity recognition in tweets,

    Q. Zhang, J. Fu, X. Liu, and X. Huang, “Adaptive co-attention network for named entity recognition in tweets,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2018, pp. 5674–5681

    2018

  6. [14]

    A large-scale chinese multimodal ner dataset with speech clues,

    D. Sui, Z. Tian, Y . Chen, K. Liu, and J. Zhao, “A large-scale chinese multimodal ner dataset with speech clues,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processi...

    2021

  7. [16]

    Who are you referring to? coreference resolution in image narrations,

    A. Goel, B. Fernando, F. Keller, and H. Bilen, “Who are you referring to? coreference resolution in image narrations,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 247–15 258

    2023

  8. [17]

    Mnre: A challenge multimodal dataset for neural relation extraction with visual evidence in social media posts,

    C. Zheng, Z. Wu, J. Feng, Z. Fu, and Y . Cai, “Mnre: A challenge multimodal dataset for neural relation extraction with visual evidence in social media posts,” in 2021 IEEE International Conference on Multimedia and Expo (ICME) , 2021, pp. 1–6

    2021

  9. [18]

    A hierarchical network for multimodal document-level relation extraction,

    L. Kong, J. Wang, Z. Ma, Q. Zhou, J. Zhang, L. He, and J. Chen, “A hierarchical network for multimodal document-level relation extraction,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 16, 2024, pp. 18 408–18 416

    2024

  10. [19]

    Grounded multimodal named entity recognition on social media,

    J. Yu, Z. Li, J. Wang, and R. Xia, “Grounded multimodal named entity recognition on social media,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 9141–9154

    2023

  11. [20]

    Semi-supervised multimodal coreference resolution in image narrations,

    A. Goel, B. Fernando, F. Keller, and H. Bilen, “Semi-supervised multimodal coreference resolution in image narrations,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 11 067–11 081

    2023

  12. [21]

    Joint multimodal entity-relation extraction based on edge-enhanced graph alignment network and word- pair relation tagging,

    L. Yuan, Y . Cai, J. Wang, and Q. Li, “Joint multimodal entity-relation extraction based on edge-enhanced graph alignment network and word- pair relation tagging,” in Proceedings of the AAAI conference on artificial intelligence, 2023, pp. 11 051–11 059

    2023

  13. [22]

    Multimodal relation extraction with efficient graph alignment,

    C. Zheng, J. Feng, Z. Fu, Y . Cai, Q. Li, and T. Wang, “Multimodal relation extraction with efficient graph alignment,” in Proceedings of the 29th ACM international conference on multimedia , 2021, pp. 5298–5306

    2021

  14. [23]

    Docred: A large-scale document-level relation extraction dataset,

    Y . Yao, D. Ye, P. Li, X. Han, Y . Lin, Z. Liu, Z. Liu, L. Huang, J. Zhou, and M. Sun, “Docred: A large-scale document-level relation extraction dataset,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019, pp. 764–777

    2019

  15. [24]

    Improving multimodal named entity recognition via entity span detection with unified multimodal transformer,

    J. Yu, J. Jiang, L. Yang, and R. Xia, “Improving multimodal named entity recognition via entity span detection with unified multimodal transformer,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 3342–3352

    2020

  16. [25]

    A span-based multimodal variational autoencoder for semi-supervised multimodal named entity recognition,

    B. Zhou, Y . Zhang, K. Song, W. Guo, G. Zhao, H. Wang, and X. Yuan, “A span-based multimodal variational autoencoder for semi-supervised multimodal named entity recognition,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. ...

    2022

  17. [26]

    Entity- level interaction via heterogeneous graph for multimodal named entity recognition,

    G. Zhao, G. Dong, Y . Shi, H. Yan, W. Xu, and S. Li, “Entity- level interaction via heterogeneous graph for multimodal named entity recognition,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 6345–6350

    2022

  18. [27]

    Prompt- ing chatgpt in mner: Enhanced multimodal named entity recognition with auxiliary refined knowledge,

    J. Li, H. Li, Z. Pan, D. Sun, J. Wang, W. Zhang, and G. Pan, “Prompt- ing chatgpt in mner: Enhanced multimodal named entity recognition with auxiliary refined knowledge,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , 2023, pp. 2787–2802. 14

    2023

  19. [28]

    Gravl-bert: graphical visual-linguistic representations for multimodal coreference resolution,

    D. Guo, A. Gupta, S. Agarwal, J.-Y . Kao, S. Gao, A. Biswas, C.-W. Lin, T. Chung, and M. Bansal, “Gravl-bert: graphical visual-linguistic representations for multimodal coreference resolution,” in Proceedings of the 29th International Conference on Computational Linguistics , ...

    2022

  20. [29]

    Good visual guidance make a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction,

    X. Chen, N. Zhang, L. Li, Y . Yao, S. Deng, C. Tan, F. Huang, L. Si, and H. Chen, “Good visual guidance make a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction,” in Findings of the Association for Computational Linguistics: NAACL 2022 ...

    2022

  21. [30]

    Rethinking multimodal entity and relation extraction from a translation point of view,

    C. Zheng, J. Feng, Y . Cai, X. Wei, and Q. Li, “Rethinking multimodal entity and relation extraction from a translation point of view,” in Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023, pp. 6810–6824

    2023

  22. [31]

    Information screening whilst exploiting! multimodal relation extraction with feature denoising and multimodal topic modeling,

    S. Wu, H. Fei, Y . Cao, L. Bing, and T.-S. Chua, “Information screening whilst exploiting! multimodal relation extraction with feature denoising and multimodal topic modeling,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2023

  23. [32]

    Transvg: End-to-end visual grounding with transformers,

    J. Deng, Z. Yang, T. Chen, W. Zhou, and H. Li, “Transvg: End-to-end visual grounding with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1769–1779

    2021

  24. [33]

    Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding,

    J. Ye, J. Tian, M. Yan, X. Yang, X. Wang, J. Zhang, L. He, and X. Lin, “Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022...

    2022

  25. [34]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 779– 788

    2016

  26. [35]

    Ssd: Single shot multibox detector,

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y . Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14 , 2016, pp. 21–37

    2016

  27. [36]

    Ref-nms: Breaking proposal bottlenecks in two-stage referring expression grounding,

    L. Chen, W. Ma, J. Xiao, H. Zhang, and S.-F. Chang, “Ref-nms: Breaking proposal bottlenecks in two-stage referring expression grounding,” in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 2, 2021, pp. 1036–1044

    2021

  28. [37]

    Missing modalities imputation via cascaded residual autoencoder,

    L. Tran, X. Liu, J. Zhou, and R. Jin, “Missing modalities imputation via cascaded residual autoencoder,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR , 2017, pp. 4971–4980

    2017

  29. [38]

    Lrmm: Learning to recommend with missing modalities,

    C. Wang, M. Niepert, and H. Li, “Lrmm: Learning to recommend with missing modalities,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018, pp. 3360–3370

    2018

  30. [39]

    Dealing with missing modalities in the visual question answer-difference prediction task through knowledge distillation,

    J. W. Cho, D.-J. Kim, J. Choi, Y . Jung, and I. S. Kweon, “Dealing with missing modalities in the visual question answer-difference prediction task through knowledge distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp....

    2021

  31. [40]

    A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities,

    M. Li, D. Yang, Y . Lei, S. Wang, S. Wang, L. Su, K. Yang, Y . Wang, M. Sun, and L. Zhang, “A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, n...

    2024

  32. [41]

    Missing modality imagination network for emotion recognition with uncertain missing modalities,

    J. Zhao, R. Li, and Q. Jin, “Missing modality imagination network for emotion recognition with uncertain missing modalities,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Langu...

    2021

  33. [42]

    Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities,

    J. Zeng, J. Zhou, and T. Liu, “Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 2924–2934

    2022

  34. [43]

    Multimodal prompting with missing modalities for visual recognition,

    Y .-L. Lee, Y .-H. Tsai, W.-C. Chiu, and C.-Y . Lee, “Multimodal prompting with missing modalities for visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 943–14 952

    2023

  35. [44]

    Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,

    Z. Guo, T. Jin, and Z. Zhao, “Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2024, pp. 1726– 1736

    2024

  36. [45]

    Longformer: The long-document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150 , 2020

    2004 arXiv

  37. [46]

    Visual transformers: Token- based image representation and processing for computer vision,

    B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, “Visual transformers: Token- based image representation and processing for computer vision,” arXiv preprint arXiv:2006.03677, 2020

    2006 arXiv

  38. [47]

    A primer in bertology: What we know about how bert works,

    A. Rogers, O. Kovaleva, and A. Rumshisky, “A primer in bertology: What we know about how bert works,” Transactions of the Association for Computational Linguistics , vol. 8, pp. 842–866, 2021

    2021

  39. [48]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , vo...

    2019

  40. [49]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning...

    2021

  41. [50]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in 2nd International Conference on Learning Representations , 2014, pp. 1–14

    2014

  42. [51]

    Exploring universal intrinsic task subspace for few-shot learning via prompt tuning,

    Y . Qin, X. Wang, Y . Su, Y . Lin, N. Ding, J. Yi, W. Chen, Z. Liu, J. Li, L. Hou, P. Li, M. Sun, and J. Zhou, “Exploring universal intrinsic task subspace for few-shot learning via prompt tuning,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 32, pp. ...

    2024

  43. [52]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Proceedings of the Advances in neural information processing systems , vol. 30, pp. 5998–6008, 2017

    2017

  44. [53]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  45. [54]

    Conditional random fields: Probabilistic models for segmenting and labeling sequence data,

    J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in Proceedings of the Eighteenth International Conference on Machine Learning, 2001, pp. 282–289

    2001

  46. [55]

    Visualbert: A simple and performant baseline for vision and language,

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  47. [56]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural information processing systems , vol. 32, 2019

    2019

  48. [57]

    Vivit: A video vision transformer,

    A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lu ˇci´c, and C. Schmid, “Vivit: A video vision transformer,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 6836–6846

    2021

  49. [58]

    Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,” Advances in neural information processing systems , vol. 35, pp. 10 078– 10 093, 2022

    2022

  50. [59]

    Video-llama: An instruction-tuned audio- visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio- visual language model for video understanding,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2023, pp. 543–553

    2023

  51. [60]

    Video-chatgpt: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. A. Rasheed, S. Khan, and F. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thai...

    2024

  52. [61]

    Parallel data helps neural entity coreference resolution,

    G. Tang and C. Hardmeier, “Parallel data helps neural entity coreference resolution,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 3162–3171

    2023

  53. [62]

    Toe: A grid-tagging discontinuous ner model enhanced by embedding tag/word relations and more fine-grained tags,

    J. Liu, D. Ji, J. Li, D. Xie, C. Teng, L. Zhao, and F. Li, “Toe: A grid-tagging discontinuous ner model enhanced by embedding tag/word relations and more fine-grained tags,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 177–187, 2022

    2022

  54. [63]

    A fast and accurate one-stage approach to visual grounding,

    Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo, “A fast and accurate one-stage approach to visual grounding,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 4683–4693

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.