RECORD
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction
Computation and Language · Submitted 4 May 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. | gap 0.83 | 0.05 | 0.03 | asserted 1.00 |
| s2 | In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. | contribution 0.82 | 0.94 | 0.14 | asserted 1.00 |
| s3 | The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. | method 0.84 | 0.67 | 0.10 | asserted 1.00 |
| s4 | In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. | method 0.92 | 0.95 | 0.04 | asserted 1.00 |
| s5 | FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size. | result 1.00 | 0.95 | 0.75 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
Human review
References
How the literature uses it
7 arXiv papers cite this in 12 sentences
Papers citing it · 6 sentences without a confident label
Cites it as background 5 papers
-
Previous studies mainly focused on combining the text and layout with language model pre-training [ref] or encoding the local layout information by constructing super-tokens [ref] .
RE$^2$: Region-Aware Relation Extraction from Visually Rich Documents · Introduction · 0.92
-
Further advances introduced geometric pre-training [ref] , graph contrastive learning [ref] , and unified frameworks for simultaneous text detection and classification [ref] .
Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments · Related Work · 0.97
-
Other methods [ref] attempt to model entity relationships based on graph convolutional networks, and they are quite similar to the language-based methods.
HIP: Hierarchical Point Modeling and Pre-training for Visual Information Extraction · Entity Extraction · 0.83
What it cites · 4 sentences without a confident label
Uses the method or tool of 2 papers
-
Furthermore, instead of using multiple self-supervised objectives for each individual modality, we introduce graph contrastive learning [ref] to learn multimodal embeddings jointly.
An Empirical Study of Graph Contrastive Learning · Introduction · 0.90
-
We adopt the standard normalized temperature-scaled cross entropy (NT-Xent) loss formulation [ref] with temperature 0.1 in all experiments.
Representation Learning with Contrastive Predictive Coding · Multimodal Graph Contrastive Learning · 1.00
Cites as background 7 papers
-
They include mask prediction, reconstruction, and matching for one or more modalities [ref] .
DiT: Self-supervised Pre-training for Document Image Transformer · Related Work · 0.96
-
They include mask prediction, reconstruction, and matching for one or more modalities [ref] .
Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning · Related Work · 0.96
-
They include mask prediction, reconstruction, and matching for one or more modalities [ref] .
Unified Pretraining Framework for Document Understanding · Related Work · 0.96
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.