Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

BERT-VQA: Visual Question Answering on Plots

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simple concatenation baseline beats a VisualBERT cross-modal transformer on yes/no plot question answering.

desk verdict A confounded negative result: the VisualBERT-vs-baseline gap on PlotQA cannot be attributed to the fusion module, so the paper's central claim is unsupported. read the letter →

arxiv 2508.13184 v1 pith:AJRW4EDJ submitted 2025-08-14 cs.LG cs.CV

Tai Vu , Robert Yang This is my paper
classification cs.LGcs.CV
keywords visualquestionansweringPlotQABERTcross-modalfusionunderstandingLSTMResNet101yes/noquestions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a sophisticated cross-modal transformer--VisualBERT--is necessary to answer yes/no questions about plots, or whether a simple baseline that concatenates an LSTM's question embedding with a ResNet image embedding is enough. On a balanced subset of the PlotQA dataset, the simple baseline reaches 83.66% validation accuracy, beating the VisualBERT-based model at 80.70% and the VisualBERT variant with added whole-image features at 58.15%. The authors take this as evidence that the cross-modality alignment module in VisualBERT is not essential for plot question answering, and that concatenation-based fusion can outperform a much larger attention-based model. The result matters for practical chart understanding because it suggests expensive transformer fusion may be unnecessary for some multimodal reasoning tasks.

What carries the argument

The load-bearing contrast is the fusion mechanism. The baseline concatenates a fixed-length LSTM question embedding with a fixed-length ResNet 101 image embedding and classifies with a shallow network; the proposed models replace that concatenation with a VisualBERT-style stack of Transformer layers that jointly processes question tokens and Faster R-CNN region embeddings, so attention can align text phrases with image regions. A second variant appends whole-image ResNet 101 features to the Transformer output before classification, testing whether global image context helps.

What would settle it

A controlled ablation that fixes the text encoder and image encoder, and swaps only the fusion layer between concatenation and the VisualBERT Transformer stack, with matched epochs and hyperparameters; if concatenation no longer beats the Transformer on the same balanced PlotQA validation set, the claim that the fusion mechanism drives the gap is falsified.

Watch

Extended reading notes

Core claim

The paper's central empirical finding is that, on a balanced yes/no subset of PlotQA, the cross-modality Transformer fusion in VisualBERT is not needed and appears to hurt. The baseline--an LSTM over GloVe embeddings concatenated with a ResNet 101 image embedding and passed to a shallow classifier--reaches 83.66% validation accuracy, while the modified VisualBERT with a Faster R-CNN/ResNet 101 region encoder reaches 80.70%, and the same model with whole-image features appended via joint fusion falls to 58.15%. The authors state that this disproves their core hypothesis that VisualBERT's cross-modality module is essential for aligning plot components with question phrases, and conclude that f

Load-bearing premise

The conclusion assumes that the accuracy gap is caused by the fusion mechanism rather than by the other differences between the models, such as LSTM with GloVe versus BERT tokenization, 8 epochs versus 100 epochs, and hyperparameters tuned separately for each model.

Editorial extensions

If this is right

  • Heavy cross-modal transformer fusion is not required for yes/no plot question answering; the concatenation baseline is both simpler and more accurate on this benchmark.
  • Appending whole-image ResNet 101 features to VisualBERT output is sharply harmful, dropping accuracy from 80.70% to 58.15%, so global image summaries can interfere with region-text alignment.
  • Because the models' false-positive and false-negative patterns mostly overlap, fusion mechanism choice affects overall accuracy more than it changes the model's yes/no biases.
  • Even the best model leaves roughly 16% of balanced yes/no questions wrong, so architecture alone does not solve plot reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run a matched ablation, so its causal conclusion that the fusion module drives the gap is an inference, not a demonstrated fact; a cleaner test would fix text encoding and vary only fusion.
  • The near-chance joint-fusion result hints that simply concatenating global image features into a region-based transformer can swamp the classifier; a learned gating mechanism might recover performance.
  • The yes/no-only scope leaves open whether transformer fusion becomes valuable for numerical or open-ended PlotQA answers, which require different output heads and more deliberate alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents BERT-VQA, a VisualBERT-based architecture for yes/no visual question answering on plots from PlotQA, together with two variants (Modified VisualBERT and Modified VisualBERT with Joint Fusion). These are compared to a baseline LSTM+CNN+shallow classifier. On the validation set (Table 1), the baseline achieves accuracy 0.8366, Modified VisualBERT 0.8070, and the joint-fusion variant 0.5815. The authors interpret this ordering as disproving their hypothesis that VisualBERT's cross-modality module is essential for aligning plot components with question phrases, and conclude that concatenation-based fusion is preferable. The paper also includes a qualitative error analysis comparing false positives and false negatives between the baseline and Modified VisualBERT.

Significance. If the comparison were valid, this negative result would be a useful data point for multimodal fusion choices in plot QA, which is an under-explored domain. The paper is honest about its limitations and the error analysis makes a reasonable attempt to characterize where the models fail. However, the central empirical claim is not supported by the reported experiments because the main comparison is confounded by multiple architectural and training differences. The paper provides no code, test-set results, or error bars, so the contribution is currently a preliminary project report rather than a rigorous scientific finding.

major comments (4)
  1. [§4.1–4.3, §5, §6.1, Table 1] The headline comparison (Baseline 0.8366 vs Modified VisualBERT 0.8070) does not isolate the cross-modality module. Section 5 admits the baseline uses LSTM with GloVe and word tokenization, while VisualBERT uses BERT subword tokenization and a Transformer. Section 6.1 states the baseline was trained for 8 epochs and the VisualBERT models for 100 epochs. Section 4.2 shows the visual pipelines also differ: the baseline embeds the whole image with ResNet-101, while Modified VisualBERT uses Faster R-CNN region proposals fed into a pretrained ResNet-101 from LXMERT. Hyperparameters were 'optimized for each model type' and are not reported. Any one of these differences could account for the accuracy gap, so the abstract's 'disproved our core hypothesis' is unsupported.
  2. [Table 1, §3] Only validation-set metrics are reported, with no error bars and no test-set evaluation. Section 3 defines a test set, but it is never used. The only run-to-run variation reported (baseline 82.4–82.8% at 1 epoch, §6.1) does not cover the final models. Without multiple seeds or confidence intervals, the observed gap between 0.8366 and 0.8070 may not be statistically meaningful, and the joint-fusion drop to 0.5815 is also a single uncharacterized run. This prevents the paper from supporting a causal attribution to the fusion mechanism.
  3. [§5 vs §4.1–4.3] The paper claims 'the image preprocessing pipeline, image feature extractors, and output shallow classifiers were exactly the same' (§5), but this is contradicted by the model descriptions. The baseline uses ResNet-101 on the entire image, while Modified VisualBERT uses Faster R-CNN region proposals plus a ResNet-101 and the joint-fusion variant adds a separate whole-image CNN stream. This internal inconsistency needs to be resolved, because the fairness claim is central to the experimental design.
  4. [§6.2] The error analysis is qualitative and based on very small counts (e.g., 11 vs 13 false positives) drawn from an unspecified subset; it does not report totals over the validation set. The statement that the two fusion mechanisms 'were more similar than different' is not supported by any quantitative measure, and the section contains the unfinished placeholder 'Hypothesis?'. This does not bear on the main accuracy comparison, but it is used as supporting evidence for behavioral similarity and needs proper grounding.
minor comments (5)
  1. [§4.1, §6.2] The manuscript contains unfinished placeholders: 'TODO: source needed' in §4.1 and 'Hypothesis?' in §6.2. These must be removed or completed before submission.
  2. [Table 1] The numerical formatting in Table 1 is inconsistent (e.g., '0 .8300', '0 .5900'), which harms readability. Use standard decimal notation and align columns.
  3. [§2] There are grammatical errors such as 'This systems leverages a the cross-modality encoder'; please proofread.
  4. [Known Limitations] The text 'an specific aggregation' should be 'a specific aggregation'; there are also several spacing issues in numbers such as '1, 119, 713'.
  5. [General] No code, model configuration files, or data-splitting scripts are provided. Releasing these would significantly improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical comparison, not a result forced by construction or by self-citation.

full rationale

The paper contains no derivation chain in which an output is defined in terms of an input, no fitted parameter is relabeled as a prediction, and no load-bearing self-citation. The central claim—that the cross-modality module in VisualBERT is not essential—is inferred from a validation-set accuracy comparison (Table 1: Baseline 0.8366 vs Modified VisualBERT 0.8070, joint fusion 0.5815). This is an empirical outcome, not a quantity constructed from the model definition. The cited prior work (VisualBERT, LXMERT, PlotQA, etc.) is external to the authors, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The manuscript does contain admissions that threaten the validity of the comparison: Section 5 states the baseline 'uses LSTM in lieu of BERT to process the text' with GloVe embeddings and word tokenizing, while VisualBERT uses BERT tokenization/embeddings, and Section 6.1 reports 8 training epochs for the baseline versus 100 for VisualBERT with hyperparameters 'optimized for each model type.' These are confounding-factor concerns, not circularity: the accuracy gap is not algebraically or definitionally equal to the absence of cross-modality value. The paper also contains internal notes ('TODO: source needed' in Section 4.1 and 'Hypothesis?' in Section 6.2) and a Known Limitations section, but none of these assert a circular dependency. Because no step in the paper reduces to its own inputs or to a self-citation chain, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. Its central claim rests on a set of modeling and evaluation assumptions: representativeness of the subsample, isolation of the fusion mechanism, transferability of pretrained visual features, and reliance on validation metrics. The fusion-isolation assumption is explicitly violated by the differing text encoders and training budgets, which is the core weakness.

free parameters (3)
  • Per-model hyperparameters
    Batch size, dropout, and learning rate were 'optimized for each model type' (Section 5), but exact values are not reported, so the comparison is not controlled.
  • Training epochs = baseline 8, VisualBERT 100
    Different training durations introduce a confound in the comparison (Section 6.1).
  • Dataset subset size = 10k/5k/5k images; 87,962/43,770/43,918 questions
    Hand-selected subset of PlotQA Yes/No questions without specified sampling seed or procedure (Section 3).
assumptions (4)
  • domain assumption Subset of PlotQA Yes/No questions is representative of plot VQA
    The paper draws conclusions about plot VQA from a Yes/No-only subset sampled without a seed (Section 3).
  • domain assumption Performance difference is attributable to the fusion module
    The paper claims VisualBERT fusion is detrimental, but the baseline uses a different text encoder, tokenizer, and epochs, so the comparison does not isolate fusion (Section 5).
  • domain assumption Pretrained ResNet 101 from LXMERT and Faster R-CNN transfer to plot images
    The visual encoder is pretrained on natural scenes (MS COCO, Visual Genome) and used without fine-tuning on plots (Sections 4.2 and 7).
  • domain assumption Validation set results are sufficient to support the conclusion
    Only validation metrics are reported in Table 1; no test set results are given (Section 6.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BERT-VQA: Visual Question Answering on Plots." pith.science (2026). https://pith.science/paper/AJRW4EDJ

@misc{pith2026250813184,
  author       = {Pith},
  title        = {Pith review of: BERT-VQA: Visual Question Answering on Plots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AJRW4EDJ}},
  note         = {Machine review of arXiv:2508.13184}
}
read the original abstract

Visual question answering has been an exciting challenge in the field of natural language understanding, as it requires deep learning models to exchange information from both vision and language domains. In this project, we aim to tackle a subtask of this problem, namely visual question answering on plots. To achieve this, we developed BERT-VQA, a VisualBERT-based model architecture with a pretrained ResNet 101 image encoder, along with a potential addition of joint fusion. We trained and evaluated this model against a baseline that consisted of a LSTM, a CNN, and a shallow classifier. The final outcome disproved our core hypothesis that the cross-modality module in VisualBERT is essential in aligning plot components with question phrases. Therefore, our work provided valuable insights into the difficulty of the plot question answering challenge as well as the appropriateness of different model architectures in solving this problem.

Figures

Figures reproduced from arXiv: 2508.13184 by the authors.

Figure 1
Figure 1. Example dot-line plot [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example line plot [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example bar plot. It includes questions that can have one of three answer types: “Yes/No”, “fixed vocabulary”, and “open vocabulary”. For this project, we are go￾ing to use only the “Yes/No” questions to test our hypothesis. There are 1, 119, 713 “Yes/No” questions in the dataset, of which 784, 115 are in the training set, 167, 871 are in the validation set, and 167, 727 are in the test set. Due to compu￾tational co… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Model architecture of the baseline model [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: Example plot with all three different types of [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Model architecture of the modified Visual [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Model architecture of the modified Visual [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: False positives comparison between the base [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Example 25’s question is: "Does the educa [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: False negatives comparison between the baseline and the modified VisualBERT. We notice from [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Amplifying Emotional Signals: Data-Efficient Deep Learning for Robust Speech Emotion Recognition

    eess.AS 2025-08 reject novelty 3.0 of 10

    A pretrained ResNet34 with augmentation reaches 66.7% accuracy on a combined RAVDESS/SAVEE emotion set, but only on a validation split, so the claimed new benchmark is unverified.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hedi Ben-Younes, R \'e mi Cadene, Matthieu Cord, and Nicolas Thome. 2017. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2612--2620

    work page 2017

  4. [4]

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325

    arXiv 2015

  5. [5]

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104--120. Springer

    work page 2020

  6. [6]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778

    2016

  7. [7]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32--73

    work page 2017

  8. [8]

    Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019 a . Relation-aware graph attention network for visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10313--10322

    work page 2019

Show all 18 references
  1. [9]

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019 b . Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557

    2019 arXiv

  2. [10]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740--755. Springer

    2014

  3. [11]

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527--1536

    2020

  4. [12]

    Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. 2018. Out of the box: Reasoning with graph convolution nets for factual visual question answering. Advances in neural information processing systems, 31

    2018

  5. [13]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. http://www.aclweb.org/anthology/D14-1162 Glove: Global vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

    2014

  6. [14]

    Sashank Reddi, Barnabas Poczos, and Alex Smola. 2015. https://doi.org/10.1609/aaai.v29i1.9576 Doubly robust covariate shift correction . Proceedings of the AAAI Conference on Artificial Intelligence, 29(1)

    2015 doi

  7. [15]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28

    2015

  8. [16]

    Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490

    2019 arXiv

  9. [17]

    Damien Teney, Peter Anderson, Xiaodong He, and Anton Van Den Hengel. 2018. Tips and tricks for visual question answering: Learnings from the 2017 challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4223--4232

    2018

  10. [18]

    Damien Teney, Lingqiao Liu, and Anton van Den Hengel. 2017. Graph-structured representations for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1--9

    2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.