Pith. sign in

REVIEW 2 major objections 5 minor 89 references

Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single lightweight module, IPRM, gets better at compositional visual reasoning by explicitly alternating iterative step-by-step computation with parallel attention, and that this combination outperforms…

desk verdict IPRM is a real and well-ablated reasoning module, but the video SOTA claim is overstated because IPRM gets object-level inputs on STAR/AGQA while the baselines see raw frames. read the letter →

arxiv 2411.13754 v1 pith:4DKKD54Y submitted 2024-11-20 cs.LG cs.AIcs.CV

Shantanu Jaiswal , Debaditya Roy , Basura Fernando , Cheston Tan This is my paper · ORCID
classification cs.LGcs.AIcs.CV
keywords visualquestionansweringcompositionalreasoningiterativecomputationparallelattentionworkingmemoryvideotransformerneuralmodule
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IPRM is a fully neural, fully differentiable reasoning module that the paper places on top of vision-language backbones to answer questions requiring several dependent steps. The paper's central claim is that two computational priors—iterative, step-by-step operation and parallel, independent operation—need each other: pure iteration overburdens memory when many things must be counted or tracked at once, while pure transformer-style parallel attention cannot compose operations sequentially. IPRM implements both through a working-memory bank of parallel operation states and result states updated over T weight-tied steps. The paper reports that IPRM outperforms prior task-specific video-QA models by about 5% on STAR and AGQAv2, improves over neurosymbolic and program-supervised models on CLEVRER-Humans and CLEVR-Humans, and also beats transformer attention blocks of comparable or larger size. A reader should care because, if correct, a single reusable reasoning block with step-by-step attention visualizations would replace task-specific modules and external supervision such as functional programs or bounding boxes on several benchmarks.

What carries the argument

The load-bearing object is a working-memory bank $\mathcal{M}=\{M_{op}, M_{res}\}$ holding $N_{op}$ parallel operation states and matching result states, updated for $T$ iterative steps. The three updates are Operation Formation, which retrieves language tokens into new latent operations $Z_{op,t}$ conditioned on $M_{op,t}$; Operation Execution, which retrieves visual tokens into $Z_{res,t}$ using attention keys modulated by $Z_{op,t}$ and $M_{res,t}$; and Operation Composition, which applies self-masked inter-operation attention over the new states plus a lookback window $W$ of old states to produce $M_{t+1}$. The composition step is what turns independent parallel slots into a coherent chain; without it, accuracy on CLEVR-Humans drops from roughly 82% to 74% at $N_{op}=6$, $T=9$ in the paper's ablations. Weight-tied parameters keep the module's parameter count constant in $T$ and $N_{op}$.

What would settle it

Take IPRM and the strongest raw-frame baselines, for example SeViLA-BLIP2 and MIST, and run all of them on STAR and AGQAv2 using identical raw-frame visual features while keeping IPRM's other hyperparameters fixed; if IPRM's average-accuracy advantage drops to about zero, the mechanism alone is not what drives the reported video gains, while if the advantage persists, the object-level input explanation is ruled out.

Watch

Extended reading notes

Core claim

The paper's discovery is that explicitly separating "what to compute" from "what has been computed," and maintaining both in parallel slots over repeated iterations, lets one network handle questions that require either long dependency chains or many simultaneous operations. At each of T steps, Operation Formation retrieves language tokens into a new set of latent operations using prior operation states as queries; Operation Execution retrieves visual tokens into results, with visual attention modulated by both new operations and prior result states; Operation Composition then lets each operation attend to the other operations and to a lookback window of memory states, before the memory is updated. The same module, weight-tied across steps, is applied to image and video benchmarks with different backbones. On STAR it reports 69.9% average accuracy versus 64.9% for SeViLA-BLIP2; on AGQAv2 it reports 60.4% overall accuracy versus 55.1% for Glance and Focus; on CLEVRER-Humans it raises finetuned per-question accuracy from 31.7% (ALOE) to 53.0%; and on CLEVR-Humans it reaches 63.8% zero-shot and 85.5% finetuned without programs or bounding-box pretraining. The internal attention maps can be read as a trace of what operation each parallel slot executed and where the model looked, which the paper uses to show correct intermediate reasoning and to diagnose errors.

Load-bearing premise

The central claim assumes that the 5% gains on STAR and AGQAv2 come from the iterative-parallel mechanism and not from the object bounding boxes, labels, poses, and human-object relations that IPRM receives on those video tasks while many baselines train on raw frames.

Editorial extensions

If this is right

  • IPRM adds about 5 percentage points of average accuracy over prior videoQA-specific methods on STAR (69.9%) and AGQAv2 (60.4%), with the largest gains on sequencing and prediction questions.
  • On image reasoning, IPRM outperforms MDETR by 3.9% zero-shot and 3.8% finetuned on CLEVR-Humans while using no functional programs and no bounding-box pretraining, and beats it by 3.6% on CLEVR-CoGenT out-of-domain generalization.
  • The module is sample-efficient: it exceeds MDETR's fully-finetuned CLEVR-Humans accuracy with only half the training data, and it keeps strong accuracy at longer program lengths where prior methods degrade.
  • IPRM can be dropped onto frozen CLIP visual backbones or LSTM/transformer language backbones, and in that setting it outperforms cross-attention and concatenation-attention blocks with fewer added parameters (5.2M versus 13.6-26.0M).
  • Because its language and visual attentions are recorded at every step, IPRM offers a step-by-step trace of what each parallel operation was doing, which the paper uses to verify correct reasoning and identify failure cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes beyond VQA, the same operation/result memory could be plugged into instruction-following or embodied-planning models, since the module only assumes a set of input tokens and a task description.
  • On video benchmarks, the strongest transferable test is an input-matched one: the paper's own Concat-Att and Cross-Att baselines share IPRM's object-level inputs and show smaller gaps, so the absolute gains over raw-frame baselines such as SeViLA and MIST may partly reflect the object annotations rather than the iterative-parallel design itself.
  • A natural follow-up probe is to check whether the learned parallel operations are stable and interpretable across random seeds; if they are, the attention visualizations could serve as a debugging tool for detecting dataset bias before it affects answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a fully differentiable neural module that interleaves iterative step-by-step reasoning with parallel operation states maintained in an explicit working memory. IPRM is designed as a drop-in block for vision-language backbones, and the authors evaluate it on video benchmarks (STAR, AGQAv2, CLEVRER-Humans) and image benchmarks (CLEVR-Humans, CLEVR-CoGenT, GQA, CLOSURE, NLVR). The paper reports accuracy improvements over prior task-specific methods and over standard concat/cross-attention transformer modules, together with ablations isolating the operation composition block, the number of parallel operations, and other hyperparameters. The authors also provide reasoning visualizations that trace both language and visual attention across iterative steps and parallel operations.

Significance. If the reported gains hold, IPRM is a useful contribution to compositional visual reasoning: it is lightweight (4.4M parameters), fully differentiable, and does not require functional programs or bounding-box supervision on the image benchmarks. The strongest evidence is the controlled comparison against Concat-Att and Cross-Att modules under identical object-level inputs, where IPRM improves by roughly 4 points on STAR, and the ablation showing that the operation composition block (OPC) contributes substantially at higher reasoning steps. Reproducibility is supported by detailed pseudocode and a promised source-code release. The principal weakness is that the video-domain 'state-of-the-art' claim over task-specific methods is confounded by unequal input representations; once that claim is appropriately restricted, the mechanism's direct benefit remains credible.

major comments (2)
  1. [3.1, Table 1, Appendix C.1, Table 7] The STAR and AGQA comparisons in Table 1 are not like-for-like. Per Appendix C.1, IPRM receives object-level inputs on these benchmarks: ground-truth or detector-predicted object bounding boxes, labels, human pose, and human-object relations, flattened with frame positional embeddings. The prior methods in Table 1 (All-in-One, Temp/ATP, MIST, InternVideo, SeViLA-BLIP2) are trained on raw video frames or frame-level features. The magnitude of this input-representation confound is visible in Table 7: on the STAR test set, IPRM with ground-truth vision reaches 79.6 average accuracy, while IPRM with predicted vision reaches 70.3, a 9.3-point swing that is larger than the claimed 5-point margin over SeViLA-BLIP2. Because the abstract and conclusion assert that IPRM 'notably outperforms prior task-specific methods' on video benchmarks, that claim is not supported by the present experiments. The valid, controlled comparison on STAR (Concat-Att-4L 65.3 and Cross-Att-4L 65.6 against IPRM 69.9) does support the mechanism, but the SOTA claim should be restricted to matched-input comparisons, or supplemented with an IPRM variant trained from raw frames.
  2. [3.2, Table 3, Appendix B.1, Table 6] The CLOSURE result is presented as 'IPRM achieves an average zero-shot accuracy of 75.6% which is highest amongst fully neural reasoning methods,' but this aggregate masks a substantial per-type weakness. Table 6 shows that IPRM scores 62.2 on compare_mat and 61.5 on compare_mat_spa, both lower than FiLM (66.2 and 65.8) and MAC (65.3 and 66.2). The overall average is carried by a large advantage on and_mat_spa (83.1 versus 63.7 for MAC). Since the paper's broader claim is consistent superiority over prior reasoning methods, the main-text statement is misleading without this per-question-type breakdown. The text should either qualify the claim or report the full Table 6 in the main body.
minor comments (5)
  1. [Section 2.1] The text refers to 'appendix sec. 13' for the attention implementation, but the appendix is organized as Sections A-D; the reference should be corrected to the actual appendix section containing the pseudocode.
  2. [Section 5] The word 'quantitative' is misspelled as 'quantiative' in the conclusion; this should be corrected.
  3. [Table 1 and Section 3] The paper states that primary experiments are averaged over three trials, but the main tables report no standard deviations or error bars; adding them would allow readers to judge the stability of the reported margins, especially the 4-to-5 point differences that drive the main claims.
  4. [Section 3.3] The discussion of the co-dependence between N_op and T is based on Figure 6, but the figure does not show exact accuracy values or error bars; reporting the numerical grid (e.g., in an appendix table) would make the ablation more informative.
  5. [Appendix C, Figures 12-14] The pseudocode uses a custom attention function 'mod_attn' (element-wise multiplication followed by a linear projection) that differs from the standard dot-product attention notation used in the main text; adding a brief note that the pseudocode reflects the actual implementation would avoid confusion for readers trying to reproduce the module.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IPRM's claims are empirical benchmark comparisons; the single self-citation (ref. [31]) is a non-load-bearing detail, and the STAR/AGQA input-representation gap is a comparability issue, not a derivation that reduces to its inputs.

full rationale

The paper reports an end-to-end trained neural architecture and compares it against external baselines; there is no chain of equations in which a predicted quantity is defined in terms of the same quantity, and no fitted parameter is renamed as a prediction. The architecture hyperparameters Nop=6, T=9, r=2, W=2 are selected via ablations on a reduced CLEVR-Humans validation setting and then fixed across benchmarks, which is standard model selection rather than circular reasoning. The only self-citation, [31] (TDAM), is used in Section 2.2 for the feature-reduction ratio r ('Here, r is a feature reduction ratio [23, 31]'); it is a minor design choice and does not support the central iterative/parallel claim. The appendix's disclosure that STAR and AGQAv2 inputs are object bounding boxes, labels, pose and relations, while many Table 1 baselines use raw frames (Appendix C.1), raises a legitimate benchmark-comparability concern, but that is an external-validity or fairness issue, not circularity: IPRM's mechanism is still evaluated against matched Concat-Att/Cross-Att baselines sharing the same object-level tokens. No circular step meets the evidentiary bar of Eq. X = Eq. Y by construction or parameter-renaming-as-prediction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 2 invented entities

The architecture introduces latent operation and result states, plus a set of hyperparameters chosen by validation. These are internal representations rather than new physical entities. The central empirical claim depends on the domain assumption that a fixed number of attention-based slots can represent compositional reasoning, and on the evaluation assumption that object-level inputs are comparable with prior methods.

free parameters (5)
  • Number of parallel operations Nop = 6
    Selected by ablation on a reduced-resolution CLEVR-Humans setting (Figure 6) and then fixed for all benchmarks.
  • Number of iterative steps T = 9
    Selected by the same ablation study; T and Nop are reported as co-dependent.
  • Memory lookback window W = 2
    Chosen from ablations (Figure 6 plot iv); decreasing it significantly hurts accuracy.
  • Feature reduction ratio r = 2
    Chosen from ablations (Figure 6 plot iii); r=8 degrades performance.
  • Internal dimension Dm = 512
    Architecture width used for all experiments; a hand-chosen hyperparameter not tuned across benchmarks.
assumptions (3)
  • domain assumption A fixed-size memory of Nop=6 parallel operation states is sufficient to represent all operations required by the evaluated questions.
    Sections 2 and 3.3 set Nop=6 by validation; no expressiveness or capacity bound is proved. Long or branching questions may exceed this slot count.
  • domain assumption Attention-based retrieval, feature modulation, and inter-operation attention can implement compositional operations such as counting, comparison, and spatial relations.
    Sections 2.1 to 2.3 define these as trainable modules; correctness is inferred only from downstream accuracy and attention visualizations, not from any formal semantics.
  • domain assumption The frozen visual features or object-level annotations contain enough information to answer the questions.
    Appendix C.1 uses ResNet features, object bounding boxes, labels, poses, or relations. The paper's predicted-vision setup on STAR drops about 9%, showing sensitivity to perception quality.
invented entities (2)
  • Parallel operation states Mop
    purpose: Latent slots meant to represent distinct reasoning operations at each iterative step.
    Internal to the network; their existence is evidenced only by downstream task accuracy and attention maps generated by the model itself.
  • Result states Mres
    purpose: Latent memories keyed to operation states, intended to store executed results for later composition.
    Same as operation states: an architectural latent construct without an external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios." pith.science (2026). https://pith.science/paper/4DKKD54Y

@misc{pith2026241113754,
  author       = {Pith},
  title        = {Pith review of: Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DKKD54Y}},
  note         = {Machine review of arXiv:2411.13754}
}
read the original abstract

Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capabilities beyond the immediate recognition and localization of objects and events. Here, we introduce a fully neural Iterative and Parallel Reasoning Mechanism (IPRM) that combines two distinct forms of computation -- iterative and parallel -- to better address complex VQA scenarios. Specifically, IPRM's "iterative" computation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need to be computed, stored, and recalled dynamically (e.g. when computing the query "determine the color of pen to the left of the child in red t-shirt sitting at the white table"). Meanwhile, its "parallel" computation allows for the simultaneous exploration of different reasoning paths and benefits more robust and efficient execution of operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts"). We design IPRM as a lightweight and fully-differentiable neural module that can be conveniently applied to both transformer and non-transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such as compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi-hop reasoning generalization (CLEVR-Humans) and causal event linking (CLEVRER-Humans). Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its errors.

Figures

Figures reproduced from arXiv: 2411.13754 by the authors.

Figure 1
Figure 1. Complex VQA scenarios (CLEVR-Humans [35], GQA [29], CLEVRER-Humans[51]), AGQA[20] and STAR[79]) wherein combination of iterative (step-by-step) computation (blue phrases) and parallel computation (orange phrases) can be beneficial for reasoning. limitations in scenarios wherein the entailed visual operations are independent of one-another, or where distinct stimuli need to be processed and tracked simultaneously. Fo… view at source ↗
Figure 2
Figure 2. IPRM’s computation flow diagram. First, a new set of N-parallel latent operations [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Operation Composition Unit We set ls to be the first token in case of transformer-based language backbones and as last hidden state in case of LSTM-based language backbones. As shown in eq. (18), ls is projected to obtain a single-token attention query pq while Mop,T+1 is projected to obtain the attention keys Pk. The attention value is simply the result states Mres,T+1, and the output of the attention function is t… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: IPRM performance on CLEVR-Humans at different training data ratios of Cross- and Concat-Att. 3.2 Compositional Image Reasoning (CLEVR-Humans, CLEVR-CoGen and GQA) Here, we evaluate IPRM on challenging compositional image reasoning benchmarks. CLEVR￾Humans tests general…
Figure 6
Figure 6. Figure 6: IPRM Model ablations in order: (i) Impact of number of parallel operations (Nop) vs computation steps (T). (ii) Impact of Operation Composition Block (OPC). (iii): Impact of reduction ratio (r) and (iv) memory window length (W). 3.3 Model ablations and reasoning visual…
Figure 7
Figure 7. Figure 7: Condensed reasoning visualization of IPRM. In the top two examples, IPRM correctly utilizes both parallel and iterative computation to arrive at the correct answer. The bottom left example shows IPRM’s cumulative lang. and visual attentions when solving a real-world GQ…
Figure 8
Figure 8. Figure 8: Top: original image and question; middle: language attentions across parallel operations (clubbed together; op_k represents parallel operation k) and computation steps. Bottom: Visual attentions across parallel ops and computation steps. Here, IPRM correctly utilizes p…
Figure 9
Figure 9. Figure 9: In this example, IPRM predicts the correct answer and its visual attention trace provides [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Example where IPRM outputs incorrect answer and the intermediate reasoning appears [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Example wherein IPRM produces correct answer but its visual attention trace suggests [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: IPRM pseudocode (1/3) 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: IPRM pseudocode (2/3) 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: IPRM pseudocode (3/3) 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 51 canonical work pages

  1. [1]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022

    2022

  2. [2]

    Andreas, M

    J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01705, 2016

    arXiv 2016

  3. [3]

    Bahdanau, H

    D. Bahdanau, H. de Vries, T. J. O’Donnell, S. Murty, P. Beaudoin, Y . Bengio, and A. Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05783, 2019

    arXiv 1912

  4. [4]

    Z. Bai, R. Wang, and X. Chen. Glance and focus: Memory prompting for multi-event video question answering. Advances in Neural Information Processing Systems, 36, 2024. 10

    2024

  5. [5]

    Bhattacharyya, S

    A. Bhattacharyya, S. Panchal, R. Pourreza, M. Lee, P. Madan, and R. Memisevic. Look, remember and reason: Grounded reasoning in videos with language models. In The Twelfth International Conference on Learning Representations

  6. [6]

    Bogin, S

    B. Bogin, S. Subramanian, M. Gardner, and J. Berant. Latent compositional representations im- prove systematic generalization in grounded question answering.Transactions of the Association for Computational Linguistics, 9:195–210, 2021

    2021

  7. [7]

    S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, L. Fei-Fei, and J. C. Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2917–2927, 2022

    2022

  8. [8]

    Bulatov, Y

    A. Bulatov, Y . Kuratov, and M. Burtsev. Recurrent memory transformer.Advances in Neural Information Processing Systems, 35:11079–11091, 2022

    2022

Show all 89 references
  1. [9]

    Capon, S

    A. Capon, S. Handley, and I. Dennis. Working memory and reasoning: An individual differences perspective. Thinking & Reasoning, 9(3):203–244, 2003

    2003

  2. [10]

    Y . Cong, W. Liao, H. Ackermann, B. Rosenhahn, and M. Y . Yang. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16372–16382, 2021

    2021

  3. [11]

    Dehghani, S

    M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018

    2018 arXiv

  4. [12]

    D. Ding, F. Hill, A. Santoro, M. Reynolds, and M. Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in neural information processing systems, 34:9112–9124, 2021

    2021

  5. [13]

    M. Ding, Z. Chen, T. Du, P. Luo, J. Tenenbaum, and C. Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. Advances in Neural Information Processing Systems, 34:887–899, 2021

    2021

  6. [14]

    Dziri, X

    N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y . Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1999–2007, 2019

    1999

  8. [16]

    H.-S. Fang, S. Xie, Y .-W. Tai, and C. Lu. Rmpe: Regional multi-person pose estimation. In Proceedings of the IEEE international conference on computer vision, pages 2334–2343, 2017

    2017

  9. [17]

    D. Fougnie. The relationship between attention and working memory. New research on short-term memory, 1:45, 2008

    2008

  10. [18]

    D. Gao, L. Zhou, L. Ji, L. Zhu, Y . Yang, and M. Z. Shou. Mist: Multi-modal iterative spatial- temporal transformer for long-form video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14773–14783, 2023

    2023

  11. [19]

    Girshick

    R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015

    2015

  12. [20]

    Grunde-McLaughlin, R

    M. Grunde-McLaughlin, R. Krishna, and M. Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287–11297, 2021

    2021

  13. [21]

    Grunde-McLaughlin, R

    M. Grunde-McLaughlin, R. Krishna, and M. Agrawala. Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning. arXiv preprint arXiv:2204.06105, 2022

    2022 arXiv

  14. [22]

    J. Hsu, J. Mao, J. Tenenbaum, and J. Wu. What’s left? concept grounding with logic-enhanced foundation models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  15. [23]

    J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018

    2018

  16. [24]

    R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804–813, 2017. 11

    2017

  17. [25]

    R. Hu, A. Rohrbach, T. Darrell, and K. Saenko. Language-conditioned graph networks for relational reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10294–10303, 2019

    2019

  18. [26]

    Huang, K

    F. Huang, K. Lu, C. Yuxi, Z. Qin, Y . Fang, G. Tian, and G. Li. Encoding recurrence into transformers. In The Eleventh International Conference on Learning Representations, 2022

    2022

  19. [27]

    Hudson and C

    D. Hudson and C. D. Manning. Learning by abstraction: The neural state machine. Advances in Neural Information Processing Systems, 32, 2019

    2019

  20. [28]

    D. A. Hudson and C. D. Manning. Compositional attention networks for machine reasoning. arXiv preprint arXiv:1803.03067, 2018

    2018 arXiv

  21. [29]

    D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  22. [30]

    Hutchins, I

    D. Hutchins, I. Schlag, Y . Wu, E. Dyer, and B. Neyshabur. Block-recurrent transformers. Advances in Neural Information Processing Systems, 35:33248–33261, 2022

    2022

  23. [31]

    Jaiswal, B

    S. Jaiswal, B. Fernando, and C. Tan. Tdam: Top-down attention module for contextually guided feature selection in cnns. In European Conference on Computer Vision, pages 259–276. Springer, 2022

    2022

  24. [32]

    A. Jha, B. Patro, L. Van Gool, and T. Tuytelaars. Barlow constrained optimization for visual question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1084–1093, 2023

    2023

  25. [33]

    C. Jing, Y . Jia, Y . Wu, X. Liu, and Q. Wu. Maintaining reasoning consistency in compositional visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5099–5108, 2022

    2022

  26. [34]

    Johnson, B

    J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901– ...

    2017

  27. [35]

    Johnson, B

    J. Johnson, B. Hariharan, L. Van Der Maaten, J. Hoffman, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference on computer vision, pages 2989–2998, 2017

    2017

  28. [36]

    Kamath, M

    A. Kamath, M. Singh, Y . LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detec- tion for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021

    2021

  29. [37]

    W. Kim, B. Son, and I. Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583–5594. PMLR, 2021

    2021

  30. [38]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  31. [39]

    T. M. Le, V . Le, S. Venkatesh, and T. Tran. Hierarchical conditional relation networks for video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9972–9981, 2020

    2020

  32. [40]

    J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7331–7341, 2021

    2021

  33. [41]

    Lewis, N

    M. Lewis, N. V . Nayak, P. Yu, Q. Yu, J. Merullo, S. H. Bach, and E. Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10537, 2022

    2022 arXiv

  34. [42]

    C. Li, H. Xu, J. Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections.arXiv preprint arXiv:2205.12005, 2022

    2022 arXiv

  35. [43]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 12

    2023 arXiv

  36. [44]

    J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022

    2022

  37. [45]

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  38. [46]

    X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, p...

    2020

  39. [47]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  40. [48]

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  41. [49]

    J. Lu, V . Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10437–10446, 2020

    2020

  42. [50]

    J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Inter- preting scenes, words, and sentences from natural supervision.arXiv preprint arXiv:1904.12584, 2019

    1904 arXiv

  43. [51]

    J. Mao, X. Yang, X. Zhang, N. Goodman, and J. Wu. Clevrer-humans: Describing physical and causal events the human way. Advances in Neural Information Processing Systems, 35:7755– 7768, 2022

    2022

  44. [52]

    Mascharka, P

    D. Mascharka, P. Tran, R. Soklaski, and A. Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4942–4950, 2018

    2018

  45. [53]

    S. S. Mondal, J. D. Cohen, and T. W. Webb. Slot abstractors: Toward scalable abstract visual reasoning. arXiv preprint arXiv:2403.03458, 2024

    2024 arXiv

  46. [54]

    S. S. Mondal, T. Webb, and J. D. Cohen. Learning to reason over visual objects. arXiv preprint arXiv:2303.02260, 2023

    2023 arXiv

  47. [55]

    B. X. Nguyen, T. Do, H. Tran, E. Tjiputra, Q. D. Tran, and A. Nguyen. Coarse-to-fine reasoning for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4558–4566, 2022

    2022

  48. [56]

    D. K. Nguyen, V . Goswami, and X. Chen. Movie: Revisiting modulated convolutions for visual counting and beyond. In International Conference on Learning Representations, 2020

    2020

  49. [57]

    R. OpenAI. Gpt-4 technical report. ArXiv, 2303, 2023

    2023

  50. [58]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  51. [59]

    Pennington, R

    J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014

    2014

  52. [60]

    Perez, F

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  53. [61]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  54. [62]

    Shrestha, K

    R. Shrestha, K. Kafle, and C. Kanan. Answer them all! toward universal visual question answering models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10472–10481, 2019. 13

    2019

  55. [63]

    A. Suhr, M. Lewis, J. Yeh, and Y . Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223, 2017

    2017

  56. [64]

    A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y . Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418–6428, 2019

    2019

  57. [65]

    Tan and M

    H. Tan and M. Bansal. Object ordering with bidirectional matchings for visual reasoning. In Proceedings of NAACL-HLT, pages 444–451, 2018

    2018

  58. [66]

    Tan and M

    H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL...

    2019

  59. [67]

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  60. [68]

    Urooj, H

    A. Urooj, H. Kuehne, B. Wu, K. Chheu, W. Bousselham, C. Gan, N. Lobo, and M. Shah. Learning situation hyper-graphs for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14879–14889, 2023

    2023

  61. [69]

    Vaishnav and T

    M. Vaishnav and T. Serre. Gamr: A guided attention model for (visual) reasoning. In The Eleventh International Conference on Learning Representations, 2022

    2022

  62. [70]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  63. [71]

    J. Wang, G. Chen, Y . Huang, L. Wang, and T. Lu. Memory-and-anticipation transformer for online action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13824–13835, 2023

    2023

  64. [72]

    J. Wang, Y . Ge, R. Yan, Y . Ge, K. Q. Lin, S. Tsutsui, X. Lin, G. Cai, J. Wu, Y . Shan, et al. All in one: Exploring unified video-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6598–6608, 2023

    2023

  65. [73]

    P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022

    2022

  66. [74]

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for vision and vision- language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2023

  67. [75]

    Y . Wang, K. Li, Y . Li, Y . He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y . Liu, Z. Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022

    2022 arXiv

  68. [76]

    T. Webb, S. S. Mondal, and J. D. Cohen. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36, 2024

    2024

  69. [77]

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  70. [78]

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceed- ings of the 2020 conference on empirical methods in natural language processing: system dem...

    2020

  71. [79]

    B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 14

    2021

  72. [80]

    C.-Y . Wu, Y . Li, K. Mangalam, H. Fan, B. Xiong, J. Malik, and C. Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587...

    2022

  73. [81]

    Q. Wu, Z. Lan, K. Qian, J. Gu, A. Geramifard, and Z. Yu. Memformer: A memory-augmented transformer for sequence modeling. In Y . He, H. Ji, S. Li, Y . Liu, and C.-H. Chang, editors, Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 308–318, Onl...

    2022

  74. [82]

    Y . Wu, Y . Zhao, B. Hu, P. Minervini, P. Stenetorp, and S. Riedel. An efficient memory- augmented transformer for knowledge-intensive nlp tasks. arXiv preprint arXiv:2210.16773, 2022

    2022 arXiv

  75. [83]

    Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21–29, 2016

    2016

  76. [84]

    K. Yi, C. Gan, Y . Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019

    1910 arXiv

  77. [85]

    K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. Tenenbaum. Neural-symbolic vqa: Disen- tangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018

    2018

  78. [86]

    S. Yu, J. Cho, P. Yadav, and M. Bansal. Self-chained image-language model for video localiza- tion and question answering. arXiv preprint arXiv:2305.06988, 2023

    2023 arXiv

  79. [87]

    Z. Yu, J. Yu, Y . Cui, D. Tao, and Q. Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6281–6290, 2019

    2019

  80. [88]

    find object close to front

    P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y . Choi, and J. Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588, 2021. A Appendix / supplemental ...

    2021

  81. [89]

    primary color is

    optimizer and gradient clipping value of 8. The learning-rate is reduced based on validation acc. plateau with reduction factor 0.5, threshold 0.001 and patience 0. Further experiment hyper- parameters and settings are provided below. Source code at: https://github.com/shantan...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.