Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Understanding Complexity in VideoQA via Visual Program Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Program complexity predicts VideoQA difficulty better than human ratings, and the hardest-filtered questions form a benchmark 1.9 times harder than NExT-QA.

desk verdict A solid, well-designed empirical paper; the program-faithfulness caveat is real but does not sink the core claim. read the letter →

arxiv 2505.13429 v1 pith:3WU5PTUT submitted 2025-05-19 cs.CV

Cristobal Eyzaguirre , Igor Vasiljevic , Achal Dave , Jiajun Wu , Rares Andrei Ambrus , Thomas Kollar , Juan Carlos Niebles , Pavel Tokmakov This is my paper · ORCID
classification cs.CV
keywords videoquestionansweringdifficultyestimationvisualprogramgenerationcodecomplexityabstractsyntaxtreebenchmarkconstructionPlexityQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video question answering is hard to benchmark because human intuition about difficulty does not match where machine models actually fail. This paper argues that an LLM's executable visual program for a question carries a reliable signal of that question's complexity, learnable from model outcomes instead of expert guesses. The introduced CodePlexity score predicts held-out model success on NExT-QA and MVBench better than human rankings, GPT-4 Likert ratings, BERT, dependency-tree depth, lines of code, and cyclomatic complexity. Applying the same score to automatically generated questions produces CodePlex-QA, on which six zero-shot models score about 1.9 times worse than on NExT-QA, with the learned subtrees pointing to shared failures in temporal ordering and fine-grained object analysis.

What carries the argument

The engine is a compiler-assisted subtree analysis: each generated program is compiled to an Abstract Syntax Tree, all valid subtrees are mined, always-co-occurring nested subtrees are merged, and each question becomes a one-hot encoding over the distinct subtrees. A logistic regression trained on the pooled success labels of four VideoQA models then assigns each subtree a weight, and the final score $\mathrm{CodePlexity}(z)=-\sigma(w x+b)$ is the negative predicted success probability. The same subtree vocabulary doubles as an interpretable diagnostic: the eight subtrees shared by all training models cluster into patterns that require ordering frames in time (loops over frames, breaks after a first match, frame-after-event lookups) and patterns that require fine-grained object or relationship checks, giving a human-readable account of common failure modes.

What would settle it

Take a fixed set of NExT-QA questions, generate programs with two deliberately different generators whose failure patterns on hard questions are uncorrelated, train CodePlexity on one generator's outputs, and evaluate on the other; if the transferred score no longer predicts held-out model success while each generator-specific score does, the central proxy is generator-relative rather than a property of the questions. A cheaper check already available from the paper's data is to compare mPEG on questions whose generated code is semantically incorrect versus correct: if the signal disappears entirely on incorrect code, CodePlexity is tracking program quality rather than question complexity.

Watch

Extended reading notes

Core claim

The central claim is that the complexity of a VideoQA question is encoded in the executable program an LLM writes to answer it, so question difficulty can be read off the program rather than judged from the natural-language text. Concretely, a generator $\pi$ turns question $q$ into code $z=\pi(q)$; each program is compiled to an Abstract Syntax Tree, subtrees are mined and merged, and the presence pattern becomes a one-hot vector $x_i$. A logistic regression trained on the pooled success/failure labels of several VideoQA models yields $\mathrm{CodePlexity}(z)=-\sigma(w x_i+b)$, the negative predicted success probability. The paper reports that, measured by the Performance Extremity Gap averaged over difficulty fractions, CodePlexity beats human annotator ELO rankings, GPT-4 Likert scores, BERT trained on the same labels, dependency-tree depth, lines of code, and cyclomatic complexity, on held-out models and on the MVBench dataset. The same score filters an LLM-generated pool of 20,791 candidate questions down to CodePlex-QA (1,981 questions after a 12% manual answerability check), where zero-shot models score about 1.9 times lower than on NExT-QA; the paper also reports that correlations weaken, but less for CodePlexity, when generated code is semantically incorrect.

Load-bearing premise

The weakest link is that the code generated from a question faithfully represents the reasoning needed to answer it; if the generator's programs are not a faithful mirror, especially for hard questions, then CodePlexity measures generator artifacts rather than question difficulty.

Editorial extensions

If this is right

  • Code complexity correlates with held-out model performance across architectures that never saw the metric during training, so the signal is not an artifact of one model family.
  • The learned subtree patterns identify shared weaknesses, chiefly fine-grained temporal ordering and object-centric spatial reasoning, giving model designers concrete places to intervene.
  • Because the measure is data-driven and re-trainable, it can be re-estimated as new VideoQA models appear, unlike fixed human-designed difficulty heuristics.
  • CodePlex-QA is harder than NExT-QA for every zero-shot model tested and harder than ATP-Hard for the strongest models, showing that question selection with this score can produce useful benchmarks without hand-written difficulty rules.
  • Code generation quality bounds the metric: stronger code generators give stronger correlations, and correlations weaken on incorrect code, so improving program synthesis directly improves complexity estimation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If program complexity faithfully tracks reasoning load, the same CodePlexity score could drive curriculum design, active sampling, or test-time budgeting in any learned QA pipeline, not just benchmark construction.
  • The paper's own number shows fine-tuning SeViLA on CodePlex-QA improves accuracy only about 2.5 points, much less than fine-tuning on NExT-QA; a testable consequence is that these hard questions expose a representation gap that small amounts of extra data will not close.
  • Because the method's input is only question-plus-program, the residual between CodePlexity and actual model performance could be used as a separate, trainable estimator of video complexity, in line with the chain-rule motivation the paper sketches.
  • One caution worth testing: if two different program generators fail on different kinds of hard questions, transferability of a CodePlexity model trained on one generator may degrade; the claim of robustness across generators is currently supported only by a two-generator comparison.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that the complexity of LLM-generated visual programs is a robust, data-driven proxy for the difficulty of VideoQA questions. The authors generate code for questions using ViperGPT and RVP, compute structural metrics (Lines of Code, Cyclomatic Complexity), and propose CodePlexity, a learned metric that combines AST subtree presence features with logistic regression (Eq. 3) to predict model success. They evaluate the resulting complexity scores with the proposed mPEG measure on held-out models and held-out questions in NExT-QA and MVBench, reporting that CodePlexity outperforms human ratings, text-based baselines, and simple code metrics. The paper also uses CodePlexity to filter automatically generated questions, constructing CodePlex-QA, which they report is 1.9 times harder than NExT-QA. The main technical contributions are the interpretable subtree analysis of failure modes and the automatic benchmark construction pipeline.

Significance. If the central claim holds, this is a useful contribution: it offers a scalable and interpretable alternative to expert-curated difficulty measures, with a clean held-out evaluation protocol (four training models and five held-out models in NExT-QA, three and two in MVBench), evaluation on two datasets, and an ablation that controls for video-source effects by using the same VidOR videos as NExT-QA. The paper also makes its code and data available, which strengthens reproducibility. The most important caveat is that the metric is computed only on generated programs, so its validity depends on how faithfully those programs reflect the reasoning required by the question. The paper's own Section 10.3 shows that correlations weaken when generated code is incorrect, which makes the faithfulness concern load-bearing rather than incidental. The lack of statistical uncertainty in the main comparisons is also a substantive gap for a paper whose quantitative claims are differences in mPEG and benchmark accuracy.

major comments (4)
  1. [Section 3.1 / Section 10.3, Table 5] The faithfulness of the generated program z = pi(q) is load-bearing, but the paper's own Table 5 shows that all complexity metrics, including CodePlexity, correlate less with model performance when generated code gives incorrect answers (e.g., ViperGPT: CodePlexity drops from 0.3128 to 0.2041; SeViLA: Cyclomatic drops from 0.1702 to 0.1351). This leaves open the confound that hard questions stress the code generator, producing structurally complex but semantically wrong programs, so CodePlexity may partly measure generator artifacts rather than intrinsic question complexity. The CodePlex-QA pipeline in Section 3.4 amplifies this concern because questions are scored purely from GPT-3.5-generated code without executing the code on the video and without semantic-fidelity verification beyond the 12% manual answerability filter. I recommend a direct test of stability across code generators (e.g., Codex, GPT-3.5, GPT-4, RVP) on the same questions, and a subgroup analysis that computes mPEG only on questions where the generated code is correct.
  2. [Section 7.4 / Section 3.4] CodePlexity is trained on programs generated by Codex (via the ViperGPT API), but CodePlex-QA is scored using programs generated by a text variant of GPT-3.5 with a different prompt structure. There is no experiment showing that the learned logistic regression weights transfer across code generators. Since Eq. 3 is applied to GPT-3.5-generated programs to select the final benchmark, a distribution shift between training and scoring could change which subtrees are activated and thus distort the complexity scores. The authors should either report CodePlexity's ranking consistency across generators on the same question set, or retrain the metric on the target generator and compare resulting benchmark difficulty.
  3. [Section 3.3, Eq. (6)] The subtree significance test is applied at p < 0.01 over a very large set of candidate subtrees S(D), potentially thousands of hypotheses, without any multiple-testing correction. The intersection set S* in Eq. (7) may therefore contain spurious subtrees, and the subsequent interpretable insights in Section 4.3 (the eight shared subroutines) may be partially driven by false positives. The authors should report the number of subtrees tested and apply a correction such as Bonferroni, Benjamini-Hochberg, or a permutation-based null to justify the p < 0.01 threshold.
  4. [Section 4.1 / Tables 1, 2, 6] The main quantitative claims are made without error bars or significance tests. For example, in Table 1, CodePlexity outperforms Cyclomatic Complexity by margins that vary from 1.0 to 8.8 mPEG points across held-out models, but there is no confidence interval or hypothesis test to distinguish these differences from noise. Similarly, the '1.9 times harder' claim in Table 2 averages across models with no uncertainty quantification. Because mPEG is an average over alpha thresholds, a bootstrap over questions and models would be an appropriate and straightforward remedy. I request bootstrap confidence intervals for the mPEG values and a paired test of CodePlexity vs. the best baseline for at least the held-out models.
minor comments (4)
  1. [Broader Impacts and Limitations] There is a typo in the Limitations paragraph: 'may overlooks critical sources' should be 'may overlook critical sources.'
  2. [Figure 3] The text reports qualitative trends for human, dependency-tree, and cyclomatic-complexity correlations, but it does not provide numerical correlation coefficients. Reporting Spearman rho or R^2 for each panel would make the comparison in Section 4.2 more precise.
  3. [Section 7.2] The human study description reports 65 recruited subjects and 30 retained annotators; please clarify how many comparisons per question were collected before consistency filtering, and whether the 150-question subset was selected randomly or with any stratification.
  4. [Table 5] The table caption says 'correlation between the complexity metrics and the mPEG metric'; mPEG is itself derived from correlations, so it would be clearer to state that the table reports the correlation between each complexity metric and the per-bin success rates used to compute PEG.

Circularity Check

1 steps flagged · score 6.0 of 10

CodePlex-QA hardness is partially built into the fitted CodePlexity score for training models; held-out models provide the independent part.

  1. fitted input called prediction [Section 3.2 (Eq. 3), Section 3.4 (Eq. 8), Section 5.2 (Table 2)]
    "We then employ a logistic regression model trained on these one-hot encodings to predict the success of models mj∈M. ... CodePlexity(z) = −yˆi = −σ(wxi + b). (3) ... Following our approach from Section 3.2, we then convert each generated question ˜q into code, and use our trained CodePlexity model (Equation 3) to estimate its complexity. A set of candidate questions ˜Q∗ can be selected by setting a threshold δ for minimum complexity: ˜Q∗ = {˜q ∈ ˜Q | C(˜q) ≥ δ}. (8)"

    Eq. 3 is fitted to success/failure labels y_i^(j) of Mtr = {VIOLET, SeViLA, ViperGPT, ATP} (Section 4.1). CodePlex-QA is constructed by thresholding this fitted score (Eq. 8). Table 2 then reports CodePlex-QA as 1.9x harder than NExT-QA using, among others, ViperGPT (60.0 to 45.8) and VIOLET (37.7 to 27.6), both members of Mtr. For these models, selecting questions with C(q) >= delta is exactly selecting questions the logistic regression predicts those models will fail on; the observed accuracy drop for them is therefore a restatement of the training signal, not independent evidence of intrinsic question complexity.

full rationale

The core metric evaluation in Table 1 is not circular: CodePlexity is trained on 80% of NExT-QA questions using only Mtr outcomes, then scored on held-out questions and on Mval = {HGA, SeViLA-ZS, InternVideo, Tarsier, LLaVa-Video}; no equation forces those held-out correlations. The MVBench experiment (Section 10.5) and the CodePlex-QA difficulty observed on Tarsier/LLaVa-Video/SeViLA-ZS/InternVideo are independent generalizations. The circular component is the benchmark-hardness validation: because CodePlexity is defined as negative predicted model success for the training models, selecting top-difficulty questions (Eq. 8) and then showing that ViperGPT and VIOLET, two of the training models, do worse on them is a restatement of the fit. The 1.9x claim should be read as combining this by-construction component with the genuinely predictive component on held-out models. No other load-bearing self-citation chain or uniqueness import appears; citations to prior code-generation work provide independent, executable support.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities or exogenous quantities are introduced; all new content is a learned metric (CodePlexity) and a generated dataset (CodePlex-QA). The central claim rests on fitted parameters and domain assumptions about code faithfulness and proxy validity.

free parameters (3)
  • Logistic regression weights w and bias b of CodePlexity = Learned on NExT-QA validation 80% split with labels from VIOLET, SeViLA, ViperGPT, ATP
    Eq. 3 defines CodePlexity(z) = -sigma(wx+b); w and b are fit to model success labels, so the complexity score is a fitted predictor of model failure.
  • Selection threshold delta = Top 10% of generated questions (calibrated on NExT-QA)
    Eq. 8 uses delta to select hard questions; setting it to the top 10% is a design choice that directly controls CodePlex-QA hardness.
  • L2 regularization constant c = 1.0
    Chosen via 5-fold cross-validated accuracy on the training split (Section 7.5); affects feature selection and generalization.
assumptions (5)
  • domain assumption Generated programs z = pi(q) faithfully represent the reasoning required by the question.
    Invoked throughout Sections 3.1 and 3.2; if the LLM-generated code is wrong or superficial, code complexity is not a measure of question complexity. Section 10.3 shows correlations depend on code correctness.
  • domain assumption Code structural complexity (LoC, cyclomatic complexity, AST subtree presence) is a valid proxy for the reasoning complexity of a VideoQA question.
    The method never derives this from a theory; it treats McCabe-style metrics and subtree features as proxies and validates them empirically.
  • domain assumption Model success labels from different architectures can be pooled and averaged into a single soft label per question.
    Section 7.5 averages labels across training models; this assumes common failure modes dominate and ignores model-specific variance, which Section 4.3 later acknowledges (each model has its own error modes).
  • ad hoc to paper The subtree significance test at p<0.01 (Eq. 6) is valid without multiple-testing correction across thousands of candidate subtrees.
    Section 3.3 sets significance at 0.99 and tests many subtrees; no Bonferroni or FDR correction is described, so the 8 shared subtrees may include false positives.
  • standard math The Kolmogorov complexity chain rule analogy (Eq. 12) justifies measuring question complexity separately from video complexity.
    Section 9.1 invokes K(x,y)=K(x)+K(y|x)+O(log...) as motivation, but the paper explicitly does not compute Kolmogorov complexity; the analogy is motivational, not a derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Complexity in VideoQA via Visual Program Generation." pith.science (2026). https://pith.science/paper/3WU5PTUT

@misc{pith2026250513429,
  author       = {Pith},
  title        = {Pith review of: Understanding Complexity in VideoQA via Visual Program Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3WU5PTUT}},
  note         = {Machine review of arXiv:2505.13429}
}
read the original abstract

We propose a data-driven approach to analyzing query complexity in Video Question Answering (VideoQA). Previous efforts in benchmark design have relied on human expertise to design challenging questions, yet we experimentally show that humans struggle to predict which questions are difficult for machine learning models. Our automatic approach leverages recent advances in code generation for visual question answering, using the complexity of generated code as a proxy for question difficulty. We demonstrate that this measure correlates significantly better with model performance than human estimates. To operationalize this insight, we propose an algorithm for estimating question complexity from code. It identifies fine-grained primitives that correlate with the hardest questions for any given set of models, making it easy to scale to new approaches in the future. Finally, to further illustrate the utility of our method, we extend it to automatically generate complex questions, constructing a new benchmark that is 1.9 times harder than the popular NExT-QA.

Figures

Figures reproduced from arXiv: 2505.13429 by the authors.

Figure 1
Figure 1. Humans struggle to judge which questions present higher challenges for machine learning models. In our study, the question on the left is universally perceived as being easier than the one on the right, which is inversely correlated with the models’ performance. We show that the complexity of the corresponding visual program can serve as a much more reliable predictor. standard code complexity metrics correlate bett… view at source ↗
Figure 2
Figure 2. Estimating question complexity via code. Our approach to estimating question complexity involves converting the question into code, decomposing the pseudo-code into abstract syntax subtrees (Si), before correlating subtree presence with model performance. 1986) T = compile(z) with nodes N and edges E. In this model, nodes represent variables, functions, and con￾trol structures, while the edges capture the logical an… view at source ↗
Figure 3
Figure 3. Correlation of various approaches for estimating question complexity with VideoQA models’ success rate on these questions. We observe that humans struggle to accurately predict what is hard for machine learning models and that code can serve as a more reliable source of prediction than natural language questions. visual-language model; we assess both its zero-shot vari￾ant (SeViLA-ZS) and a fine-tuned version (SeViL… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Detailed analysis of subtrees that correlate with challenging questions among several models. We find that, although each model has its own error-modes, 8 subroutines are shared among all 3 of them (right). One of the patterns we find then analyzing the shared code str…
Figure 5
Figure 5. Figure 5: Example questions in CodePlex-QA generated with our approach. It features many challenges that are under-represented in existing, manually-designed benchmarks, motivating development of new approaches with enhanced spatio-temporal modeling capacity. We further report m…
Figure 7
Figure 7. Figure 7: Comparison of the average number of entities (subjects and objects) in videos where models perform poorly on low com￾plexity tasks (easier questions) versus where they perform well on high complexity tasks (harder questions). More entities make easy questions harder, w…
Figure 9
Figure 9. Figure 9: Module Assign Const. Assign Const. False None [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 11
Figure 11. Figure 11: If condition Assign Const. Break [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 17
Figure 17. Figure 17: Temporal support (i.e. number of frames a question needs) according to the generated program. All models tested per￾form significantly worse on questions that require more frames. Counting questions are listed separately, as they potentially re￾quire every frame to be…
Figure 18
Figure 18. Figure 18: Filtering NExT-QA using our approach indeed results in a more challenging subset for all the evaluated models. How￾ever, our full dataset construction pipeline results in an even more challenging benchmark by first generating a more diverse pool of samples to select f…
Figure 20
Figure 20. Figure 20: Correlation of VideoQA models’ success rate on MVBench for various approaches for estimating question com￾plexity. As was the case for NExT-QA, we observe that code complexity correlated strongly with question complexity. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DAEP: Difficulty-Aware Evidence Planning for Medical Video Corpus Temporal Answer Grounding

    cs.CV 2026-08 conditional novelty 5.0 of 10

    DAEP, a difficulty-aware video retrieval and temporal grounding system, ranked first among ten systems on the NLPCC 2026 medical video question answering shared task.

Reference graph

Works this paper leans on

89 extracted references · 67 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A shared neural substrate for action verbs and observed actions in human posterior parietal cortex

    Aflalo, T., Zhang, C., Rosario, E., Pouratian, N., Orban, G., and Andersen, R. A shared neural substrate for action verbs and observed actions in human posterior parietal cortex. Science advances, 6 0 (43), 2020

    2020

  3. [3]

    W., and Brundage, M

    Agarwal, S., Krueger, G., Clark, J., Radford, A., Kim, J. W., and Brundage, M. Evaluating CLIP : towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02818, 2021

    arXiv 2021

  4. [4]

    Neural module networks

    Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural module networks. In CVPR, 2016

    2016

  5. [5]

    Is space-time attention all you need for video understanding? In ICML, 2021

    Bertasius, G., Wang, H., and Torresani, L. Is space-time attention all you need for video understanding? In ICML, 2021

    2021

  6. [6]

    Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., and Niebles, J. C. Revisiting the “video" in video-language understanding. In CVPR, 2022

    2022

  7. [7]

    H., Lu, P., Nocedal, J., and Zhu, C

    Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing, 16 0 (5): 0 1190--1208, 1995

    1995

  8. [8]

    ActivityNet : A large-scale video benchmark for human activity understanding

    Caba Heilbron, F., Escorcia, V., Ghanem, B., and Carlos Niebles, J. ActivityNet : A large-scale video benchmark for human activity understanding. In CVPR, 2015

    2015

Show all 89 references
  1. [9]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  2. [10]

    E., et al

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2023

    2023

  3. [11]

    and Gr \`e zes, J

    Decety, J. and Gr \`e zes, J. Neural mechanisms subserving the perception of human actions. Trends in cognitive sciences, 3 0 (5): 0 172--178, 1999

    1999

  4. [12]

    Brain activity during observation of actions

    Decety, J., Grezes, J., Costes, N., Perani, D., Jeannerod, M., Procyk, E., Grassi, F., and Fazio, F. Brain activity during observation of actions. influence of action content and subject's strategy. Brain: a journal of neurology, 120 0 (10): 0 1763--1777, 1997

    1997

  5. [13]

    Elo, A. E. The proposed uscf rating system, its development, theory, and applications. Chess Life, 22 0 (8): 0 242--247, 1967

    1967

  6. [14]

    K., Winn, J., and Zisserman, A

    Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes ( VOC ) challenge. International journal of computer vision, 88: 0 303--338, 2010

    2010

  7. [15]

    and Soto, A

    Eyzaguirre, C. and Soto, A. Differentiable adaptive computation time for visual reasoning. In CVPR, 2020

    2020

  8. [16]

    Masked autoencoders as spatiotemporal learners

    Feichtenhofer, C., Li, Y., He, K., et al. Masked autoencoders as spatiotemporal learners. NeurIPS, 2022

    2022

  9. [17]

    GPTScore : Evaluate as you desire

    Fu, J., Ng, S.-K., Jiang, Z., and Liu, P. GPTScore : Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023

    2023 arXiv

  10. [18]

    Y., Wang, L., and Liu, Z

    Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., and Liu, Z. VIOLET : End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021

    2021 arXiv

  11. [19]

    Recursive visual programming

    Ge, J., Subramanian, S., Shi, B., Herzig, R., and Darrell, T. Recursive visual programming. In ECCV, 2024

    2024

  12. [20]

    Adaptive computation time for recurrent neural networks

    Graves, A. Adaptive computation time for recurrent neural networks. In NIPS, 2016

    2016

  13. [21]

    AgQA : A benchmark for compositional spatio-temporal reasoning

    Grunde-McLaughlin, M., Krishna, R., and Agrawala, M. AgQA : A benchmark for compositional spatio-temporal reasoning. In CVPR, 2021

    2021

  14. [22]

    and Kembhavi, A

    Gupta, T. and Kembhavi, A. Visual programming: Compositional visual reasoning without training. In CVPR, 2023

    2023

  15. [23]

    V., Sethi, R., and Ullman, J

    Hoe, A. V., Sethi, R., and Ullman, J. D. Compilers—principles, techniques, and tools. Pearson Addison Wesley Longman, 1986

    1986

  16. [24]

    Learning to reason: End-to-end module networks for visual question answering

    Hu, R., Andreas, J., Rohrbach, M., Darrell, T., and Saenko, K. Learning to reason: End-to-end module networks for visual question answering. In ICCV, 2017

    2017

  17. [25]

    Huang, D.-A., Ramanathan, V., Mahajan, D., Torresani, L., Paluri, M., Fei-Fei, L., and Niebles, J. C. What makes a video a video: Analyzing temporal information in video understanding models and datasets. In CVPR, 2018

    2018

  18. [26]

    Hudson, D. A. and Manning, C. D. GQA : A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019

    2019

  19. [27]

    Ji, J., Krishna, R., Fei-Fei, L., and Niebles, J. C. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, 2020

    2020

  20. [28]

    and Han, Y

    Jiang, P. and Han, Y. Reasoning with heterogeneous graph alignment for video question answering. In AAAI Conference on Artificial Intelligence, 2020. URL https://api.semanticscholar.org/CorpusID:213506079

    2020

  21. [29]

    CLEVR : A diagnostic dataset for compositional language and elementary visual reasoning

    Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. CLEVR : A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017 a

    2017

  22. [30]

    Inferring and executing programs for visual reasoning

    Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman, J., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Inferring and executing programs for visual reasoning. In ICCV, 2017 b

    2017

  23. [31]

    Kenton, J. D. M.-W. C. and Toutanova, L. K. BERT : Pre-training of deep bidirectional transformers for language understanding. In naacL-HLT, 2019

    2019

  24. [32]

    W., Tapaswi, M., and Fidler, S

    Kim, S. W., Tapaswi, M., and Fidler, S. Visual reasoning by progressive module networks. In ICLR, 2018

    2018

  25. [33]

    and Bojar, O

    Kocmi, T. and Bojar, O. Curriculum learning and minibatch bucketing in neural machine translation. In RANLP, 2017

    2017

  26. [34]

    Kolmogorov, A. N. On tables of random numbers. Sankhy \=a : The Indian Journal of Statistics, Series A , pp.\ 369--376, 1963

    1963

  27. [35]

    and Kramer, O

    Kramer, O. and Kramer, O. Scikit-learn. Machine learning for evolution strategies, pp.\ 45--53, 2016

    2016

  28. [36]

    Dense-captioning events in videos

    Krishna, R., Hata, K., Ren, F., Fei-Fei, L., and Carlos Niebles, J. Dense-captioning events in videos. In ICCV, 2017

    2017

  29. [37]

    BLIP-2 : Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2 : Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023

    2023

  30. [38]

    MVBench : A comprehensive multi-modal video understanding benchmark

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. MVBench : A comprehensive multi-modal video understanding benchmark. In CVPR, 2024

    2024

  31. [39]

    A technique for the measurement of attitudes

    Likert, R. A technique for the measurement of attitudes. Archives of psychology, 1932

    1932

  32. [40]

    Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In NeurIPS Workshop on Instruction Tuning and Instruction Following, 2023 a

    2023

  33. [41]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023 b

    2023

  34. [42]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. LLaVA-NeXT : Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  35. [43]

    L., Nejadasl, F

    Liu, X., Pintea, S. L., Nejadasl, F. K., Booij, O., and Van Gemert, J. C. No frame left behind: Full video action recognition. In CVPR, 2021

    2021

  36. [44]

    C., Adeli, E., and Li, F.-F

    Luo, Z., Xie, W., Kapoor, S., Liang, Y., Cooper, M., Niebles, J. C., Adeli, E., and Li, F.-F. MOMA : Multi-object multi-actor activity parsing. NeurIPS, 2021

    2021

  37. [45]

    Y., Wu, J., Niebles, J

    Luo, Z., Durante, Z., Li, L., Xie, W., Liu, R., Jin, E., Huang, Z., Li, L. Y., Wu, J., Niebles, J. C., et al. MOMA-LRG : Language-refined graphs for multi-object multi-actor activity parsing. NeurIPS, 2022

    2022

  38. [46]

    VideoGPT+ : Integrating image and video encoders for enhanced video understanding

    Maaz, M., Rasheed, H., Khan, S., and Khan, F. VideoGPT+ : Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.09418, 2024

    2024 arXiv

  39. [47]

    Self-refine: Iterative refinement with self-feedback

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. NeurIPS, 2023

    2023

  40. [48]

    EgoSchema : A diagnostic benchmark for very long-form video language understanding

    Mangalam, K., Akshulakov, R., and Malik, J. EgoSchema : A diagnostic benchmark for very long-form video language understanding. In NeurIPS, 2023

    2023

  41. [49]

    McCabe, T. J. A complexity measure. IEEE Transactions on Software Engineering, SE-2 0 (4): 0 308--320, 1976

    1976

  42. [50]

    and Vondrick, C

    Menon, S. and Vondrick, C. Visual classification via description from large language models. In ICLR, 2022

    2022

  43. [51]

    OpenAI. Chatgpt. https://www.openai.com, 2023 a

    2023

  44. [52]

    GPT-4 technical report, 2023 b

    OpenAI. GPT-4 technical report, 2023 b

    2023

  45. [53]

    and Schitter, C

    Palan, S. and Schitter, C. Prolific. a subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17: 0 22--27, 2018

    2018

  46. [54]

    A., Stretcu, O., Neubig, G., Pocz \'o s, B., and Mitchell, T

    Platanios, E. A., Stretcu, O., Neubig, G., Pocz \'o s, B., and Mitchell, T. Competence-based curriculum learning for neural machine translation. In ACL, 2019

    2019

  47. [55]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  48. [56]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023

    2023

  49. [57]

    Annotating objects and relations in user-generated videos

    Shang, X., Di, D., Xiao, J., Cao, Y., Yang, X., and Chua, T.-S. Annotating objects and relations in user-generated videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp.\ 279--287. ACM, 2019

    2019

  50. [58]

    HuggingGPT : Solving AI tasks with ChatGPT and its friends in hugging face

    Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. HuggingGPT : Solving AI tasks with ChatGPT and its friends in hugging face. NeurIPS, 2023

    2023

  51. [59]

    A., Varol, G., Wang, X., Farhadi, A., Laptev, I., and Gupta, A

    Sigurdsson, G. A., Varol, G., Wang, X., Farhadi, A., Laptev, I., and Gupta, A. Hollywood in homes: Crowdsourcing data collection for activity understanding. In ECCV, 2016

    2016

  52. [60]

    Solomonoff, R. J. Algorithmic probability: Theory and applications. Information theory and statistical learning, pp.\ 1--23, 2009

    2009

  53. [61]

    T., and Leordeanu, M

    Soviany, P., Ardei, C., Ionescu, R. T., and Leordeanu, M. Image difficulty curriculum for generative adversarial networks ( CuGAN ). In WACV, 2020

    2020

  54. [62]

    less is more

    Spitkovsky, V. I., Alshawi, H., and Jurafsky, D. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In ACL, 2010

    2010

  55. [63]

    Modular visual question answering via code generation

    Subramanian, S., Narasimhan, M., Khangaonkar, K., Yang, K., Nagrani, A., Schmid, C., Zeng, A., Darrell, T., and Klein, D. Modular visual question answering via code generation. In ACL, 2023

    2023

  56. [64]

    ViperGPT : Visual inference via python execution for reasoning

    Sur \' s, D., Menon, S., and Vondrick, C. ViperGPT : Visual inference via python execution for reasoning. In ICCV, 2023

    2023

  57. [65]

    T., Fu, J., Phan, M

    Tay, Y., Wang, S., Luu, A. T., Fu, J., Phan, M. C., Yuan, X., Rao, J., Hui, S. C., and Zhang, A. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. In ACL, 2019

    2019

  58. [66]

    A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J

    Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. YFCC100M : The new data in multimedia research. Communications of the ACM, 59 0 (2): 0 64--73, 2016

    2016

  59. [67]

    Learning the curriculum with bayesian optimization for task-specific word representation learning

    Tsvetkov, Y., Faruqui, M., Ling, W., MacWhinney, B., and Dyer, C. Learning the curriculum with bayesian optimization for task-specific word representation learning. In ACL, 2016 a

    2016

  60. [68]

    Learning the curriculum with bayesian optimization for task-specific word representation learning

    Tsvetkov, Y., Faruqui, M., Ling, W., MacWhinney, B., and Dyer, C. Learning the curriculum with bayesian optimization for task-specific word representation learning. In ACL, 2016 b

    2016

  61. [69]

    P., and Ferrari, V

    Tudor Ionescu, R., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D. P., and Ferrari, V. How hard can it be? estimating the difficulty of visual search in an image. In CVPR, 2016

    2016

  62. [70]

    Neural discrete representation learning

    Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. NeurIPS, 2017

    2017

  63. [71]

    Tarsier: Recipes for training and evaluating large video description models

    Wang, J., Yuan, L., Zhang, Y., and Sun, H. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024

    2024 arXiv

  64. [72]

    InternVideo : General video foundation models via generative and discriminative learning

    Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., et al. InternVideo : General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022 a

    2022 arXiv

  65. [73]

    Language models with image descriptors are strong few-shot video-language learners

    Wang, Z., Li, M., Xu, R., Zhou, L., Lei, J., Lin, X., Wang, S., Yang, Z., Zhu, C., Hoiem, D., et al. Language models with image descriptors are strong few-shot video-language learners. NeurIPS, 2022 b

    2022

  66. [74]

    STC : A simple to complex framework for weakly-supervised semantic segmentation

    Wei, Y., Liang, X., Chen, Y., Shen, X., Cheng, M.-M., Feng, J., Zhao, Y., and Yan, S. STC : A simple to complex framework for weakly-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 0 (11): 0 2314--2320, 2016

    2016

  67. [75]

    B., and Gan, C

    Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., and Gan, C. STAR : A benchmark for situated reasoning in real-world videos. In NeurIPS, 2021

    2021

  68. [76]

    Wurm, M. F. and Caramazza, A. Two ‘what’ pathways for action and object recognition. Trends in cognitive sciences, 26 0 (2): 0 103--116, 2022

    2022

  69. [77]

    Next-QA : Next phase of question-answering to explaining temporal actions

    Xiao, J., Shang, X., Yao, A., and Chua, T.-S. Next-QA : Next phase of question-answering to explaining temporal actions. In CVPR, 2021

    2021

  70. [78]

    mPLUG-2 : A modularized multi-modal foundation model across text, image and video

    Xu, H., Ye, Q., Yan, M., Shi, Y., Ye, J., Xu, Y., Li, C., Bi, B., Qian, Q., Wang, W., et al. mPLUG-2 : A modularized multi-modal foundation model across text, image and video. arXiv preprint arXiv:2302.00402, 2023

    2023 arXiv

  71. [79]

    W., Salakhutdinov, R., and Manning, C

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018

    2018

  72. [80]

    Neural-symbolic VQA : Disentangling reasoning from vision and language understanding

    Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., and Tenenbaum, J. Neural-symbolic VQA : Disentangling reasoning from vision and language understanding. NeurIPS, 2018

    2018

  73. [81]

    Self-chained image-language model for video localization and question answering

    Yu, S., Cho, J., Yadav, P., and Bansal, M. Self-chained image-language model for video localization and question answering. In NeurIPS, 2023 a

    2023

  74. [82]

    ANetQA : A large-scale benchmark for fine-grained compositional reasoning over untrimmed videos

    Yu, Z., Zheng, L., Zhao, Z., Wu, F., Fan, J., Ren, K., and Yu, J. ANetQA : A large-scale benchmark for fine-grained compositional reasoning over untrimmed videos. In CVPR, 2023 b

    2023

  75. [83]

    S., Cao, J., Farhadi, A., and Choi, Y

    Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., and Choi, Y. Merlot: Multimodal neural script knowledge models. NeurIPS, 2021

    2021

  76. [84]

    Socratic models: Composing zero-shot multimodal reasoning with language

    Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. Socratic models: Composing zero-shot multimodal reasoning with language. In ICLR, 2023

    2023

  77. [85]

    Video instruction tuning with synthetic data, 2024

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data, 2024. URL https://arxiv.org/abs/2410.02713

    2024 arXiv

  78. [86]

    Where does it exist: Spatio-temporal video grounding for multi-form sentences

    Zhang, Z., Zhao, Z., Zhao, Y., Wang, Q., Liu, H., and Gao, L. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In CVPR, 2020

    2020

  79. [87]

    Video question answering: Datasets, algorithms and challenges

    Zhong, Y., Ji, W., Xiao, J., Li, Y., Deng, W., and Chua, T.-S. Video question answering: Datasets, algorithms and challenges. In EMNLP, 2022

    2022

  80. [88]

    J., and Rohrbach, M

    Zhou, L., Kalantidis, Y., Chen, X., Corso, J. J., and Rohrbach, M. Grounded video description. In CVPR, 2019

    2019

  81. [89]

    Zvonkin, A. K. and Levin, L. A. The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russian Mathematical Surveys, 25 0 (6): 0 83, 1970

    1970

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.