Pith. sign in

REVIEW 3 major objections 5 minor 39 references

This paper shows that interleaving reasoning traces with machine-checkable symbolic anchors—and backtracking on violation—cuts error snowballing in multimodal language models from 65% to 14% with under 18% inference overhead.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:07 UTC pith:3UCL25OL

load-bearing objection The empirical core is real and worth referee time; the BCP formalism overreaches, but the central ablation holds. the 3 major comments →

arxiv 2607.16727 v1 pith:3UCL25OL submitted 2026-07-18 cs.AI cs.CV

Constraint-Anchored Reasoning Traces

Zehua Cheng , Wei Dai , Jiahao Sun This is my paper
classification cs.AI cs.CV
keywords multimodal reasoningchain-of-thoughterror correctionneuro-symbolic systemsconstraint propagationvisual question answeringlarge language modelserror snowballing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the dominant failure mode of chain-of-thought reasoning in multimodal models—error snowballing, where one early mistake corrupts all subsequent steps—can be arrested by a neuro-symbolic feedback loop. It introduces constraint-anchored reasoning traces (CART): the model is trained to interleave ordinary language steps with lightweight symbolic assertions about the image (e.g., count(red_objects)=3). A dual verifier checks each assertion against extracted visual features and checks the accumulated set for logical consistency; on any violation, a backtrack controller rewinds to the last consistent checkpoint. The central claim is that this active, mid-reasoning verification—not just training on annotated traces—drives the snowball rate down from 0.65 to 0.14 across benchmarks and adds at most 18% latency. A formal bound (Theorem 2) predicts exponential reduction in snowball rate with the number of post-error anchors, and the empirical results are consistent with that mechanism.

Core claim

The central discovery is that error cascades in autoregressive multimodal reasoning can be reliably interrupted by making the model emit 'anchors'—small symbolic facts about the visual scene—that an external module can verify as they arise. When a generated anchor fails a neural visual-grounding check or contradicts previously accepted anchors via Boolean constraint propagation, generation halts and rewinds to the last consistent state, so a single early mistake no longer poisons the whole trace. The paper shows this in open-source multimodal LLMs fine-tuned on 218K constraint-annotated traces: the snowball rate falls from 0.65 to 0.14, end-task accuracy on compositional visual QA rises by 4

What carries the argument

The central object is the constraint anchor—a bounded-arity symbolic assertion about the image drawn from a finite-domain constraint language—and the Constraint Propagation Module (CPM) that verifies it. The CPM combines a learned neural grounding head (which checks an anchor against mean-pooled region features) with Boolean Constraint Propagation (BCP, a unit-propagation-based satisfiability check of the accumulated anchor set). A backtrack controller reverts generation to the most recent satisfied anchor upon violation, and a variable-frequency emission mechanism lets the model decide how many anchors to emit per step. The load-bearing theoretical result is Theorem 2, which bounds the snow

Load-bearing premise

The load-bearing premise is that the Boolean consistency check—implemented as unit propagation—is a complete test for the paper's finite-domain anchor constraints; if contradictory anchor sets pass undetected, the effective false-acceptance rate rises and the snowball-reduction bound degrades.

What would settle it

Generate random sets of anchors from the paper's constraint language with planted contradictions and measure the fraction on which unit propagation fails to derive the empty clause; if that fraction exceeds the paper's assumed δ≈0.07, the Theorem 2 bound no longer holds at the stated parameters.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • CART reduces the snowball rate from 0.65 to 0.14 on compositional visual QA, and the ablation without backtracking shows detection alone is not enough.
  • Runtime verification adds 10–18% latency, whereas sampling-based self-consistency with 5 chains costs 5×; the paper argues this makes active verification an order of magnitude more efficient than inference-time scaling.
  • The framework transfers zero-shot to benchmarks unseen in training (89.1 F1 on an object-hallucination test and 54.8 on an open-ended evaluation), indicating the emission/verification behavior is a general capability, not a dataset artifact.
  • Theorem 2 and its corollaries imply exponential snowball reduction with post-error anchor count, and a no-harm guarantee bounds false-alarm cost on easy instances to under one percentage point.
  • The ablation table shows every component—backtrack controller, BCP logic, neural grounding, variable-frequency emission—contributes; removing any one degrades accuracy and raises the snowball rate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A plausible extension of the anchor/backtrack mechanism to non-visual reasoning (e.g., arithmetic or fact-checking) is suggested by the fact that the verifier only needs a finite-domain constraint language and an external grounder; the same variable-frequency emission policy could learn when such checks pay off in other modalities.
  • The appendix's β-mixing analysis implies the independence assumption hides real correlation between anchor verifications; the independence bound (0.14) is likely optimistic relative to the dependence-adjusted bound (≈0.28), so tightening cross-anchor independence during training is a concrete next lever.
  • Replacing unit propagation with a complete SAT solver for the same constraint language would directly test how much of the stated guarantee depends on the completeness of the consistency check; if the bound tightens, that is a low-cost system upgrade.
  • The paper's reliance on a frozen open-vocabulary detector for entity grounding means the effective false-acceptance rate is partly inherited from detector quality; as grounding modules improve, the system should scale without retraining the generator.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Constraint-Anchored Reasoning Traces (CART), a neuro-symbolic framework for multimodal LLMs that interleaves natural-language reasoning steps with machine-checkable symbolic constraint anchors. A Constraint Propagation Module (CPM), combining a learned neural grounding head and Boolean Constraint Propagation (BCP), verifies each anchor against the image and against previously accepted anchors; on violation, a backtrack controller rewinds to the last consistent checkpoint. The authors construct 218K training instances from GQA, CLEVR-CoGenT, and VCR, fine-tune LLaVA-NeXT and Qwen2-VL via LoRA, and report large gains on five benchmarks, a reduction in snowball rate from 0.65 to 0.14, and 10--18% inference overhead. They also provide a theoretical analysis (Theorems 2--5 and Proposition 1) bounding the snowball rate under assumptions on CPM reliability, anchor-error coupling, and backtrack recovery.

Significance. If the principal empirical findings hold, CART is a practically attractive and original intervention: it addresses error snowballing at the source, with a clean ablation isolating runtime verification from training-only effects, and it ships with reproducible-style detail (218K data construction, hyperparameters, diagnostic metrics). The theoretical framework is also a useful formalization of error-arresting mechanisms. However, the current formal claim about BCP completeness (Proposition 1) is incorrect as stated, and some headline numbers (snowball-base rate, inference overhead) are internally inconsistent. The empirical core appears defensible, but the paper's formal and quantitative presentation needs revision before the claims can be accepted.

major comments (3)
  1. [§3.2.3 / Prop. 1] Proposition 1 asserts that checking satisfiability of Σ_t∪{c} via Boolean Constraint Propagation runs in O(|Σ_t|·D), and the proof treats unit propagation as a complete satisfiability test for L_c. Unit propagation is not complete for general CNF; it is complete only for restricted classes such as Horn clauses. L_c includes finite-domain predicates whose natural encodings require large positive clauses (e.g., exactly-one constraints over D values) and relational axioms (e.g., antisymmetry of spatial relations), which are not Horn. An unsatisfiable constraint store can therefore yield no empty clause under unit propagation. As implemented, BCP is sound but incomplete, so the advertised 'mutual logical consistency' gate is weaker than claimed. The effective false-acceptance rate δ in Assumption 1 is not justified by Proposition 1 unless δ is measured end-to-end, as the appendix does. Pleas
  2. [Corollary 3 / §5.1] The numerical bound S_CART ≤ 0.140 in Corollary 3 is computed from γ=0.72, δ=0.07, ε_rec=0.12, and S0=0.68, all estimated on the same GQA diagnostic subset whose measured S_CART is 0.14. The agreement is therefore by construction, not by prediction. The appendix honestly labels this an internal-consistency check, but the main text (Section 5.1) says 'validating Theorem 2', and the abstract/conclusion implicitly treat the numerical agreement as evidence. The main text should state this circularity explicitly and, if the theorem is to be empirically supported, estimate parameters on one split and evaluate the bound on a disjoint split.
  3. [Table 2 / Table 3 / Abstract] The abstract claims CART adds 'at most 18% inference overhead', and Table 3 reports a 1.12× factor for LLaVA-NeXT-13B. However, Table 2 (same backbone, GQA diagnostic subset) reports base latency 1.82 s and CART Full latency 2.41 s, i.e., an overhead factor of about 1.32×. The baseline latencies also differ between Table 2 (1.82 s) and Table 3 (2.15 s) for the same model. As written, the 'at most 18%' claim is contradicted by the paper's own Table 2. Please reconcile these measurements or qualify the overhead claim to the specific profiling conditions of Table 3.
minor comments (5)
  1. [§4.3 and App. C.1] The diagnostic subset is described as 2,000 instances in the main text and Table 2, but Appendix C.1 states a 1,000-instance subset per benchmark and bases confidence intervals and permutation tests on 1,000. Please harmonize these numbers.
  2. [Abstract / Assumption 6 / Corollary 3] The base snowball rate appears as 0.65 in the abstract and Section 5.1, as S0=0.68 in Corollary 3, and as pass-through SPT0=0.61 in Assumption 6. Please clarify which quantity is which and use consistent notation in the main text.
  3. [Table 2, A4 row] The entry '0.318.216.3 2.05' appears malformed; the snowball/CVR/latency columns for the 'w/o Neural Grnd' row need proper formatting.
  4. [Appendix A.5, proof of Prop. 1] The proof states that converting a constraint to CNF yields O(D) literals and then appeals to a 'standard result' that unit propagation terminates after |Σ|·D literal inspections. Without an explicit encoding of L_c (including exactly-one constraints and relation axioms), this claim is not checkable, and the standard result does not imply completeness.
  5. [App. D, <RETRY> token] The discussion of the randomly initialized <RETRY> embedding before fine-tuning is important and non-obvious; consider moving a brief version into Section 3.2.4, since readers will otherwise wonder how a new token can condition generation before training.

Circularity Check

0 steps flagged

No significant circularity: the empirical results are externally benchmarked, and the theoretical bound is explicitly an internal-consistency check rather than an out-of-sample prediction.

full rationale

The paper's derivation chain is not circular. The headline numbers (snowball 0.65→0.14, +4.6 GQA, POPE 89.1) are measurements on external benchmarks with a fully specified training and inference pipeline; the ablations are subtractive and isolate component contributions. Theorem 2 is a conditional mathematical bound: under explicit assumptions on CPM reliability (δ,α), error–anchor coupling γ, recovery ε_rec, and pass-through snowball S0, it derives SCART ≤ ε_rec + (S0−ε_rec)(1−γ(1−δ))^K. The proof is a standard total-probability decomposition and does not take the measured SCART as an input. Corollary 3 explicitly warns that the agreement with the empirical SCART=0.14 is 'not out-of-sample validation' and 'should therefore be read as an internal-consistency check on the analysis rather than an independent confirmation of the empirical result.' The §5.1 phrase 'validating Theorem 2' is an overstatement of this internal check, but it is not a circular step because the bound is not algebraically forced to equal the measurement. The BCP/satisfiability question in Proposition 1 is a correctness and completeness concern (unit propagation is incomplete for general CNF and the paper does not establish that L_c encodes to a class for which UP decides satisfiability), but that is not a circularity—it does not reduce outputs to inputs. There are no load-bearing self-citations, no fitted parameter renamed as a prediction, and no ansatz smuggled through the authors' own prior work. The central claims stand on the external benchmarks and the disclosed internal-consistency check is appropriately labeled.

Axiom & Free-Parameter Ledger

6 free parameters · 8 axioms · 0 invented entities

The central claim depends on roughly six measured or tuned parameters and eight domain assumptions, several of which are empirically estimated rather than proven; the BCP completeness assumption is unsupported.

free parameters (6)
  • visual grounding threshold τ_v = 0.85
    Selected on held-out GQA-val by maximizing F1 of violation detection; controls CPM accept/reject behavior (§B.1).
  • retry budget B = 3
    Chosen from GQA-val pilot ablation over {1,2,3,5,10}; affects ε_rec in Assumption 4 (§3.2.4).
  • loss weights λ_c, λ_b = 0.5, 0.3
    Grid-searched on GQA-val over {0.1,0.3,0.5,1.0}^2 (§B.1).
  • LoRA rank = 64
    Ablated on GQA-val (rank 32: -1.1%; rank 128: +0.2% within noise) (§B.1).
  • predicate set size = 14
    Selected from coverage ablation (7 → 66.9%, 14 → 69.4%, 18 → 69.8% GQA) (§5.1).
  • snowball-bound parameters γ, δ, ε_rec, S0 = 0.72, 0.07, 0.12, 0.68 (GQA)
    Estimated on the same GQA diagnostic subset used to report SCART=0.14; used in Corollary 3 to compute the bound (Appendix C.2).
axioms (8)
  • domain assumption Assumption 1: the CPM is (δ, α)-reliable for anchors
    Soundness/completeness of the grounding head and BCP; justified only by empirical estimates, not by construction (§A.1).
  • domain assumption Assumption 2: conditional independence of anchor verifications
    Simplifies Theorem 2's proof; explicitly acknowledged as idealization and relaxed via β-mixing in §A.4.
  • domain assumption Assumption 3: error-anchor coupling probability ≥ γ
    Assumes each post-error anchor is false with probability γ; central to exponential improvement; empirically estimated at 0.58–0.81 (§A.1).
  • domain assumption Assumption 4: backtrack recovery bounds snowball by ε_rec
    Assumes recovery failure probability is bounded; implicitly depends on retry budget B (§A.1).
  • domain assumption Assumption 5: error-free trace ⇒ correct answer; snowball ⇒ wrong answer
    Idealizes trace correctness to answer correctness (§A.1).
  • domain assumption Assumption 6: pass-through snowball rate S_PT0 ≤ S0
    Needed so Theorem 2's Case 1 can use the base model's S0; measured as 0.61 vs 0.68 on GQA (§A.1).
  • ad hoc to paper Unit propagation is a complete satisfiability check for the finite-domain constraint language L_c
    Proposition 1 asserts satisfiability checking via BCP in O(|Σ|·D), but unit propagation is incomplete for general CNF and the paper does not show L_c is Horn (§3.2.3, App. A.5).
  • domain assumption Top-1 Grounding DINO boxes + RoI-Align mean-pooled features carry enough visual information for the grounding head to verify relational/attribute anchors
    The visual grounding check rests on these features (§3.2.2); grounding accuracy is 82.7–89.3% IoU≥0.5, so the premise is approximate.

pith-pipeline@v1.3.0-alltime-deepseek · 24482 in / 20136 out tokens · 178250 ms · 2026-08-01T20:07:25.394815+00:00 · methodology

0 comments
read the original abstract

Autoregressive multimodal large language models (MLLMs) suffer from error snowballing: a single incorrect inference early in a chainof-thought (CoT) trace corrupts all downstream reasoning. We find that in state-of-the-art open-source MLLMs, once the first error occurs, the reasoning cascades into failure across all remaining steps in 65% of such cases (a metric we term the snowball rate). Existing mitigations-sampling multiple chains, post-hoc self-verification, or full program synthesis-either lack symbolic grounding, catch errors too late, or sacrifice the flexibility of natural language reasoning. We propose Constraint-Anchored Reasoning Traces (CART), a neuro-symbolic framework that trains MLLMs to interleave natural language reasoning steps with symbolic constraint assertions: lightweight, machine-checkable statements about visual content (e.g., count(red_objects) = 3). A dual-pronged Constraint Propagation Module-combining a learned neural grounding head with Boolean Constraint Propagation-continuously verifies these anchors against extracted visual features and checks their mutual logical consistency. When a contradiction is detected, a backtrack controller halts generation and reverts to the last consistent checkpoint, preventing error propagation. A variable-frequency emission mechanism allows the model to adaptively control anchor density, avoiding trace bloat. We construct 218K training instances by augmenting GQA, CLEVR-CoGenT, and VCR with ground-truth constraint annotations derived from scene graphs, and fine-tune open-source MLLMs (LLaVA-NeXT, Qwen2-VL) via LoRA. On five benchmarks, CART reduces the snowball rate from 0.65 to 0.14, improves GQA accuracy by +4.6 percentage points over trainingonly baselines, and achieves 89.1 F1 on POPE-all with at most 18% inference overhead.

Figures

Figures reproduced from arXiv: 2607.16727 by Jiahao Sun, Wei Dai, Zehua Cheng.

Figure 1
Figure 1. Figure 1: Overview of the CART framework. Given an image and question, the MLLM generates interleaved reasoning steps [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 7 linked inside Pith

  1. [1]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 39–48

    2016

  2. [2]

    2003.Constraint processing

    Rina Dechter. 2003.Constraint processing. Elsevier

    2003

  3. [3]

    Ailin Deng, Zhirui Lu, Yanhong Chen, Qiuyu Wu, Zhenfei Cao, Lian Wang, and Jiaqi Liu. 2024. Seeing is believing: Mitigating hallucination in large vision- language models via CLIP-guided decoding.arXiv preprint arXiv:2402.15300 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    William F Dowling and Jean H Gallier. 1984. Linear-time algorithms for testing the satisfiability of propositional Horn formulae.The Journal of Logic Programming 1, 3 (1984), 267–284

    1984

  5. [5]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. InInternational Conference on Machine Learning. PMLR, 1321–1330

    2017

  6. [6]

    Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Com- positional visual reasoning without training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14953–14962

    2023

  7. [7]

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. InProceedings of the IEEE international conference on computer vision. 2961–2969

    2017

  8. [8]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations

    2022

  9. [9]

    Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko

  10. [10]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. InProceedings of the CVPR. 6700–6709

    2019

  11. [11]

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. InProceedings of the IEEE conference on computer vision and pattern recognition. 2901–2910

    2017

  12. [12]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. InAdvances in Neural Information Processing Systems, Vol. 35. 22199–22213

    2022

  13. [13]

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al

  14. [14]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen

  15. [15]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowl- edge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  16. [16]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. 2024. Grounding DINO: Mar- rying dino with grounded pre-training for open-set object detection. InECCV. Springer, 38–55

    2024

  17. [17]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. InICCV. 10012–10022

    2021

  18. [18]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, Vol. 35. 2507–2521

    2022

  19. [19]

    Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Mar- ianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning.arXiv preprint arXiv:2301.13379(2023)

    Pith/arXiv arXiv 2023

  20. [20]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  21. [21]

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. 2024. Scaling Open- Vocabulary Object Detection.Advances in Neural Information Processing Systems 36 (2024)

    2024

  22. [22]

    2021.{Zero-offload}: Democratizing{billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.{Zero-offload}: Democratizing{billion-scale} model training. In2021 USENIX Annual Technical Conference (USENIX ATC 21). 551–564

    2021

  23. [23]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, Vol. 36

    2023

  24. [24]

    InAdvances in Neural Information Processing Systems, Vol

    Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Systems, Vol. 36

  25. [25]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations

    2023

  27. [27]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, Vol. 35. 24824–24837

    2022

  28. [28]

    Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vision. 11888–11898

    2023

  29. [29]

    Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. 2018. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding. InAdvances in Neural Information Processing Systems, Vol. 31

    2018

  30. [30]

    Weihao Yu, Zhengyuan Yao, Dongchao Luo, Junyan Li, Xiaodan Wang, James Park, and Mohit Bansal. 2023. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities.arXiv preprint arXiv:2308.02490(2023)

    Pith/arXiv arXiv 2023

  31. [31]

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. InProceedings of the CVPR. 6720– 6731

    2019

  32. [32]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. InAdvances in Neural Information Processing Systems, Vol. 36

    2023

  33. [33]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923(2023)

    Pith/arXiv arXiv 2023

  34. [34]

    𝐾Ù 𝑖=1 𝑀𝑖 # ≤P

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. 2023. DDCoT: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neural Information Processing Systems36 (2023). ACM MM 2026, October 2026, Dublin, Ireland Cheng et al. A Theoretical Analysis We provide formal guarantees for CART’s error-arresting m...

    2023

  35. [36]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Language model cascades: Token-level uncertainty and beyond.arXiv preprint arXiv:2404.10136 (2023)

    Pith/arXiv arXiv 2023

  36. [39]

    Try a different approach:

    The adjusted bound isSCART≤ 0.12+0.56×( 0.036+0.24)= 0.275, compared to0.140under independence. The empiricalSCART = 0.14 lies below both bounds, suggesting the actual dependence structure is more benign than the worst case. A.5 Computational Complexity Proposition 1 (BCP Tractability).Suppose the constraint lan- guageL𝑐 has maximum variable domain size 𝐷...

    2026

  37. [2017]

    InProceedings of the IEEE International Conference on Computer Vision

    Learning to reason: End-to-end module networks for visual question answering. InProceedings of the IEEE International Conference on Computer Vision. 804–813

  38. [2022]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grounded Language-Image Pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10965–10975

  39. [2023]

    Evaluating Object Hallucination in Large Vision-Language Models.arXiv preprint arXiv:2305.10355(2023)

    Pith/arXiv arXiv 2023