Pith. sign in

REVIEW 3 major objections 5 minor 59 references

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single text-only parse into a logic tree of vision and audio experts can pick the right 16 frames for long-video QA without iterative model calls.

desk verdict A real systems win on compositional multimodal frame selection under tight budgets; the LLM-tree fidelity gap is real but does not erase the empirical case. read the letter →

arxiv 2603.18558 v2 pith:3IK6UIOM submitted 2026-03-19 cs.CV cs.AI

Dan Ben-Ami , Gabriele Serussi , Kobi Cohen , Chaim Baskin This is my paper · ORCID
classification cs.CVcs.AI
keywords VideoQuestionAnsweringFrameSelectionNeuro-SymbolicReasoningMultimodalUnderstandingLong-formFuzzyLogicCompositionTraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-video question answering is bottlenecked by which frames a model is allowed to see under a tight context budget. Similarity-based selectors collapse compositional, cross-modal questions into one embedding and lose order and modality bindings; agentic selectors recover structure only by expensive iterative vision-language calls. HiMu claims that a single text-only language-model call can decompose the question into a hierarchical logic tree of atomic predicates, each scored by a cheap expert (vision, OCR, speech, non-speech audio), then composed with fuzzy temporal operators into a continuous satisfaction curve from which a small set of frames is chosen. Under a fixed 16-frame budget it reports state-of-the-art accuracy among frame-selection methods on three benchmarks, lifts seven different answering models as a drop-in module, and matches the accuracy of uniform sampling that uses four times as many frames, all without retraining or multi-round selection loops. A sympathetic reader cares because the result says compositional multimodal evidence can be recovered at single-shot cost rather than being traded against latency.

What carries the argument

The hierarchical multimodal logic tree: leaves are atomic expert predicates (CLIP, open-vocabulary detection, OCR, ASR, CLAP); internal nodes are continuous fuzzy operators (And, Or, Seq, RightAfter) that produce a satisfaction curve T(t); PASS then spreads selection across peaks of that curve.

What would settle it

Hold the answering model and the 16-frame budget fixed, replace HiMu's tree with deliberately shallow or misrouted trees (or with flat leaf-score summation) on the same Video-MME and LongVideoBench questions, and check whether the reported accuracy gap over uniform sampling and over similarity selectors disappears.

Watch

Extended reading notes

Core claim

Compositional multimodal frame selection for long-form video QA does not require iterative large vision-language inference. A single text-only parse of the query into a hierarchical logic tree, leaves routed to lightweight vision and audio experts, signals normalized and bandwidth-matched, then composed bottom-up by fuzzy operators that enforce co-occurrence, sequence and tight adjacency, yields a per-frame satisfaction curve from which a fixed budget of frames can be selected. Under that regime HiMu is the strongest reported selector on Video-MME, LongVideoBench and HERBench-Lite and matches uniform sampling at four times the frame budget.

Load-bearing premise

The whole pipeline assumes that one text-only language-model parse produces a faithful hierarchical tree of atomic expert predicates; if the tree is shallow or misrouted, the satisfaction curve and selected frames collapse.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. HiMu is a training-free frame selector for long-form VideoQA. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates routed to lightweight vision (CLIP, OVD, OCR) and audio (ASR, CLAP) experts; signals are median/MAD-normalized, bandwidth-smoothed, and composed bottom-up with fuzzy And/Or/Seq/RightAfter operators into a satisfaction curve T(t), from which PASS selects K frames. Under a fixed K=16 budget the method reports SOTA accuracy among frame selectors on Video-MME, LongVideoBench val and HERBench-Lite with Qwen3-VL-8B, improves over uniform sampling across seven LVLMs as a drop-in module, and matches uniform sampling at 4× the frame budget, while remaining cheaper than agentic multi-call systems.

Significance. If the results hold, HiMu meaningfully advances the efficiency–accuracy Pareto front for query-aware frame selection. The design is training-free, caches expert features, incorporates audio as first-class selection evidence, and supplies per-leaf attribution for interpretability—practical strengths for multi-query long-video pipelines. Controlled K=16 comparisons against Uniform/BOLT/AKS/T*, multi-model generalization, expert/composition ablations, frame-budget curves, and FLOPs plots constitute a solid empirical package that other groups can build on or combine with token-compression methods.

major comments (3)
  1. The central claim that hierarchical composition (not merely multi-expert scoring) drives the gains is only partially isolated. Table 3 shows Flat Fusion drops overall accuracy by 5.49 pp, which is the largest ablation, but Flat Fusion still aggregates the same multi-modal leaves; there is no oracle-tree upper bound, no human audit of operator choice/routing/nesting depth, and no stratified breakdown on the temporal-order or cross-modal subsets that the introduction and abstract advertise as the motivating failure mode of global embeddings. Without that evidence it remains possible that a non-negligible fraction of trees are near-flat and that reported gains over BOLT/AKS/T* partly reflect multi-expert scoring rather than true hierarchical temporal logic.
  2. Default experiments use the same LLM family for tree parsing and answering (Sec. 4.1). Supplementary Table 8 shows only ~1 pp end-to-end variance across four parsers, yet that measures final QA accuracy, not tree fidelity. The Limitations section itself states that malformed or shallow trees misroute predicates; a modest human audit or oracle-tree experiment on a compositional subset would substantially strengthen the causal link between the neuro-symbolic design and the SOTA numbers.
  3. Asymmetric comparisons to agentic systems (VideoZoomer 128 frames, VideoChat-A1 512 frames, VSLS 32 frames) in Table 2 are informative for the Pareto narrative but place HiMu at a numerical advantage that is not fully controlled. The paper should either re-run those systems at K=16 with the same backbone or more carefully qualify the claim that 16 compositionally selected frames are “more effective” than hundreds of densely sampled frames.
minor comments (5)
  1. Figure 2 FLOPs axis and Table 1 latency numbers for baselines are estimated rather than re-measured on the same hardware; a short note on estimation methodology would improve reproducibility.
  2. PASS parameters (Np, Nn, w, Δ) are set to floor(sqrt(K)) schedules; a one-sentence justification or sensitivity note in the main text (beyond the supplement) would help readers.
  3. Abstract and introduction use both MLLM and LVLM; pick one acronym for consistency.
  4. Eqs. (5)–(8) for Seq and RightAfter are clear, but a short worked numerical example of a two-step Seq would aid readers unfamiliar with fuzzy temporal operators.
  5. HERBench-Lite absolute numbers remain low (~43 %); a brief discussion of the “fusion deficit” relative to oracle frame sets would contextualize the modest absolute gains.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: HiMu’s claims are empirical frame-selection gains on external benchmarks, not results forced by definition or self-citation.

full rationale

HiMu is a training-free systems paper. Its load-bearing chain is operational, not algebraic: a single text-only LLM call builds a hierarchical logic tree (Sec. 3.1); leaves are scored by cached modality experts (CLIP, OVD, OCR, ASR, CLAP); signals are normalized/smoothed (Eqs. 1–2) and composed with standard fuzzy And/Or/Seq/RightAfter operators (Eqs. 3–8) into T(t); PASS then picks K frames for a downstream LVLM. Accuracy is measured against external ground-truth answers on Video-MME, LongVideoBench, and HERBench-Lite under a fixed K=16 budget, with controlled baselines (Uniform, BOLT, AKS, T*) on the same backbone and ablations (Flat Fusion, leave-one-expert, frame budget, parser/backbone swaps). Nothing in that chain defines the reported accuracy in terms of a fitted target or renames a known result as a first-principles prediction. Using the same LLM family for tree parsing and answering is a design choice, not a circular reduction: selection still produces frames that are scored by independent experts, and end-to-end accuracy remains an external measurement (Supp. Table 8 shows ~1 pp variance across parsers). Author-related HERBench is one of three benchmarks and is not required for the Video-MME / LongVideoBench SOTA or 4×-budget claims. No uniqueness theorem, ansatz-via-self-citation, or fitted-input-as-prediction pattern appears. Score 0 is the honest finding.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

HiMu is an empirical systems paper. Its load-bearing content is not a theorem but a pipeline whose success depends on a small set of hand-chosen processing constants, standard pretrained experts, and the assumption that one LLM-generated logic tree is a faithful enough query representation. No new physical entities are introduced; the invented objects are algorithmic constructs.

free parameters (4)
  • sigmoid sharpness gamma = 3.0
    Controls contrast in median/MAD normalization of expert scores; default 3.0 is chosen by hand.
  • RightAfter decay kappa = 2.0
    Sets how tightly cause and effect must co-occur temporally; default 2.0 is hand-chosen.
  • modality smoothing bandwidths sigma = 0.5 / 1.5 / 2.0
    Per-expert Gaussian widths (CLIP/OVD/OCR 0.5, ASR 1.5, CLAP 2.0) align modalities and affect conjunctions.
  • PASS peak/neighbor/window schedule = floor(sqrt(K)) family
    Np, Nn, w, and Delta are set as floor(sqrt(K)) style rules rather than derived; they control diversity of selected frames.
assumptions (4)
  • domain assumption A single text-only LLM can decompose a natural-language video question into a hierarchical logic tree of atomic expert predicates.
    This is the first pipeline stage and is treated as reliable enough for selection; limitations explicitly warn about malformed trees.
  • ad hoc to paper Product t-norm, probabilistic sum, Seq, and RightAfter are adequate continuous surrogates for the intended logical/temporal constraints.
    These fuzzy operators are design choices that define the satisfaction curve; alternatives are not exhaustively justified.
  • domain assumption Cached lightweight experts (CLIP, OVD, OCR, ASR, CLAP) provide sufficiently accurate atomic evidence for frame ranking without LVLM scoring during selection.
    The method's efficiency and accuracy claims rest on these pretrained models being good enough leaf scorers.
  • domain assumption Under a fixed small frame budget, better compositional selection of frames is the main bottleneck for long-video QA accuracy.
    This motivates the whole paper and is supported by budget curves, but remains an empirical systems assumption rather than a proof.
invented entities (3)
  • hierarchical multimodal logic tree over expert leaves
    purpose: Represent compositional multimodal queries as an evaluable program instead of one global embedding.
    Central algorithmic object introduced by the paper; not an external physical entity, but a new intermediate representation.
  • PASS (Peak-And-Spread Selection)
    purpose: Convert the continuous satisfaction curve into a diverse K-frame set with local motion context.
    A paper-specific selection procedure; ablated only lightly against top-K.
  • per-frame satisfaction curve T(t)
    purpose: Aggregate fuzzy composition into a ranking signal for frame selection.
    Defined by the pipeline operators; useful construct, not independently measured outside the method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering." pith.science (2026). https://pith.science/paper/3IK6UIOM

@misc{pith2026260318558,
  author       = {Pith},
  title        = {Pith review of: HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IK6UIOM}},
  note         = {Machine review of arXiv:2603.18558}
}
abstract

Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows. Within the controlled frame-budget regime that governs practical deployment, prior selectors score frames against a single global query embedding; as a result, compositional multimodal questions that involve temporal ordering or cross-modal cues such as ``what happens on screen right after the narrator mentions the reaction?'' are flattened into a representation that loses sub-event ordering and modality bindings. We introduce \textbf{HiMu}, a training-free framework for compositional multimodal frame selection. A single text-only LLM call decomposes the query into a hierarchical logic tree whose leaves are atomic predicates, each routed to a lightweight expert spanning vision (CLIP, open-vocabulary detection, OCR) and audio (speech recognition and non-speech sound matching). Expert signals are normalized, smoothed to align across modalities, and composed bottom-up through fuzzy-logic operators that enforce temporal sequencing and adjacency, yielding a continuous per-frame satisfaction curve. Under the standard 16-frame budget on Video-MME, LongVideoBench, and HERBench-Lite, HiMu achieves state-of-the-art accuracy among frame selection methods and improves over uniform sampling across seven diverse MLLMs as a drop-in module, matching the accuracy of uniform sampling at $4\times$ its frame budget, without retraining and without multiple iterative MLLM calls during selection.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 6 linked inside Pith

  1. [1]

    Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, et al. Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    arXiv 2025

  2. [2]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qin- glong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)...

    2024

  3. [3]

    GPT-4o system card.arXiv preprint arXiv:2410.21276, 2024

    OpenAI, Aaron Hurst, Adam Lerer, et al. GPT-4o system card.arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  4. [4]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763, 2021

    2021

  5. [5]

    Sigmoid loss for lan- guage image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for lan- guage image pre-training. InProceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 11975–11986, October 2023

    2023

  6. [6]

    Bolt: Boost large vision-language model without training for long-form video understanding

    Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3318–3327, June 2025

    2025

  7. [7]

    Adaptive keyframe sampling for long video understanding

    Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 29118–29128, June 2025

    2025

  8. [8]

    Mdp3: A training-free approach for list-wise frame selection in video-llms

    Hui Sun, Shiyin Lu, Huanyu Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Ming Li. Mdp3: A training-free approach for list-wise frame selection in video-llms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 24090–24101, October 2025

    2025

Show all 59 references
  1. [9]

    Videoagent: A memory-augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. InComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29 – October 4, 2024, Proceedings, Part XXII, pa...

    2024

  2. [10]

    Videoagent: Long-form video understanding with large language model as agent

    Xuan Wang, Yiming Zhang, Omer Zohar, and Sivan Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. InEuropean Conference on Computer Vision (ECCV), pages 58–76, 2024

    2024

  3. [11]

    LVAgent: Long video understanding by multi-round dynamical collaboration of MLLM agents

    Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, and Yali Wang. LVAgent: Long video understanding by multi-round dynamical collaboration of MLLM agents. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20237–20246, Oc...

    2025

  4. [12]

    Longvideoagent: Multi-agent reasoning with long videos.arXiv preprint arXiv:2512.20618, 2025

    Runtao Liu, Ziyi Liu, Jiaqi Tang, Yue Ma, Renjie Pi, Jipeng Zhang, and Qifeng Chen. Longvideoagent: Multi-agent reasoning with long videos.arXiv preprint arXiv:2512.20618, 2025

    2025

  5. [13]

    SeViLA: Self-chained video localization and answering via llm

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. SeViLA: Self-chained video localization and answering via llm. InAdvances in Neural Information Processing Systems, 2023

    2023

  6. [14]

    A.i.r.: Enabling adaptive, iterative, and reasoning-based frame selection for video question answering

    Yuanhao Zou, Shengji Jin, Andong Deng, Youpeng Zhao, Jun Wang, and Chen Chen. A.i.r.: Enabling adaptive, iterative, and reasoning-based frame selection for video question answering. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  7. [15]

    Video-MME: The first-ever compre- hensive evaluation benchmark of multi-modal LLMs in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever compre- hensive evaluation benchmark of multi-modal LLMs in video analysis. InAdvances in Neural Information Processin...

    2024

  8. [16]

    LongVideoBench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A benchmark for long-context interleaved video-language understanding. InAdvances in Neural Information Processing Systems, volume 37, pages 28828–28857, 2024

    2024

  9. [17]

    HERBench: A benchmark for multi-evidence integration in video question answering

    Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, and Chaim Baskin. HERBench: A benchmark for multi-evidence integration in video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  10. [18]

    Frame-voyager: Learning to query frames for video large language models

    Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, and Qianru Sun. Frame-voyager: Learning to query frames for video large language models. InThe Thirteenth International Conference on L...

    2025

  11. [19]

    Flexible frame selection for efficient video reasoning

    Shyamal Buch, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. Flexible frame selection for efficient video reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29071–29082, June 2025

    2025

  12. [20]

    M-LLM based video frame selection for efficient video understanding

    Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, and Trishul Chilimbi. M-LLM based video frame selection for efficient video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2025

  13. [21]

    End- to-end videoqa with frame scoring mechanisms and adaptive sampling

    Jianxin Liang, Xiaojun Meng, Yueqian Wang, Chang Liu, Qun Liu, and Dongyan Zhao. End- to-end videoqa with frame scoring mechanisms and adaptive sampling. InNatural Language Processing and Chinese Computing (NLPCC 2025), volume 16103 ofLecture Notes in Com- puter Science, pages...

    2025

  14. [22]

    Re-thinking temporal search for long-form video understanding

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding. InProceedings of the IEEE/CVF...

    2025

  15. [23]

    YOLO- World: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. YOLO- World: Real-time open-vocabulary object detection. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024

    2024

  16. [24]

    Logic-in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding

    Weiyu Guo, Ziyang Chen, Shaoguang Wang, Jianxiang He, Yijie Xu, Jinhui Ye, Ying Sun, and Hui Xiong. Logic-in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. InThe Thirty-ninth Annual Conference on Neural Information Proces...

    2025

  17. [25]

    Neus-qa: Grounding long-form video understanding in temporal logic and neuro-symbolic reasoning.arXiv preprint arXiv:2509.18041, 2025

    Sahil Shah, S P Sharan, Harsh Goel, Minkyu Choi, Mustafa Munir, Manvik Pasula, Radu Marculescu, and Sandeep Chinchali. Neus-qa: Grounding long-form video understanding in temporal logic and neuro-symbolic reasoning.arXiv preprint arXiv:2509.18041, 2025

    2025

  18. [26]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. InIEEE International Conference on Acoustics, Speech and Signal Processing (IC...

    2023

  19. [27]

    VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2025

  20. [28]

    Videozoomer: Reinforcement-learned temporal focusing for long video reasoning

    Yang Ding, Xin Lai, Yizhen Zhang, Wei Li, Ruihang Chu, and Yujiu Yang. Videozoomer: Reinforcement-learned temporal focusing for long video reasoning. InThe Fourteenth Inter- national Conference on Learning Representations, 2026

    2026

  21. [29]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. LongVU: Spatiote...

    2025

  22. [30]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, ...

    2024

  23. [31]

    docTR: Document text recognition, 2021

    Mindee. docTR: Document text recognition, 2021. Open-source OCR library

    2021

  24. [32]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InProceedings of the 40th International Conference on Machine Learning, pages 28492–28518, 2023. 15

    2023

  25. [33]

    Data filtering networks

    An Fang, Andrew M Jose, Anmol Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. InInternational Conference on Learning Representations, 2024

    2024

  26. [34]

    VideoChat-A1: Thinking with long videos by chain-of-shot reasoning.arXiv preprint arXiv:2506.06097, 2025

    Zikang Wang, Boyu Chen, Zhengrong Yue, Yi Wang, Yu Qiao, Limin Wang, and Yali Wang. VideoChat-A1: Thinking with long videos by chain-of-shot reasoning.arXiv preprint arXiv:2506.06097, 2025

    2025

  27. [35]

    Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  28. [36]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  29. [37]

    EasyOCR: Ready-to-use OCR with 80+ supported languages, 2020

    JaidedAI. EasyOCR: Ready-to-use OCR with 80+ supported languages, 2020

    2020

  30. [38]

    LLaVA-OneVision-1.5: Fully open framework for democratized multimodal training

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. LLaVA-One...

    2025 arXiv

  31. [39]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gemini Team, Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    2025 arXiv

  32. [40]

    InternVL3.5: Advancing open-source multi- modal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multi- modal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025. 16 HiMu:...

    2025 arXiv

  33. [41]

    person

    ** OVD ** - Open - V o c a b u l a r y Object D e t e c t i o n ( YOLO - World ) - For : Ph ysi ca l objects , people , and visual a t t r i b u t e s - E xa mpl es : " person " , " car " , " dog " , " red car " , " man in suit " - S up por ts a t t r i b u t e + noun phrases ...

  34. [42]

    Exit " ,

    ** OCR ** - On - Screen Text R e c o g n i t i o n - For : Text visible on screen - signs , labels , jersey numbers , names , s c o r e b o a r d s - E xa mpl es : " Exit " , "10" , " Warning " , " Korea "

  35. [43]

    person sp ea kin g

    ** CLIP ** - Sem an tic Visual U n d e r s t a n d i n g - For : Actions , scenes , visual states , atmosphere , abs tr ac t visual c onc ep ts - CLIP is VISUAL ONLY - queries must de sc rib e s o m e t h i n g you can SEE in a video frame - Good : " person sp ea kin g " , " d...

  36. [44]

    Add an ASR leaf with related spoken ke yw ord s a l o n g s i d e visual leaves

    ** ASR ** - Speech R e c o g n i t i o n - For : Spoken words , dialogue , narration , verbal r e f e r e n c e s - Use SHORT k eyw or ds (1 -3 words ) , never full s e n t e n c e s - C RI TIC AL : People TALK about what is shown . Add an ASR leaf with related spoken ke yw or...

  37. [45]

    do or bel l ringing

    ** CLAP ** - E n v i r o n m e n t a l Audio Events - For : Non - speech sounds , music , sound effects , ambient audio - E xa mpl es : " do or bel l ringing " , " ap pl aus e " , " glass b re aki ng " [/ IF_CLAP ] ### THE O P E R A T O R S - ** AND **: All ch ild re n must co...

  38. [46]

    Never make a tree with only one expert type .] [ ELSE : Use mu lt ipl e visual experts when po ssi bl e .] [ IF_ASR ]

    M U L T I M O D A L : Each MCQ option should combine more then one expert type , [ IF_ASR : visual AND audio ev id en ce when pos si bl e . Never make a tree with only one expert type .] [ ELSE : Use mu lt ipl e visual experts when po ssi bl e .] [ IF_ASR ]

  39. [47]

    [/ IF_ASR ]

    ASR OVERLAP : Add ASR leaves with short ke yw ord s a l o n g s i d e visual leaves - n a r r a t o r s often de scr ib e what is shown . [/ IF_ASR ]

  40. [48]

    MCQ S T R U C T U R E : AND ( shared_context , OR ( opt_1 , opt_2 , ...) ) - factor shared el em ent s OUT of the OR

  41. [49]

    D E C O M P O S E RICH D E S C R I P T I O N S : Create s ep ara te leaves for each element : OVD for objects / people , CLIP for se tt ing s / states

  42. [50]

    SEQ ONLY FOR KNOWN ORDER

  43. [51]

    NAMES -> OCR [ IF_ASR : + ASR ( spoken ) ]

  44. [52]

    VISUAL STATES -> CLIP , not ASR alone

  45. [53]

    Same " ,

    META - OPTIONS : " Same " , " All of the above " , etc . -> ALWAYS skip in the OR

  46. [54]

    ACTIONS IN OPTIONS : AND ( OVD : object , CLIP : action )

  47. [55]

    TE MPO RA L CAUSE : cause child = action ( CLIP ) , not person

  48. [56]

    [ IF_ASR ]

    O V E R L A P P I N G EXPERTS e n c o u r a g e d . [ IF_ASR ]

  49. [57]

    VISUAL G R O U N D I N G : Never build ASR - only options

  50. [58]

    O V E R L A P P I N G P R E D I C A T E S : Mix experts with o v e r l a p p i n g terms for robust d e t e c t i o n

  51. [59]

    op ": " AND

    S U B T I T L E S -> ASR + OCR . [/ IF_ASR ] ### OUTPUT FORMAT Return a single JSON object : {" op ": " AND "|" OR "|" SEQ "|" R I G H T _ A F T E R "|" LEAF " , " chi ld ren ": [...] , " expert ": < a v a i l a b l e experts > , " query ": " string "} The prompt also includes...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.