Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EgoAdapt claims that jointly training a lightweight distillation student with a Gumbel-Softmax policy that selects which modalities to process per segment lets a single framework reproduce heavy egocentric-perception teachers at a…

desk verdict A practical efficiency recipe for egocentric perception with a plausible central claim, but the energy multiplier rests on an analytical model and one ablation looks undertrained; worth a serious referee, conditional on code and hardware validation. read the letter →

arxiv 2506.21080 v1 pith:XH4GLOEP submitted 2025-06-26 cs.CV cs.AIcs.LG

Sanjoy Chowdhury , Subrata Biswas , Sayan Nag , Tushar Nagarajan , Calvin Murdock , Ishwarya Ananthabhotla , Yijun Qian , Vamsi Krishna Ithapu
show 2 more authors
Dinesh Manocha Ruohan Gao
This is my paper
classification cs.CVcs.AIcs.LG
keywords egocentricperceptioncross-modaldistillationadaptivemodalityselectionpolicylearningGumbel-Softmaxactivespeakerlocalizationactionrecognitionbehavioranticipation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EgoAdapt tries to establish that cross-modal distillation and adaptive modality selection, each previously used on its own to make perception models cheaper, become far more effective when trained jointly in one differentiable pipeline. The framework pairs a lightweight student that mimics a heavy teacher's features and logits with a policy network that decides, per time segment, which of the available video, audio, and behavioral streams are worth processing, using Gumbel-Softmax sampling so the discrete choices stay trainable by backpropagation. On three egocentric tasks the result is near-teacher accuracy at a fraction of the cost: within 0.14 mAP of the MUST teacher on EasyCom active speaker localization at about nine times fewer MACs, within 0.83 accuracy points of the TIM teacher on EPIC-Kitchens action recognition at about 3.7 times fewer MACs, and the lowest estimated energy of any compared method on Aria Everyday Activities behavior anticipation. The paper's reason to care is deployment: egocentric perception for AR/VR runs on power-, memory-, and thermal-constrained devices, where savings of this size separate a model that fits on the device from one that does not.

What carries the argument

The load-bearing object is the joint training loop over two modules. The distillation student (CFD, $\Phi$) is a set of lightweight encoders for video frames, audio, and behavioral signals, fused by a late-fusion network and trained with three losses: $L_1$ feature matching against the teacher's visual feature, KL-divergence distillation of the teacher's class logits, and ground-truth cross-entropy. The policy network (TeMPLe, $\Pi$) is an LSTM consuming per-segment features, with parallel fully-connected heads producing, for each modality and each audio channel, a binary selection sampled through Gumbel-Softmax, a relaxation that keeps discrete decisions differentiable, plus an efficiency penalty $C_k = (|U_k|_0 / C)^2$ on the fraction of segments that keep a given stream. The three-stage schedule, train $\Phi$, train $\Pi$, then joint finetune on $L_\Theta = \eta_1 L_\Pi + \eta_2 L_\Phi$, is the mechanism that prevents the two learning signals from interfering. For action recognition the policy instead enforces a strict one-frame rule: audio previewing flags up to three salient events, and the policy selects a single representative frame per event.

What would settle it

Power-instrument the EgoAdapt ASL pipeline on an actual device (for example, run the 4-bit and full-precision variants on a mobile GPU or AR-class SoC using EasyCom test clips while measuring end-to-end energy per inference) and compare against the reported 0.003 J per inference and the 9.6x saving over the MUST teacher; if measured energy does not reproduce the modeled reduction within a reasonable factor, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

Stated as the authors would state it: a fully differentiable pipeline can simultaneously shrink a multisensory teacher into a lightweight student and learn, on the fly, which sensors that student should actually read. The Cross-Modal Feature Distillation module trains the student $\Phi$ to approximate the teacher's visual feature $z_V$ (via an $L_1$ loss), its softened logits (via a KL-divergence loss $L_{KD}$), and the task labels (via cross-entropy $L_{GT}$), while the Task-Aware Multisensory Policy Learning network (TeMPLe) runs an LSTM over segment features and emits binary keep/drop decisions per modality, and per audio channel, through Gumbel-Softmax sampling, with a cost term $C_k = (|U_k|_0 / C)^2$ that penalizes excessive use of expensive streams. The three-stage schedule, distill, then policy-train, then joint finetune on $L_\Theta = \eta_1 L_\Pi + \eta_2 L_\Phi$, is presented as the step that reconciles the two learning signals. With it, EgoAdapt reports 89.74 mAP on EasyCom ASL at 0.070 GMACs and 0.39M parameters against the MUST teacher's 89.88 mAP at 0.642 GMACs; 56.74 top-1 action accuracy on EPIC-Kitchens at 7.14 GMACs against TIM's 57.57 at 26.62 GMACs; and the lowest estimated energy (0.003 J) on AEA behavior anticipation. The ablations place the load on the joint stage: full training beats a random-policy variant by about 22 mAP points on ASL, and the stage-3 finetune alone moves ASL mAP from 83.64 to 89.74.

Load-bearing premise

The headline efficiency and energy numbers are produced by an analytical model that sums operation counts, memory read/write costs, and sensor active time, with no hardware measurements behind them, so if real devices price those operations differently, the claimed savings shrink.

Editorial extensions

If this is right

  • One framework spans three egocentric tasks, action recognition, active speaker localization, and behavior anticipation, by changing only the policy's action space, so the modality-gating machinery transfers without architectural redesign.
  • Teacher-level accuracy survives the efficiency cuts: within 0.14 mAP of MUST at about 9x fewer MACs on ASL, and within 0.83 points of TIM at about 3.7x fewer MACs on EPIC-Kitchens.
  • The joint component is the point: full joint training beats a random-policy baseline by about 22 mAP points on ASL, and the final joint finetune alone moves ASL mAP from 83.64 to 89.74.
  • Learned gating doubles as noise robustness: as audio SNR falls from -5 dB to -20 dB on ASL, video usage rises from 20.71% to 88.89% while mAP only drops from 84.27% to 80.21%.
  • The framework is teacher-agnostic: substituting other teachers (MoViNet, MeMViT, MBT for action recognition; MAVASL, TalkNet for ASL) keeps the low-compute profile at corresponding accuracy levels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next check, not run in the paper, is a power-instrumented deployment: the 9.6x energy reduction is computed from an analytical model of operations, memory traffic, and sensor active time, so measuring actual power draw on AR-class hardware would confirm whether the savings survive real memory and sensor costs.
  • The strict one-frame policy for action recognition encodes a strong prior that a single image holds the decisive semantic content of an action; extending the framework to motion-defined tasks such as hand-object interaction or long-horizon activities would show how far that prior generalizes.
  • The policy's reported behavior under audio noise, shifting from audio to video as SNR degrades, suggests an explicit online signal-quality controller could reproduce the gating with a simpler, more inspectable rule than a learned LSTM policy and without the hand-chosen selection threshold.
  • Because the selection threshold $\delta$ is chosen empirically per task, making it a learned or calibrated quantity and evaluating under distribution shift would determine whether the gating decisions generalize beyond the benchmark conditions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. EgoAdapt proposes a unified framework that combines cross-modal feature distillation (CFD) with a task-aware multisensory policy learning module (TeMPLe) for three egocentric perception tasks: action recognition, active speaker localization (ASL), and behavior anticipation. The student model is trained to imitate heavy teacher models (TIM, MUST) while a Gumbel-Softmax-based policy selects modalities, audio channels, or frames per segment. Experiments on EPIC-Kitchens, EasyCom, and AEA report near-teacher accuracy with substantially lower GMACs, parameters, and estimated energy (up to 89.09% GMAC reduction, 82.02% parameter reduction, and 9.6x energy reduction). The paper includes ablations of losses, training stages, modality choices, and teacher model choices, plus a noisy-condition analysis for ASL.

Significance. If the efficiency estimates are trustworthy, EgoAdapt is a useful step toward practical egocentric perception on resource-constrained AR/VR devices. The idea of jointly optimizing a lightweight distillation student and an adaptive modality-selection policy is well motivated, and the paper evaluates on three datasets with multiple SOTA baselines and teacher models, which is a solid empirical scope. The multi-teacher experiments (Supp. E) and the noisy-condition analysis (Supp. H) are valuable and show the framework is not tied to a single teacher. The strengths are the breadth of tasks, the explicit MACs/parameter/energy reporting, and the abalations identifying the role of each loss. The main risks are that the headline energy savings rely on an unvalidated analytical model and that the policy gain appears inflated by an undertrained 'w/o TeMPLe' comparator.

major comments (3)
  1. [Table 2 vs. Table 5] Because this gap directly supports the central claim that adaptive selection matters, it must be resolved before the paper can be accepted.
  2. [Supp. G.2 and Tables 2-3] This issue is load-bearing because the efficiency contribution's headline number is an energy claim, not just a MACs claim.
  3. [Sec. 3.3.4, Eq. (12)] This matters for reproducibility of the joint training procedure, which is the paper's methodological novelty.
minor comments (5)
  1. [Eq. (1) and Eq. (2)] Equation (1) uses M_t^s = argmax_M f_s(...) but f_s is not defined as a learnable saliency function, and Eq. (2) appears syntactically malformed (the expression 'st ∈ argmax_{k∈sti:ti+w} (st) AΠ(t)' is hard to parse); please rewrite both equations.
  2. [Table 3] The trajectory error at T700 ms is slightly worse for EGOADAPT (13.36) than for EGOADAPT w/o TeMPLe (12.98), which is inconsistent with the general claim that the policy always improves performance; please discuss this case.
  3. [Figure 4 caption] The caption states the final model operates at '~5.5x less MACs' but Table 2 shows 0.070 versus 0.642 GMACs, a 9.2x difference; please correct the factor.
  4. [Abstract and Sec. 4.3] The abstract claims the method is 'on-par and in many cases outperforming' SOTA models, but Table 1 shows EGOADAPT is slightly below the TIM teacher on verb (76.65 vs. 77.19), noun (66.83 vs. 67.22), and action (56.74 vs. 57.57); consider phrasing this as 'closely matching' to avoid overclaiming.
  5. [General] No code, seeds, or run-to-run variance is reported; given the small accuracy differences in some comparisons (e.g., 0.14 mAP over the teacher in Table 2), reporting the mean and standard deviation over at least three seeds would substantially strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: distillation targets are external teacher outputs, efficiency is directly counted, and teacher-agnostic ablations keep self-citations non-load-bearing.

full rationale

The derivation chain is self-contained. CFD is trained with LKD, L1, and LGT (Eqs. 3-6) against teacher logits/features and ground-truth labels; TeMPLe is trained with Gumbel-Softmax plus the cost-penalized objective (Eq. 12); the final joint loss (Eq. 13) combines them. None of these equations defines the reported efficiency or accuracy numbers in terms of the paper's conclusions. GMACs are counted with PyTorch's FLOP counter, parameters are counted directly, and energy is the Supp. G.2 analytical sum over MACs, memory traffic, and sampled sensor time, not a number imported from the teacher papers. Accuracy is measured on held-out EPIC-Kitchens/EasyCom/AEA ground truth. The student approaching its teacher (MUST/TIM) is the intended distillation objective, not a hidden reuse of the target result; the paper also shows teacher-agnostic behavior with external teachers (MoViNet, MeMViT, MBT, TalkNet, GazeMLE, GLC) in Supp. E, so the self-citations to MUST and EgoDistill are not load-bearing. The 'w/o TeMPLe' baseline's low mAP relative to fixed modality combinations is a potential experimental-control concern, but it does not make the policy comparisons circular because the reported policy gains are against ground-truth labels and independent compute counts. No step reduces by construction to its own input.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The headline results rest on validated hyperparameters (alpha, beta, eta, lambda, gamma, tau, delta), on the assumption that distillation transfers knowledge from the teacher to a much smaller student, and on the analytical energy model. No new physical or information-theoretic entities are introduced.

free parameters (6)
  • threshold delta
    Used in Eq. (2) to decide frame selection; the paper states it is 'determined through empirical evaluation' with no sensitivity analysis.
  • loss weights alpha, beta = 0.90, 0.85
    Control the balance of KD, feature L1, and ground-truth losses; tuned on validation and ablated in Supp. O.
  • objective weights eta1, eta2 = 0.95, 1.2
    Balance policy and distillation losses in Eq. (13); tuned on validation and ablated in Supp. N.
  • modality costs lambda = [1, 0.05, 0.03] (visual, audio, behavior)
    Trade-off between accuracy and compute in Eq. (12); Table 4 shows the choice matters.
  • penalty gamma = 10
    Penalizes wrong predictions in the policy objective, affecting the accuracy/efficiency trade-off.
  • Gumbel-Softmax temperature tau = 1.0 for Epic-Kitchens, 10.0 for EasyCom
    Controls discreteness of the sampled policy decisions.
assumptions (4)
  • domain assumption Knowledge distillation transfers task-relevant knowledge from the teacher to the lightweight student via KL-divergence and feature L1 losses.
    The method's accuracy depends on the student being able to emulate the teacher from fewer modalities and frames; assumed in Sec. 3.2.
  • standard math Gumbel-Softmax provides a valid, low-variance gradient estimator for the discrete modality-selection decisions.
    Standard result from Jang et al. [52], invoked in Eqs. (8)-(9).
  • domain assumption The analytical energy model (MACs plus memory read/writes plus sensor active time) approximates real device energy consumption.
    The headline 'energy up to 9.6x' is computed from this model (Supp. G.2), not from on-device measurement.
  • domain assumption The chosen teacher models (TIM, MUST) are reliable and their predictions provide good supervision for the downstream tasks.
    The student's accuracy is bounded by the teacher's; any bias in the teacher propagates to the student, and the comparison against SOTA is only as meaningful as the teacher's own standing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception." pith.science (2026). https://pith.science/paper/XH4GLOEP

@misc{pith2026250621080,
  author       = {Pith},
  title        = {Pith review of: EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XH4GLOEP}},
  note         = {Machine review of arXiv:2506.21080}
}
read the original abstract

Modern perception models, particularly those designed for multisensory egocentric tasks, have achieved remarkable performance but often come with substantial computational costs. These high demands pose challenges for real-world deployment, especially in resource-constrained environments. In this paper, we introduce EgoAdapt, a framework that adaptively performs cross-modal distillation and policy learning to enable efficient inference across different egocentric perception tasks, including egocentric action recognition, active speaker localization, and behavior anticipation. Our proposed policy module is adaptable to task-specific action spaces, making it broadly applicable. Experimental results on three challenging egocentric datasets EPIC-Kitchens, EasyCom, and Aria Everyday Activities demonstrate that our method significantly enhances efficiency, reducing GMACs by up to 89.09%, parameters up to 82.02%, and energy up to 9.6x, while still on-par and in many cases outperforming, the performance of corresponding state-of-the-art models.

Figures

Figures reproduced from arXiv: 2506.21080 by the authors.

Figure 1
Figure 1. We introduce EGOADAPT, a unified framework com￾bining cross-modal distillation and policy learning for efficient multisensory egocentric perception. It optimizes the use of con￾stituent modalities for efficient inference and is adaptable to vari￾ous egocentric perception tasks, including action recognition, ac￾tive speaker localization, and behavior anticipation. source constraints, but also to enable faster respons… view at source ↗
Figure 2
Figure 2. Illustration of EGOADAPT. Our framework consists of two main components, a lightweight policy module Π and a distillation module Φ composed of different sub-networks that are trained jointly (via late fusion with learnable weights) for various egocentric tasks, including action recognition, active speaker localization, and behaviour anticipation. The policy module is adaptable, dynamically selecting the optimal moda… view at source ↗
Figure 3
Figure 3. Acc. vs. log(Energy), size ∝ params for AR. EGOADAPT shows best￾to-comparable performances with least com￾pute when compared to SOTA methods [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Qualitative examples of egocentric action recognition on EPIC-Kitchens. The green and red boxes represent correct and incorrect predictions, respectively. EGOADAPT picks the most informative frame to predict the ‘Noun’ classes, which is subsequently used to predict the…
Figure 6
Figure 6. Figure 6: Qualitative examples of ASL on EasyCom. The red/blue boxes indicate active/non-active speakers, and the red heatmap indicates model prediction. EGOADAPT can make correct predictions for scenes with motion blur (col. 4), partial vision (col. 5), and multi-speakers (col.…
Figure 7
Figure 7. Figure 7: Qualitative examples of egocentric behavior antici￾pation on the AEA Dataset. Cross/circle symbols denote previ￾ous/anticipated behaviors. Our model reasonably anticipates fu￾ture behaviors in common scenarios like long-term fixation and human-object interaction. combi…
Figure 8
Figure 8. Figure 8: illustrates the performance of the Active Speaker Lo￾calization (ASL) model under varying audio noise levels, quantified by signal-to-noise ratio (SNR). As the SNR de￾creases from -5 dB to -20 dB (indicating progressively nois￾ier audio conditi‘ons), the model adaptive…
Figure 9
Figure 9. Figure 9: More qualitative examples of Action Recognition on the Epic-Kitchens Dataset. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: More qualitative examples of Active Speaker Localization on the EasyCom Dataset. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Qualitative examples of egocentric behavior anticipation on the AEA Dataset. Cross/circle symbols denote previ￾ous/anticipated behaviors [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Failure case for egocentric action recognition on EPIC-Kitchens dataset. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Failure case for egocentric ASL on EasyCom Dataset. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning

    cs.CV 2025-08 reject novelty 6.0 of 10

    A new audio-visual reasoning benchmark and a two-part metric (factual consistency, core inference) claim to expose a gap between answer accuracy and reasoning quality in AV-LLMs.

Reference graph

Works this paper leans on

130 extracted references · 61 canonical work pages · cited by 1 Pith paper

  1. [1]

    Creating the future: Augmented reality, the next human-machine interface

    Michael Abrash. Creating the future: Augmented reality, the next human-machine interface. In 2021 IEEE Interna- tional Electron Devices Meeting (IEDM), 2021. 17

    2021

  2. [2]

    Sound- net: Learning sound representations from unlabeled video

    Yusuf Aytar, Carl V ondrick, and Antonio Torralba. Sound- net: Learning sound representations from unlabeled video. Advances in neural information processing systems , 29,

  3. [3]

    Conditional computation in neural networks for faster models

    Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015. 2

    arXiv 2015

  4. [4]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas L ´eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. 2

    arXiv 2013

  5. [5]

    Knowledge distillation: A good teacher is patient and consistent

    Lucas Beyer, Xiaohua Zhai, Am ´elie Royer, Larisa Mar- keeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 10925–10934, 2022. 2

    2022

  6. [6]

    Tim: A time interval ma- chine for audio-visual action recognition

    Jacob Chalk, Jaesung Huh, Evangelos Kazakos, Andrew Zisserman, and Dima Damen. Tim: A time interval ma- chine for audio-visual action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18153–18163, 2024. 5, 6, 7, 16, 18

    2024

  7. [7]

    Se- mantic audio-visual navigation

    Changan Chen, Ziad Al-Halah, and Kristen Grauman. Se- mantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15516–15525, 2021. 16

    2021

  8. [8]

    Run, don’t walk: chasing higher flops for faster neural networks

    Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, and S-H Gary Chan. Run, don’t walk: chasing higher flops for faster neural networks. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12021–12031, 2023. 15

    2023

Show all 130 references
  1. [9]

    Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices

    Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze. Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2019. 17

    2019

  2. [10]

    You look twice: Gaternet for dynamic filter selection in cnns

    Zhourong Chen, Yang Li, Samy Bengio, and Si Si. You look twice: Gaternet for dynamic filter selection in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9172–9180, 2019. 2

    2019

  3. [11]

    Listen to the pixels

    Sanjoy Chowdhury, Subhrajyoti Dasgupta, Sudip Das, and Ujjwal Bhattacharya. Listen to the pixels. In 2021 IEEE In- ternational Conference on Image Processing (ICIP), pages 2568–2572. IEEE, 2021. 16

    2021

  4. [12]

    Audvisum: Self-supervised deep reinforcement learning for diverse audio-visual summary generation

    Sanjoy Chowdhury, Aditya Patra, Subhrajyoti Dasgupta, and Ujjwal Bhattacharya. Audvisum: Self-supervised deep reinforcement learning for diverse audio-visual summary generation. In BMVC, page 315, 2021. 16

    2021

  5. [13]

    Adverb: Visually guided audio dereverberation

    Sanjoy Chowdhury, Sreyan Ghosh, Subhrajyoti Dasgupta, Anton Ratnarajah, Utkarsh Tyagi, and Dinesh Manocha. Adverb: Visually guided audio dereverberation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 7884–7896, 2023. 16

    2023

  6. [14]

    Apollo: unified adapter and prompt learning for vision lan- guage models

    Sanjoy Chowdhury, Sayan Nag, and Dinesh Manocha. Apollo: unified adapter and prompt learning for vision lan- guage models. arXiv preprint arXiv:2312.01564, 2023. 16

    2023 arXiv

  7. [15]

    Meerkat: Audio-visual large language model for grounding in space and time

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In European Conference on Computer Vision, pages 52–70. Springer, 2024. 16

    2024

  8. [16]

    Melfusion: Synthesizing music from image and language cues using diffusion mod- els

    Sanjoy Chowdhury, Sayan Nag, KJ Joseph, Balaji Vasan Srinivasan, and Dinesh Manocha. Melfusion: Synthesizing music from image and language cues using diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 26826–26835,

  9. [17]

    Magnet: A multi-agent framework for finding audio-visual needles by reasoning over multi-video haystacks

    Sanjoy Chowdhury, Mohamed Elmoghany, Yohan Abeysinghe, Junjie Fei, Sayan Nag, Salman Khan, Mo- hamed Elhoseiny, and Dinesh Manocha. Magnet: A multi-agent framework for finding audio-visual needles by reasoning over multi-video haystacks. arXiv preprint arXiv:2506.07016, 2025. 16

    2025 arXiv

  10. [18]

    Aurelia: Test-time reasoning distillation in audio-visual llms

    Sanjoy Chowdhury, Hanan Gani, Nishit Anand, Sayan Nag, Ruohan Gao, Mohamed Elhoseiny, Salman Khan, and Di- nesh Manocha. Aurelia: Test-time reasoning distillation in audio-visual llms. arXiv preprint arXiv:2503.23219, 2025. 16

    2025 arXiv

  11. [19]

    Avtrustbench: Assessing and enhancing re- liability and robustness in audio-visual llms

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Yaoting Wang, Mohamed Elhoseiny, Ruohan Gao, and Di- nesh Manocha. Avtrustbench: Assessing and enhancing re- liability and robustness in audio-visual llms. arXiv preprint arXiv:2501.02135, 2025. 16

    2025 arXiv

  12. [20]

    Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100. International Journa...

    2022

  13. [21]

    Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100. International Journa...

    2022

  14. [22]

    Asd-transformer: 9 Efficient active speaker detection using self and multimodal transformers

    Gourav Datta, Tyler Etchart, Vivek Yadav, Varsha Hedau, Pradeep Natarajan, and Shih-Fu Chang. Asd-transformer: 9 Efficient active speaker detection using self and multimodal transformers. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Process...

    2022

  15. [23]

    Episodic memory question answering

    Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh. Episodic memory question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19119–19128, 2022. 1

    2022

  16. [24]

    Forecast- ing action through contact representations from first person video

    Eadom Dessalene, Chinmaya Devaraj, Michael Maynord, Cornelia Ferm ¨uller, and Yiannis Aloimonos. Forecast- ing action through contact representations from first person video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):6703–6714, 2021. 16

    2021

  17. [25]

    Easycom: An augmented re- ality dataset to support algorithms for easy communication in noisy environments

    Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Krishna Ithapu, and Ravish Mehra. Easycom: An augmented re- ality dataset to support algorithms for easy communication in noisy environments. arXiv preprint arXiv:2107.04174 ,

    arXiv

  18. [26]

    Can llms generate human-like wayfinding in- structions? towards platform-agnostic embodied instruc- tion synthesis

    Vishnu Sashank Dorbala, Sanjoy Chowdhury, and Dinesh Manocha. Can llms generate human-like wayfinding in- structions? towards platform-agnostic embodied instruc- tion synthesis. arXiv preprint arXiv:2403.11487, 2024. 16

    2024 arXiv

  19. [27]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichten- hofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 6824–6835, 2021. 1

    2021

  20. [28]

    Modeling actions through state changes

    Alireza Fathi and James M Rehg. Modeling actions through state changes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2579– 2586, 2013. 16

    2013

  21. [29]

    X3d: Expanding architectures for efficient video recognition

    Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 203–213, 2020. 2

    2020

  22. [30]

    Spatially adaptive computation time for residual networks

    Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1039– 1048, 2017. 2

    2017

  23. [31]

    Self-supervised moving vehicle tracking with stereo sound

    Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba. Self-supervised moving vehicle tracking with stereo sound. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 7053–7062,

  24. [32]

    Audio–visual representation learning for anomaly events detection in crowds

    Junyu Gao, Hao Yang, Maoguo Gong, and Xuelong Li. Audio–visual representation learning for anomaly events detection in crowds. Neurocomputing, 582:127489, 2024. 16

    2024

  25. [33]

    Dynamic zoom-in network for fast object detection in large images

    Mingfei Gao, Ruichi Yu, Ang Li, Vlad I Morariu, and Larry S Davis. Dynamic zoom-in network for fast object detection in large images. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 6926–6935, 2018. 3

    2018

  26. [34]

    Listen to look: Action recognition by previewing audio

    Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10457–10467,

  27. [35]

    Modality distillation with multiple stream networks for ac- tion recognition

    Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for ac- tion recognition. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 103–118, 2018. 2, 4

    2018

  28. [36]

    Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceler- ation

    Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceler- ation. Advances in neural information processing systems, 31, 2018. 6, 17

    2018

  29. [37]

    Frameexit: Conditional early exiting for ef- ficient video recognition

    Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhos- sein Habibian. Frameexit: Conditional early exiting for ef- ficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15608–15618, 2021. 2

    2021

  30. [38]

    Omni- vore: A single model for many visual modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omni- vore: A single model for many visual modalities. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16102–16112, 2022. 1, 6

    2022

  31. [39]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Mar- tin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Meng...

    2022

  32. [40]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings...

    2024

  33. [41]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings...

    2024

  34. [42]

    Adaptive computation time for recurrent neu- ral networks

    Alex Graves. Adaptive computation time for recurrent neu- ral networks. arXiv preprint arXiv:1603.08983, 2016. 2

    2016 arXiv

  35. [43]

    Spottune: transfer learning through adaptive fine-tuning

    Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grau- man, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4805–4814, 2019. 3

    2019

  36. [44]

    Cross modal distillation for supervision transfer

    Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2827–2836, 2016. 2, 4

    2016

  37. [45]

    Rethinking imagenet pre-training

    Kaiming He, Ross Girshick, and Piotr Doll ´ar. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4918– 4927, 2019. 1

    2019

  38. [46]

    Ronin: Robust neural inertial navigation in the wild: Bench- mark, evaluations, & new methods

    Sachini Herath, Hang Yan, and Yasutaka Furukawa. Ronin: Robust neural inertial navigation in the wild: Bench- mark, evaluations, & new methods. In 2020 IEEE inter- national conference on robotics and automation (ICRA) , pages 3146–3152. IEEE, 2020. 2

    2020

  39. [47]

    Object-region video transformers

    Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 3148–3159, 2022. 16

    2022

  40. [49]

    Distill- ing the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 4

    2015 arXiv

  41. [50]

    Channel gating neural net- works

    Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G Edward Suh. Channel gating neural net- works. Advances in Neural Information Processing Sys- tems, 32, 2019. 3

    2019

  42. [51]

    Mutual context network for jointly estimating egocentric gaze and action

    Yifei Huang, Minjie Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato. Mutual context network for jointly estimating egocentric gaze and action. IEEE Transactions on Image Processing, 29:7795–7806, 2020. 16

    2020

  43. [52]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. 4, 5

    2016 arXiv

  44. [53]

    The audio-visual conversational graph: From an egocentric-exocentric perspective

    Wenqi Jia, Miao Liu, Hao Jiang, Ishwarya Ananthab- hotla, James M Rehg, Vamsi Krishna Ithapu, and Ruo- han Gao. The audio-visual conversational graph: From an egocentric-exocentric perspective. In CVPR, 2024. 1

    2024

  45. [54]

    Egocentric deep multi-channel audio-visual active speaker localization

    Hao Jiang, Calvin Murdock, and Vamsi Krishna Ithapu. Egocentric deep multi-channel audio-visual active speaker localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10544– 10552, 2022. 2, 5, 6, 17, 18, 19

    2022

  46. [55]

    Multitask learning to im- prove egocentric action recognition

    Georgios Kapidis, Ronald Poppe, Elsbeth Van Dam, Lucas Noldus, and Remco Veltkamp. Multitask learning to im- prove egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019. 16

    2019

  47. [56]

    Epic-fusion: Audio-visual temporal binding for egocentric action recognition

    Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019. 17

    2019

  48. [57]

    Epic-fusion: Audio-visual temporal binding for egocentric action recognition

    Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 5492–5501, 2019. 5, 6

    2019

  49. [58]

    Motion guided attention fusion to recognize interactions from videos

    Tae Soo Kim, Jonathan Jones, and Gregory D Hager. Motion guided attention fusion to recognize interactions from videos. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , pages 13076– 13086, 2021. 16

    2021

  50. [59]

    Movinets: Mobile video networks for efficient video recog- nition

    Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recog- nition. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16020–16030,

  51. [60]

    Scsam- pler: Sampling salient clips from video for efficient action recognition

    Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsam- pler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6232–6242, 2019. 2

    2019

  52. [61]

    In the eye of transformer: Global-local correlation for egocentric gaze estimation

    Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg. In the eye of transformer: Global-local correlation for egocentric gaze estimation. arXiv preprint arXiv:2208.04464 , 2022. 6, 7, 17, 19

    2022 arXiv

  53. [62]

    Delving into ego- centric actions

    Yin Li, Zhefan Ye, and James M Rehg. Delving into ego- centric actions. In CVPR, 2015. 16

    2015

  54. [63]

    In the eye of the be- holder: Gaze and actions in first person video

    Yin Li, Miao Liu, and James M Rehg. In the eye of the be- holder: Gaze and actions in first person video. IEEE trans- actions on pattern analysis and machine intelligence , 45 (6):6731–6747, 2021. 6, 17, 19

    2021

  55. [64]

    Egocentric pre- diction of action target in 3d

    Yiming Li, Ziang Cao, Andrew Liang, Benjamin Liang, Lu- oyao Chen, Hang Zhao, and Chen Feng. Egocentric pre- diction of action target in 3d. In 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 20971–20980. IEEE, 2022. 1

    2022

  56. [65]

    A light weight model for active speaker detection

    Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, and Liangyin Chen. A light weight model for active speaker detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22932–22941, 2023. 5, 6, 17

    2023

  57. [66]

    Joint hand motion and interaction hotspots prediction from egocentric videos

    Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xi- aolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282–3292, 2022. 1

    2022

  58. [67]

    Jointly recog- nizing object fluents and tasks in egocentric videos

    Yang Liu, Ping Wei, and Song-Chun Zhu. Jointly recog- nizing object fluents and tasks in egocentric videos. In 11 Proceedings of the IEEE International Conference on Com- puter Vision, pages 2924–2932, 2017. 16

    2017

  59. [68]

    3d-to-2d distillation for indoor scene parsing

    Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4464–4474, 2021. 2

    2021

  60. [69]

    Aria ev- eryday activities dataset

    Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexan- der Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria ev- eryday activities dataset. arXiv preprint arXiv:2402.13349,

    arXiv

  61. [70]

    Something-else: Com- positional action recognition with spatial-temporal interac- tion networks

    Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Com- positional action recognition with spatial-temporal interac- tion networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pa...

    2020

  62. [71]

    Deciding how to decide: Dynamic routing in artificial neural networks

    Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In Interna- tional Conference on Machine Learning, pages 2363–2372. PMLR, 2017. 2

    2017

  63. [72]

    Foleygen: Visually-guided audio generation

    Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation. In 2024 IEEE 34th In- ternational Workshop on Machine Learning for Signal Pro- cessing (MLSP), pages 1–6. IEEE, 2024. 16

    2024

  64. [73]

    Ar-net: Adaptive frame resolution for effi- cient action recognition

    Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for effi- cient action recognition. In ECCV 2020, 2020. 2, 3

    2020

  65. [74]

    Adafuse: Adaptive temporal fusion network for efficient action recognition

    Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. arXiv preprint arXiv:2102.05775, 2021. 2, 5, 6, 17

    2021 arXiv

  66. [75]

    Integrating human gaze into attention for egocentric activity recognition

    Kyle Min and Jason J Corso. Integrating human gaze into attention for egocentric activity recognition. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1069–1078, 2021. 16

    2021

  67. [76]

    Towards determining per- ceived audience intent for multimodal social media posts using the theory of reasoned action

    Trisha Mittal, Sanjoy Chowdhury, Pooja Guhan, Snikitha Chelluri, and Dinesh Manocha. Towards determining per- ceived audience intent for multimodal social media posts using the theory of reasoned action. Scientific Reports, 14 (1):10606, 2024. 16

    2024

  68. [77]

    Safari: Adaptive sequence transformer for weakly super- vised referring expression segmentation

    Sayan Nag, Koustava Goswami, and Srikrishna Karanam. Safari: Adaptive sequence transformer for weakly super- vised referring expression segmentation. In European Conference on Computer Vision, pages 485–503. Springer,

  69. [78]

    Attention bottlenecks for multimodal fusion

    Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in neural information pro- cessing systems, 34:14200–14213, 2021. 16, 18

    2021

  70. [79]

    Auto- focus: Efficient multi-scale inference

    Mahyar Najibi, Bharat Singh, and Larry S Davis. Auto- focus: Efficient multi-scale inference. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 9745–9755, 2019. 3

    2019

  71. [80]

    Adamml: Adaptive multi-modal learning for efficient video recognition

    Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Xi- meng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7576–7585, 2021....

    2021

  72. [81]

    Per-clip video object seg- mentation

    Kwanyong Park, Sanghyun Woo, Seoung Wug Oh, In So Kweon, and Joon-Young Lee. Per-clip video object seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1352– 1361, 2022. 1

    2022

  73. [82]

    Relational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967–3976, 2019. 2

    2019

  74. [83]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Kop- pula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Sy...

    2024

  75. [84]

    Keeping your eye on the ball: Trajectory attention in video transformers

    Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. Ad- vances in neural information processing systems , 34: 12493–125...

    2021

  76. [85]

    Clipping: Distilling clip-based models with a student base for video- language retrieval

    Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Song- cen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with a student base for video- language retrieval. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogniti...

    2023

  77. [86]

    E2 (go) motion: Motion augmented event stream for egocentric action recognition

    Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Marco Cannici, Emanuele Gusso, Matteo Matteucci, and Barbara Caputo. E2 (go) motion: Motion augmented event stream for egocentric action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern r...

    2022

  78. [87]

    Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision- language model

    Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chel- lappa, and Amjad Almahairi. Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision- language model. arXiv preprint arXiv:2312.12423 , 2023. 16

    2023 arXiv

  79. [88]

    V olta: Vision-language transformer with weakly-supervised local- feature alignment

    Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik J Shah, Yann LeCun, and Rama Chellappa. V olta: Vision-language transformer with weakly-supervised local- feature alignment. Transactions on Machine Learning Re- search, 2023. 16

    2023

  80. [89]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vis...

    2023

  81. [90]

    V-desirr: Very fast deep embedded single image reflection removal

    BH Prasad, Lokesh R Boregowda, Kaushik Mitra, Sanjoy Chowdhury, et al. V-desirr: Very fast deep embedded single image reflection removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2390– 2399, 2021. 16

    2021

  82. [91]

    Revisiting spatio-temporal layouts for compositional action recognition

    Gorjan Radevski, Marie-Francine Moens, and Tinne Tuyte- laars. Revisiting spatio-temporal layouts for compositional action recognition. arXiv preprint arXiv:2111.01936, 2021. 16

    2021 arXiv

  83. [92]

    Multimodal distillation for egocentric action recognition

    Gorjan Radevski, Dusan Grujicic, Matthew Blaschko, Marie-Francine Moens, and Tinne Tuytelaars. Multimodal distillation for egocentric action recognition. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 5213–5224, 2023. 2

    2023

  84. [93]

    Faster r-cnn: Towards real-time object detection with re- gion proposal networks.IEEE transactions on pattern anal- ysis and machine intelligence, 39(6):1137–1149, 2016

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with re- gion proposal networks.IEEE transactions on pattern anal- ysis and machine intelligence, 39(6):1137–1149, 2016. 16

    2016

  85. [94]

    Channel-wise knowledge distillation for dense prediction

    Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pages 5311–5320,

  86. [95]

    Charades-ego: A large- scale dataset of paired third and first person videos

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large- scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018. 1

    2018 arXiv

  87. [96]

    Representation learning for semantic align- ment of language, audio, and visual modalities

    Parthasaarathy Sudarsanam, Irene Mart´ın-Morat´o, and Tuo- mas Virtanen. Representation learning for semantic align- ment of language, audio, and visual modalities. arXiv preprint arXiv:2505.14562, 2025. 16

    2025 arXiv

  88. [97]

    How to evaluate deep neural network processors: Tops/w (alone) considered harmful

    Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. How to evaluate deep neural network processors: Tops/w (alone) considered harmful. IEEE Solid-State Cir- cuits Magazine, 2020. 17

    2020

  89. [98]

    Egodistill: Egocentric head motion distillation for efficient video understanding

    Shuhan Tan, Tushar Nagarajan, and Kristen Grauman. Egodistill: Egocentric head motion distillation for efficient video understanding. Advances in Neural Information Pro- cessing Systems, 36:33485–33498, 2023. 2, 3, 4

    2023

  90. [99]

    Codi-2: In-context interleaved and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context interleaved and interactive any-to-any generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27425–27434, 2024. 16

    2024

  91. [100]

    Is someone speaking? exploring long-term temporal features for audio-visual ac- tive speaker detection

    Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking? exploring long-term temporal features for audio-visual ac- tive speaker detection. In Proceedings of the 29th ACM international conference on multimedia, pages 3927–3935,

  92. [101]

    H+ o: Unified egocentric recognition of 3d hand-object poses and interactions

    Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+ o: Unified egocentric recognition of 3d hand-object poses and interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4511– 4520, 2019. 16

    2019

  93. [102]

    Contrastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019. 2

    1910 arXiv

  94. [103]

    View while moving: Efficient video recognition in long- untrimmed videos

    Ye Tian, Mengyu Yang, Lanshan Zhang, Zhizhen Zhang, Yang Liu, Xiaohui Xie, Xirong Que, and Wendong Wang. View while moving: Efficient video recognition in long- untrimmed videos. In Proceedings of the 31st ACM Inter- national Conference on Multimedia, pages 173–183, 2023. 2

    2023

  95. [104]

    Direction of arrival estimation in highly rever- berant environments using soft time-frequency mask

    Vladimir Tourbabin, Jacob Donley, Boaz Rafaely, and Rav- ish Mehra. Direction of arrival estimation in highly rever- berant environments using soft time-frequency mask. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 383–387. IEEE,

    2019

  96. [105]

    A closer look at spatiotem- poral convolutions for action recognition

    Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotem- poral convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018. 2

    2018

  97. [106]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 5

    2017

  98. [107]

    Convolutional networks with adaptive inference graphs

    Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. InProceedings of the Euro- pean conference on computer vision (ECCV) , pages 3–18,

  99. [108]

    Ego-only: Egocentric action detection without exocentric transferring

    Huiyu Wang, Mitesh Kumar Singh, and Lorenzo Torresani. Ego-only: Egocentric action detection without exocentric transferring. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5250–5261, 2023. 5, 6, 7, 8, 17

    2023

  100. [109]

    Efficient video transformers with spatial-temporal token selection

    Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zuxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. In European Conference on Computer Vision, pages 69–86. Springer, 2022. 2

    2022

  101. [110]

    Dis- tilling object detectors with fine-grained feature imitation

    Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Dis- tilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933–4942, 2019. 2

    2019

  102. [111]

    Videos as space-time region graphs

    Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European conference on computer vision (ECCV), pages 399–417, 2018. 16

    2018

  103. [112]

    Skipnet: Learning dynamic routing in convolutional networks

    Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the Euro- pean conference on computer vision (ECCV) , pages 409– 424, 2018. 2

    2018

  104. [113]

    Loconet: Long-short context network for active speaker detection

    Xizi Wang, Feng Cheng, and Gedas Bertasius. Loconet: Long-short context network for active speaker detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 18462–18472, 2024. 5, 6, 17

    2024

  105. [114]

    Adaptive focus for efficient video recognition

    Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In proceedings of the IEEE/CVF in- ternational conference on computer vision , pages 16249– 16258, 2021. 2, 3

    2021

  106. [115]

    Adafocus v2: End-to-end training of spatial dy- namic networks for video recognition

    Yulin Wang, Yang Yue, Yuanze Lin, Haojun Jiang, Zihang Lai, Victor Kulikov, Nikita Orlov, Humphrey Shi, and Gao Huang. Adafocus v2: End-to-end training of spatial dy- namic networks for video recognition. In 2022 IEEE/CVF 13 Conference on Computer Vision and Pattern Recognitio...

    2022

  107. [116]

    Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion a...

    2022

  108. [117]

    Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion a...

    2022

  109. [118]

    Measured albedo in the wild: Filling the gap in intrinsics evaluation

    Jiaye Wu, Sanjoy Chowdhury, Hariharmano Shanmugaraja, David Jacobs, and Soumyadip Sengupta. Measured albedo in the wild: Filling the gap in intrinsics evaluation. In 2023 IEEE International Conference on Computational Photog- raphy (ICCP), pages 1–12. IEEE, 2023. 16

    2023

  110. [119]

    Blockdrop: Dynamic inference paths in residual net- works

    Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8817–8826, 2018. 2

    2018

  111. [120]

    Adaframe: Adaptive frame se- lection for fast video recognition

    Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S Davis. Adaframe: Adaptive frame se- lection for fast video recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1278–1287, 2019. 3

    2019

  112. [121]

    Rethinking audio-visual synchroniza- tion for active speaker detection

    Abudukelimu Wuerkaixi, You Zhang, Zhiyao Duan, and Changshui Zhang. Rethinking audio-visual synchroniza- tion for active speaker detection. In 2022 IEEE 32nd In- ternational Workshop on Machine Learning for Signal Pro- cessing (MLSP), pages 01–06. IEEE, 2022. 5, 6, 17

    2022

  113. [122]

    Efficient deep visual and inertial odometry with adaptive visual modality selection

    Mingyu Yang, Yu Chen, and Hun-Seok Kim. Efficient deep visual and inertial odometry with adaptive visual modality selection. In European Conference on Computer Vision , pages 233–250. Springer, 2022. 2, 5, 6

    2022

  114. [123]

    Beyond short clips: End-to-end video- level learning with collaborative memories

    Xitong Yang, Haoqi Fan, Lorenzo Torresani, Larry S Davis, and Heng Wang. Beyond short clips: End-to-end video- level learning with collaborative memories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 7567–7576, 2021. 1

    2021

  115. [124]

    Spherical world-locking for audio-visual localization in egocentric videos

    Heeseung Yun, Ruohan Gao, Ishwarya Ananthabhotla, Anurag Kumar, Jacob Donley, Chao Li, Gunhee Kim, Vamsi Krishna Ithapu, and Calvin Murdock. Spherical world-locking for audio-visual localization in egocentric videos. In European Conference on Computer Vision (ECCV), 2024. 5, 6...

    2024

  116. [125]

    Is an object-centric video representation beneficial for trans- fer? In Proceedings of the Asian Conference on Computer Vision, pages 1976–1994, 2022

    Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Is an object-centric video representation beneficial for trans- fer? In Proceedings of the Asian Conference on Computer Vision, pages 1976–1994, 2022. 16

    1976

  117. [126]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022. 2

    2022

  118. [127]

    Cascaded interactional targeting network for egocentric video analysis

    Yang Zhou, Bingbing Ni, Richang Hong, Xiaokang Yang, and Qi Tian. Cascaded interactional targeting network for egocentric video analysis. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 1904–1913, 2016. 16

    1904

  119. [128]

    Eco: Efficient convolutional network for online video understanding

    Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional network for online video understanding. In Proceedings of the European conference on computer vision (ECCV), pages 695–712, 2018. 2 14 EGOADAPT : Adaptive Multisensory Distillation and Poli...

    2018

  120. [129]

    in addition to MUST [124] as our teacher model to compare the performance of E GOADAPT as reported in Tab. 11. Experimental results demonstrate that in all the cases our proposed approach is able to closely replicate the teacher model’s performance while operating at a very lo...

  121. [130]

    Sync-TalkNet

    learns a long-Short context network. Sync-TalkNet

  122. [131]

    ASD-Trans [22] employs a ResNet-18 to ex- tract audio features

    models cross-modal information with complex atten- tion modules. ASD-Trans [22] employs a ResNet-18 to ex- tract audio features. LW-ASD [65] proposes a GRU based active speaker detection model. F.2. Action Recognition MoViNet [59] proposes a three-step approach to improve comp...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.