Pith. sign in

REVIEW 4 major objections 5 minor 9 references

PCIE_Interaction Solution for Ego4D Social Interaction Challenge

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dual-stream audio-visual system with a max-score gaze filter achieved 0.81 mAP on Looking At Me and 0.71 mAP on Talking To Me, topping the Ego4D challenge leaderboards.

desk verdict Credible leaderboard results in a challenge-report wrapper, with one under-supported component-level claim about the visual max-score filter. read the letter →

arxiv 2505.24404 v1 pith:3XYTQR5W submitted 2025-05-30 cs.CV

Kanokphan Lertniphonphan , Feng Chen , Junda Xu , Fengbu Lan , Jun Xie , Tao Zhang , Zhepeng Wang This is my paper
classification cs.CV
keywords egocentricvideosocialinteractiondetectionlookingatmetalkingtoaudio-visualfusiongazeestimationutterance-levelclassificationEgo4Dchallenge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This report describes a system for the Ego4D social interaction challenge that tackles two tasks with one design: Looking At Me (LAM) uses only visual gaze analysis, while Talking To Me (TTM) combines visual and audio streams. The authors claim that for LAM, deblurring face crops and ensembling three gaze models reaches 0.81 mAP and 0.93 Top-1 accuracy, the best on the leaderboard. For TTM, they report that a visual max-score filter, which keeps the strongest frame-level gaze score over each utterance segment, aligns frame-level predictions with utterance-level labels and boosts visual-only performance. Fusing that filtered visual score with a Whisper audio model, weighted by face quality, gives 0.71 mAP and 0.60 Top-1 accuracy, also the best reported. The core claim is that separate modality models plus simple score filtering beat joint audio-visual training for detecting social interaction.

What carries the argument

Two mechanisms carry the argument. The visual max-score filter converts frame-level gaze scores into an utterance-level score by keeping only the largest score over the audio segment; it embodies the assumption that a speaker talking to the wearer will look at the wearer at least once. The quality-weighted fusion module then combines the filtered visual score with the audio score using frame-level face alignment quality as the weight, and a median filter smooths outputs over time. For LAM, the load-bearing machinery is an ensemble of InternLSTM, GazeLSTM, and GazePose outputs on deblurred face crops, again smoothed by median filtering.

What would settle it

On the validation split, replace the visual max-score filter with mean or median pooling of frame-level scores; if visual-only TTM mAP does not drop, or if a large share of true 'talking to me' utterances have no frame with a high gaze score, the filter's core assumption is false.

Watch

Extended reading notes

Core claim

The central claim of the paper is that the mismatch between frame-level gaze predictions and utterance-level 'talking to me' labels is best resolved by taking the maximum frame-level visual score within each audio segment, rather than by training a joint audio-visual model or averaging scores. On the authors' own leaderboard results, this recipe, together with face-crop deblurring and an ensemble of three gaze architectures, produced a LAM mAP of 0.81 and a TTM mAP of 0.71, both first in their respective tracks. The paper treats audio as the main detector of verbal engagement and the filtered visual stream as a suppressor of false positives from conversations that are not directed at the camera wearer.

Load-bearing premise

The TTM result depends on the premise that in every segment labeled 'talking to me', the speaker looks at the camera wearer in at least one frame, so the single highest frame-level gaze score is a trustworthy utterance-level signal.

Editorial extensions

If this is right

  • The visual max-score filter gives a reusable way to aggregate frame-level gaze scores whenever labels are defined over longer segments, not just in this challenge.
  • An audio-only TTM classifier can be strengthened by a visual gating term that removes confident speech detections from non-interactive scenes.
  • Ensembling existing gaze models with input deblurring can top a leaderboard without introducing a new architecture.
  • Whisper-based audio embeddings, even from a small variant, are competitive enough that audio need not be trained jointly with vision.
  • Quality-weighted fusion outperforms plain score averaging in the reported comparison, making the face-alignment score a useful reliability signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The max-score filter likely trades off recall against precision: an utterance in which the speaker never faces the camera would be missed entirely, and a single brief glance could produce a false positive.
  • One testable extension is to apply the same filter to continuous interaction detection, where utterance boundaries must be estimated rather than given, to see if the alignment benefit survives boundary errors.
  • The reported gap between validation and test for Whisper large-v3 suggests that audio model selection should be based on a robust held-out split rather than on the official validation set alone.
  • Because leaderboard numbers are computed on a private test split, the 0.81 and 0.71 figures are not directly comparable with later methods unless the same evaluation protocol is reused.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript is a short technical report for the Ego4D Social Interaction Challenge at CVPR 2025. The authors describe a Looking At Me (LAM) solution that combines face-crop deblurring, several visual baselines (GazeLSTM, GazePose, InternLSTM), ensembling, and median temporal smoothing, and a Talking To Me (TTM) solution that runs separate audio (Whisper) and visual (GazeLSTM) branches, applies a visual max-score filter to convert frame-level gaze predictions to utterance-level scores, fuses the modalities with quality-weighted fusion, and post-processes with median filtering. The paper reports leaderboard mAP of 0.81 for LAM and 0.71 for TTM, with code made publicly available.

Significance. If the reported leaderboard numbers are taken at face value, this is a competitive, potentially top-ranked engineering result on both Ego4D social interaction tracks. The leaderboard tables are externally checkable, the code release is a concrete reproducibility artifact, and the LAM comparison against several baselines is a useful reference point. The main significance is therefore empirical rather than methodological. The TTM explanation, however, currently rests on an insufficiently validated aggregation filter, and the reported internal ablations are too sparse to establish why the individual components work.

major comments (4)
  1. [Section 2.2, Table 2] The visual max-score filter is the central load-bearing component of the TTM explanation, but it is internally inconsistent with the paper's own characterization of TTM labels. The text states that speakers 'might not look at the camera wearer at all' during a positive utterance; for such an utterance the maximum frame-level visual score cannot be high, so the filter cannot align visual predictions with the utterance-level label. Conversely, for a negative utterance a single brief glance can become the maximum and inflate false positives. The only supporting evidence is the test mAP jump from 54.86% to 63.27% in Table 2, with no validation row, no per-utterance analysis, and no comparison against mean or median aggregation. This needs a concrete fix: report the filter's validation performance, compare alternative pooling functions, and analyze cases where positive utterances have no looking-at-me frame.
  2. [Table 2] The layout of Table 2 is ambiguous: the header has four columns (Validation mAP/Acc and Test mAP/Acc), but the rows '+ Visual filter', 'Averaging', and 'Quality Score weight Fusion' contain only two values. It is unclear whether those values are test-set metrics or something else, which makes the reported ablations impossible to interpret precisely. In addition, no run counts or error bars are given. For example, Whisper small and large-v3 differ by only 0.34 mAP on validation (70.17% vs 70.51%), and the large model is worse on test accuracy (56.92% vs 60.80%); without repeated runs or confidence intervals, the statement that 'the large model performs better on validation data' is not statistically supported.
  3. [Section 3.2 (LAM), Table 1] The LAM method is described as an ensemble of InternLSTM, GazeLSTM, and GazePose with deblurring and median filtering, but Table 1 reports only the individual baselines, not the final ensemble. As a result, the quantitative contribution of each claimed component (deblurring, ensemble diversity, and the median filter) is unknown. The leaderboard mAP of 0.81 is externally valid, but the paper's explanation of how that result is obtained is not supported by the presented ablation data. A single row showing the final ensemble with and without each component would resolve this.
  4. [Section 2.2 and Section 3.2 (TTM)] The description of the final TTM submission is too underspecified to reproduce from the text alone: the paper mentions ensembling 'multiple fusion strategies' and applying 'temporal post-processing' without defining the ensemble weights, the set of fusion strategies, or the parameters of the post-processing. Since the code is available, this is not a fatal gap, but the main text should identify at least the fusion strategies and the post-processing window, and should clarify whether the reported 0.71 mAP uses the same quality-weighted fusion shown in Table 2 or a different ensemble.
minor comments (5)
  1. [Abstract] There is a typo: 'F or the TTM task' should read 'For the TTM task'.
  2. [Section 3.2, Table 4] In Table 4 the team name is listed as 'PCIE LAM (Ours)', but the table reports Talking To Me results; this should be 'PCIE TTM (Ours)' to avoid confusion with the LAM table.
  3. [Section 3.2] The phrase 'our submission result on the TAM leader board' contains a typo: 'TAM' should be 'TTM'.
  4. [Figure 2] Figure 2 is captioned 'Looking At Me results', but the surrounding text also uses it to illustrate gaze aversion in TTM contexts; the caption and figure should be aligned with what is actually shown.
  5. [References] The reference [4] is used for both the Ego4D benchmark and the GazeLSTM baseline; these are distinct works and should be cited separately so the reader can identify the exact architecture used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are externally benchmarked leaderboard scores, and its method components are empirical ablations and prior externally evaluated systems rather than self-justifying definitions.

full rationale

The paper's load-bearing claims are the 0.81 LAM and 0.71 TTM mAP results, which are measured against the Ego4D challenge leaderboard and are externally checkable. The TTM visual max-score filter in Section 2.2 is a fixed heuristic (retaining the maximum frame-level visual score per audio segment), not a fitted parameter renamed as a prediction; its support is the empirical validation/test comparison in Table 2, and the paper itself acknowledges the assumption's limitation by noting that speakers 'might not look at the camera wearer at all,' which is a correctness risk rather than a circularity. The LAM ensemble uses InternLSTM [5], the team's own prior challenge solution, alongside GazeLSTM [4] and GazePose [8], and the TTM fusion follows the external QUAVF work [6]; these citations are used as implementation references and baseline components, not as a self-citation chain that forces the outcome. No equation or definition reduces to its own input, and no fitted parameter is relabeled as a prediction. The noted generalizability concern with Whisper large-v3 (better on validation, not test) is ordinary model selection, not circular reasoning.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central empirical claim depends on several hand-chosen engineering choices: model selection, the max-score aggregation rule, fusion weighting, and unspecified ensemble and filtering details. No new physical or conceptual entities are introduced.

free parameters (5)
  • Whisper model choice (small vs large-v3) = large-v3 selected on validation, test mAP 66.17 vs small 66.16
    Model selection was based on validation performance; the paper notes it did not generalize to test, so the choice is an empirical fit to development data.
  • Visual max-score filter aggregation = maximum over each utterance segment
    The filter is a hand-chosen aggregation rule; no threshold or comparison to mean or median is reported.
  • Quality-weighted fusion weights = derived from face alignment scores [1]
    The fusion weighting is taken from prior work [6] and not derived in this paper; it remains a tuned scheme.
  • Ensemble composition and parameter configurations = unspecified combination of InternLSTM, GazeLSTM, GazePose
    The final LAM ensemble combines multiple models with 'diverse parameter configurations'; exact weights, number of models, and hyperparameters are not reported.
  • Median filter window size = not specified
    Temporal post-processing uses an unspecified median filter; the result depends on window duration.
assumptions (3)
  • domain assumption For each TTM utterance labeled as talking to me, the speaker has at least one frame where gaze is toward the camera wearer, making the max visual score a meaningful utterance-level signal.
    Introduced in Section 2.2 as the motivation for the visual max-score filter; the paper provides no per-utterance analysis verifying this property.
  • domain assumption Face alignment quality scores are a reliable proxy for the reliability of visual predictions, so weighting visual and audio scores by them improves fusion.
    The quality-weighted fusion is adopted from [6] and used without evaluating alternatives or sensitivity.
  • domain assumption The provided face crops and synchronized audio segments are correctly aligned to the same target identities.
    The whole pipeline assumes the challenge preprocessing gives correct face tracks and audio-visual synchronization, stated in Sections 1 and 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PCIE_Interaction Solution for Ego4D Social Interaction Challenge." pith.science (2026). https://pith.science/paper/3XYTQR5W

@misc{pith2026250524404,
  author       = {Pith},
  title        = {Pith review of: PCIE_Interaction Solution for Ego4D Social Interaction Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XYTQR5W}},
  note         = {Machine review of arXiv:2505.24404}
}
read the original abstract

This report presents our team's PCIE_Interaction solution for the Ego4D Social Interaction Challenge at CVPR 2025, addressing both Looking At Me (LAM) and Talking To Me (TTM) tasks. The challenge requires accurate detection of social interactions between subjects and the camera wearer, with LAM relying exclusively on face crop sequences and TTM combining speaker face crops with synchronized audio segments. In the LAM track, we employ face quality enhancement and ensemble methods. For the TTM task, we extend visual interaction analysis by fusing audio and visual cues, weighted by a visual quality score. Our approach achieved 0.81 and 0.71 mean average precision (mAP) on the LAM and TTM challenges leader board. Code is available at https://github.com/KanokphanL/PCIE_Ego4D_Social_Interaction

Figures

Figures reproduced from arXiv: 2505.24404 by the authors.

Figure 1
Figure 1. Our proposed Social Interaction Detection framework [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Looking At Me results Team mAP (↑) Acc (↑) PCIE LAM (Ours) 0.71 0.60 ex2eg 0.68 0.61 z(test) 0.67 0.62 HsiCheLin 0.67 0.58 EgoAdapt 0.67 0.62 Host 30426 Team 0.54 0.54 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

  1. [5]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18973--18990, 2021

    work page 2022

  2. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [2]

    How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)

    Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017

    2017

  4. [3]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023

    arXiv 2023

  5. [4]

    Nafssr: Stereo image super-resolution using nafnet

    Xiaojie Chu, Liangyu Chen, and Wenqing Yu. Nafssr: Stereo image super-resolution using nafnet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1239--1248, 2022

    work page 2022

  6. [6]

    PCIE_LAM Solution for Ego4D Looking At Me Challenge

    Kanokphan Lertniphonphan, Jun Xie, Yaqing Meng, Shijing Wang, Feng Chen, and Zhepeng Wang. Pcie\_lam solution for ego4d looking at me challenge. ArXiv, abs/2406.12211, 2024

    work page Pith review arXiv 2024

  7. [7]

    Hsi-Che Lin, Chien-Yi Wang, Min-Hung Chen, Szu-Wei Fu, and Y. Wang. Quavf: Quality-aware audio-visual fusion for ego4d talking to me challenge. ArXiv, abs/2306.17404, 2023

    arXiv 2023

  8. [8]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022

    2022

Show all 9 references
  1. [9]

    Team pku-wict-mipl ego4d look at me challenge 2023 technical report

    Xiyu Wei, Dejie Yang, Yuxin Peng, and Yang Liu. Team pku-wict-mipl ego4d look at me challenge 2023 technical report

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.