REVIEW 3 major objections 5 minor 27 references
From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Graph-based inputs lift egocentric video question answering from 35% to 44.21% accuracy.
desk verdict A legitimate challenge report with a clear graph-based recipe for egocentric VQA, but the 44.21% headline rests on a per-micro-category ensemble whose split is never disclosed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery that carries the result is graph serialization: video and external knowledge are converted into natural-language graph descriptions inserted into the answer prompt. SceneNet formalizes each video segment as $G = (N, E_B, A)$, with typed nodes ($Agent$, $Environment$, $DynamicObjects$), binary edges for containment and initial location, and action hyperedges carrying timestamps and source/target results; KnowledgeNet builds an object-rooted ConceptNet subgraph $G_i = (V_i, R, E_i)$, keeps assertions whose normalized confidence exceeds 0.7, and renders paths as templates such as 'cupboard is used for storing dishes'. A sentence-embedding filter keeps only paths relevant to a kitchen context, and the surviving text is given to the multimodal LLM alongside the question and, for KnowledgeNet, the raw video.
What would settle it
On a fresh random sample of HD-EPIC questions with the per-micro-category selection rule fixed before any labels are seen, recompute accuracy for video-only, SceneNet, KnowledgeNet, and the selected ensemble; if the ensemble gain over the 35% to 38% single-stream scores shrinks or reverses, the 44.21% headline is an artifact of selection.
Extended reading notes
Core claim
The paper's central claim is that a fixed multimodal LLM answers egocentric video questions better when the video is first distilled into a symbolic graph that is then handed to the model as text. SceneNet alone lifts accuracy from 35.00% to 37.96%, with its largest category gains on Recipe (+22.88), 3D Perception (+12.46), and Object Motion (+12.64); KnowledgeNet alone reaches 38.04%, with consistent gains across most categories and the largest on Recipe (+8.75) and Nutrition (+4.33). The two methods win on different micro-categories, so the authors pick the best configuration per micro-category for the challenge submission and obtain 44.21% overall. The ablations further show that SceneNet works best without raw video, while KnowledgeNet loses most of its value when the video is removed (overall 36.58% with video versus 27.14% without), so the discovered mechanism is a combination of complementary abstractions rather than a single graph type.
Load-bearing premise
The headline accuracy depends on choosing, for each micro-category, the method that performed best, and the report does not say whether that choice was made on a validation split or on the final test labels; if the latter, the 44.21% number would overestimate generalization.
Editorial extensions
If this is right
- Per-micro-category method selection is worth about nine accuracy points over a video-only baseline, so question-type routing is a cheap and effective ensemble strategy.
- SceneNet without raw video beats SceneNet with raw video, implying that a well-structured scene graph can carry enough spatio-temporal information for many questions and that raw video can add noise.
- KnowledgeNet without video collapses to 27.14% overall, so commonsense knowledge must be visually grounded to help.
- Mixing both graph types in a single prompt underperforms either alone, so naive multi-source fusion is not the right way to combine them.
- The largest gains appear in categories that need location tracking or procedural knowledge, pointing to where graph-based conditioning should be applied first.
Reading between the lines
- The paper's per-micro-category selector is a coarse stand-in for a question-aware policy; a model that decides at run time whether to consult scene structure, commonsense, or raw video could capture the same gains without needing precomputed category labels.
- If the 44.21% figure survives clean validation-set selection, the same wrapper recipe—extract graphs with an MLLM, serialize to text, condition the answer—should transfer to other long-form video QA benchmarks without retraining the underlying model.
- The ablation pattern suggests scene graphs and commonsense graphs encode complementary failures: scene graphs track where things are, commonsense graphs know what things are for, so a gating mechanism rather than concatenation is the promising route to a unified model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This challenge report describes SceneNet and KnowledgeNet, two graph-based input augmentations for egocentric video question answering on the HD-EPIC VQA benchmark. SceneNet extracts MLLM-generated scene graphs from video segments, while KnowledgeNet expands ConceptNet-based knowledge graphs from detected objects. Both are fed to Gemini 2.0 Flash either alone or with raw video. Table 1 reports per-category and overall accuracy: Video Only 35.00%, SceneNet 37.96%, and KnowledgeNet 38.04%. Section 3.3 states that the final challenge submission ('DeepFrames') selected the best-performing method per micro-category, yielding an overall accuracy of 44.21%. Section 3.2 reports 50-sample-per-micro-category ablations across video/no-video and graph combinations.
Significance. If the 44.21% result is obtained through a properly validated selection rule rather than oracle selection on the evaluation labels, the paper would offer a useful neuro-symbolic recipe for egocentric VQA, with clearly structured graph representations, a same-model video-only baseline, and detailed per-category analysis. The paper is transparent about practical constraints such as the 11.7% malformed JSON rate, the 40-minute processing window, and the use of Gemini 2.0 Flash for both graph generation and inference. However, the central claim currently lacks the statistical and experimental-split support needed to establish that the reported improvement over the video-only baseline generalizes.
major comments (3)
- [Section 3.3 (Submission Strategy)] The headline 44.21% accuracy is obtained by 'select[ing] the best-performing method for each micro-category,' but the manuscript never states whether these selections were made on a separate validation split or on the evaluation labels that produce the reported number. If the selection used the evaluation labels, the reported accuracy is an oracle ensemble rather than an estimate of generalization. Please state the split used for selection and, if a validation split was used, report the accuracy of the final selected ensemble on the held-out test split, along with the number of questions per micro-category.
- [Section 3.2 (Ablations, Table 2)] All ablations use 50 samples per micro-category. For a binary multiple-choice accuracy near 50%, the standard error is approximately 7 percentage points, so differences such as the Action row (S-Net 40.50 vs. K-Net 49.00 in the video condition) or the Ingredient row (K-Net 43.00 vs. S-Net 40.66) are well within noise. The paper reports no confidence intervals, error bars, or significance tests, so neither the per-micro-category selection in Section 3.3 nor the qualitative conclusions in Section 3.2 (e.g., 'S-Net without video outperformed the combined S-Net with video setup') are statistically supported. Please provide sample sizes, confidence intervals, and a paired significance test for the main comparisons.
- [Section 3.1 (Results, Table 1)] The central per-category comparison reports only point estimates. The overall differences (Video Only 35.00%, SceneNet 37.96%, KnowledgeNet 38.04%) are about 2–3 percentage points, and several per-category differences (e.g., Nutrition: 33.67 vs. 34.67 vs. 38.00) are small. Without the number of questions per category and per micro-category, and without confidence intervals, it is impossible to tell whether these differences are meaningful. Please state the exact evaluation split and the question counts used for Table 1.
minor comments (5)
- [Table 1] The arrow notation contains stray 'x' characters ('↓x6.70', '↓x5.50'), which appear to be typographical errors and should be cleaned up.
- [References [7] and [8]] Reference [7] is titled 'Gemini: Integrating large language models and large vision models for general-purpose multimodal ai' but the listed author list does not match the actual Gemini authors; reference [8] (Google. Gemini 2.0 flash) lacks a proper venue, year, and version identifier. Both citations should be corrected.
- [Section 2.2, Domain Filtering (Eq. 11)] The construction of the reference sentence set S used for domain filtering is not described; please state how many sentences it contains, how they were generated or selected, and whether the retained top-30 paths are sensitive to this choice.
- [Section 3.2 (S+K-Net)] The 'S+K-Net' configuration is not defined: it is unclear whether the scene graph and knowledge graph are concatenated in a single prompt, fed through separate prompts, or merged into one graph. Please specify the input format for this ablation.
- [Section 2.1.1] The schema for scene-graph generation is described only in prose; including the exact JSON prompt template (or a representative example) would improve reproducibility, since the scene graph content is entirely prompt-dependent.
Circularity Check
No circular derivation found; the graph-based methods are evaluated empirically against a same-model video-only baseline.
full rationale
This paper's central claims are empirical evaluations of two graph-conditioned MLLM pipelines against a video-only Gemini 2.0 Flash baseline on the external HD-EPIC VQA benchmark. SceneNet and KnowledgeNet are constructed from MLLM-generated scene graphs and ConceptNet-based knowledge graphs, respectively, and their effectiveness is measured by per-category accuracy in Table 1. Neither graph representation is defined in terms of the final accuracy, and no equation in Sections 2.1 and 2.2 reduces the reported scores to the method's own inputs. The final 44.21% figure is produced by the ensemble strategy in Section 3.3, where the authors 'selected the best-performing method for each micro-category.' This is a model-selection step rather than a derivation, and the paper does not state the split used for that selection. If the selection used the evaluation labels, the number would be an oracle upper bound rather than a generalization estimate, but that is an experimental-validity and reporting concern, not circularity, because no quoted equation or definition forces the result to equal its inputs. There are no load-bearing self-citations and no imported uniqueness theorems; all cited sources are external datasets, models, or tools. The shared use of Gemini for both graph generation and answer inference is a modeling limitation, not circular reasoning. Overall, the comparisons are self-contained and externally benchmarked, so the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- ConceptNet expansion depth d =
3
- ConceptNet edge confidence threshold =
0.7
- Maximum retained paths per object =
30
- Number and content of reference sentences for domain filtering =
not stated
- Temporal divisor for video acceleration =
adaptive
assumptions (5)
- domain assumption ConceptNet provides reliable commonsense assertions for kitchen entities.
- domain assumption The reference sentences used for domain filtering capture kitchen-relevant semantics.
- domain assumption The MLLM-generated scene graphs are faithful to the video content.
- domain assumption The same MLLM used for inference can also accurately produce graph-structured summaries.
- domain assumption The benchmark's validation set is representative of the test set.
Cite this review
Pith. "Pith review of From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge." pith.science (2026). https://pith.science/paper/ICQDMSFS
@misc{pith2026250608553,
author = {Pith},
title = {Pith review of: From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge},
year = {2026},
howpublished = {\url{https://pith.science/paper/ICQDMSFS}},
note = {Machine review of arXiv:2506.08553}
}
read the original abstract
This report presents SceneNet and KnowledgeNet, our approaches developed for the HD-EPIC VQA Challenge 2025. SceneNet leverages scene graphs generated with a multi-modal large language model (MLLM) to capture fine-grained object interactions, spatial relationships, and temporally grounded events. In parallel, KnowledgeNet incorporates ConceptNet's external commonsense knowledge to introduce high-level semantic connections between entities, enabling reasoning beyond directly observable visual evidence. Each method demonstrates distinct strengths across the seven categories of the HD-EPIC benchmark, and their combination within our framework results in an overall accuracy of 44.21% on the challenge, highlighting its effectiveness for complex egocentric VQA tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Flamingo: a visual language model for few-shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Paul Luc, An- toine Miech, Ian Barr, Yana Hasson, Mohammad Azar, Matthew Botvinick, Andrew Zisserman, and Karen Si- monyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Sys- tems, 2022. 1
work page 2022
-
[2]
3d scene graph: A structure for unified semantics, 3d space, and cam- era
Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and cam- era. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 5664–5673, 2019. 2
work page 2019
-
[3]
Lu ´ıs C. Lamb Artur d’Avila Garcez. Neurosymbolic ai: the 3rd wave. Artificial Intelligence Review, 56(11):12387– 12406, 2023. 1
work page 2023
-
[4]
Comet: Com- monsense transformers for automatic knowledge graph con- struction
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Com- monsense transformers for automatic knowledge graph con- struction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762– 4779, 2019. 2
work page 2019
-
[5]
Towards Neuro- Symbolic Video Understanding, page 220–236
Minkyu Choi, Harsh Goel, Mohammad Omama, Yunhao Yang, Sahil Shah, and Sandeep Chinchali. Towards Neuro- Symbolic Video Understanding, page 220–236. Springer Na- ture Switzerland, 2024. 1
work page 2024
-
[6]
The epic-kitchens dataset: Collection, challenges and base- lines
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and base- lines. In International Journal of Computer Vision , pages 1255–1274, 2021. 1
work page 2021
-
[7]
Gemini: Integrating large language models and large vision models for general-purpose multimodal ai
Yixuan Du, Jiechao Feng, Tianrui Tang, Jiarui Zhang, Jiarui Chen, Yanwei Wang, Zicheng Zhang, Fei Wu, Yutong Xu, Jie Hu, et al. Gemini: Integrating large language models and large vision models for general-purpose multimodal ai. arXiv preprint arXiv:2303.17580, 2023. 1
arXiv 2023
- [8]
Show all 27 references
-
[9]
Ego4d: Around the world in 3,000 hours of egocentric video
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...
2022
-
[10]
Learning by asking questions
Drew Hudson and Christopher Manning. Learning by asking questions. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 11–20, 2019. 1
2019
-
[11]
Patching open- vocabulary vision models with commonsense
Gabriel Ilharco, Xiang Lisa Li, Trevor Darrell, Luke Zettle- moyer, Jean-Baptiste Alayrac, et al. Patching open- vocabulary vision models with commonsense. In CVPR,
-
[12]
Shamma, Michael Bernstein, and Li Fei-Fei
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. 2
2015
-
[13]
Sanjoy Kundu, Shubham Trehan, and Sathyanarayanan N. Aakur. Algo: Object-grounded visual commonsense reason- ing for open-world egocentric action recognition, 2024. 2
2024
-
[14]
Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven CH Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. arXiv preprint arXiv:2201.12086, 2022. 1
2022 arXiv
-
[15]
Neuro-symbolic concept learner: Interpreting scenes, words, and sentences from nat- ural supervision
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. Neuro-symbolic concept learner: Interpreting scenes, words, and sentences from nat- ural supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8681– 8690...
2019
-
[16]
Augmented common- sense knowledge for remote object grounding
Bahram Mohammadi, Yicong Hong, Yuankai Qi, Qi Wu, Shirui Pan, and Javen Qinfeng Shi. Augmented common- sense knowledge for remote object grounding. Proceed- ings of the AAAI Conference on Artificial Intelligence , 38 (5):4269–4277, 2024. 2
2024
-
[17]
Towards unbiased and ro- bust spatio-temporal scene graph generation and anticipa- tion, 2025
Rohith Peddi, Saurabh, Ayush Abhay Shrivastava, Parag Singla, and Vibhav Gogate. Towards unbiased and ro- bust spatio-temporal scene graph generation and anticipa- tion, 2025. 2
2025
-
[18]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learni...
2011
-
[19]
Sentence-bert: Sentence embeddings using siamese bert-networks
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguis- tics, 2019. 3
2019
-
[20]
Action scene graphs for long- form understanding of egocentric videos, 2023
Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long- form understanding of egocentric videos, 2023. 2
2023
-
[21]
Atomic: An atlas of ma- chine commonsense for if-then reasoning
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bha- gavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. Atomic: An atlas of ma- chine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, pag...
2019
-
[22]
Concept- net 5.5: An open multilingual graph of general knowledge
Robyn Speer, Joshua Chin, and Catherine Havasi. Concept- net 5.5: An open multilingual graph of general knowledge. AAAI, 2017. 2
2017
-
[23]
Concept- net 5.5: An open multilingual graph of general knowledge,
Robyn Speer, Joshua Chin, and Catherine Havasi. Concept- net 5.5: An open multilingual graph of general knowledge,
-
[24]
Learning situation hyper-graphs for video question answer- ing
Aisha Urooj, Hilde Kuehne, Bo Wu, Kim Chheu, Walid Bousselham, Chuang Gan, Niels Lobo, and Mubarak Shah. Learning situation hyper-graphs for video question answer- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14879–14889,
-
[25]
Scene graph generation by iterative message passing
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5410–5419, 2017. 2
2017
-
[26]
Neuro-symbolic visual reasoning: Disentangling ”vi- sual” from ”reasoning”
Kexin Yi, Jiajun Wu, Amar Park, Ramakrishna Vedantam, Mateusz Malinowski, Dhruv Batra, Devi Parikh, and Damien Teney. Neuro-symbolic visual reasoning: Disentangling ”vi- sual” from ”reasoning”. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition,...
2018
-
[27]
Jasper and stella: distillation of sota embedding models,
Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.