Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

Diane Larlus; Dima Damen; Gabriela Csurka; Michael Wray

REVIEW 3 major objections 4 minor 42 references

Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Learning separate embedding spaces for verbs and nouns improves fine-grained video-action retrieval.

desk verdict The PoS-disentanglement idea is genuinely new and the paper is worth reading, but the central claim is underdetermined: no ablation isolates PoS-aware relevance from the added capacity of multiple branches. read the letter →

arxiv 1908.03477 v1 pith:PBOSBLTL submitted 2019-08-09 cs.CV

Michael Wray , Diane Larlus , Gabriela Csurka , Dima Damen This is my paper · ORCID

classification cs.CV

keywords cross-modalretrievalfine-grainedactionparts-of-speechembeddingvideo-textzero-shotmulti-modalEPIC-KitchensMSR-VTT

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cross-modal retrieval between video and text usually learns one shared embedding space for both modalities. This paper claims that splitting the caption by part of speech—learning a separate multi-modal embedding for verbs and another for nouns—before fusing these views into a final shared space produces better fine-grained action retrieval than single-space baselines. The proposed model, JPoSE, is trained end-to-end with both PoS-aware and PoS-agnostic triplet losses. On EPIC-Kitchens, it reports the first fine-grained action retrieval results and improves both video-to-text and text-to-video mAP over the compared methods, including generalized zero-shot settings. On MSR-VTT, disentangling nouns from the rest of the caption likewise improves general video retrieval.

What carries the argument

The load-bearing mechanism is the PoS-MMEN, a separate multi-modal embedding network per part of speech whose positive and negative sets are defined within that part of speech. For example, the verb branch treats 'cut tomato' and 'cut carrots' as relevant while the noun branch treats 'cut tomato' and 'take tomato' as relevant; the same video features therefore get multiple views, one specialized to actions and one to objects. The branch embeddings are combined with concatenation, max, or average pooling, and the best configuration learns a final embedding with a PoS-agnostic MMEN. All components are trained jointly with the combined loss $\hat{L}+\sum_k \alpha_k L_k$, where $L_k$ is the PoS-aware loss for branch $k$ and $\hat{L}$ is the final action-retrieval loss.

What would settle it

Train JPoSE and the single-embedding MMEN on EPIC with all settings identical except that the verb/noun branch split is replaced by a random partition of caption words into two fixed groups; if the random-split model matches the real JPoSE's mAP, the gains come from the two-branch architecture and joint training rather than from linguistic parts of speech.

Watch

Extended reading notes

Core claim

The paper's central claim is that disentangling parts of speech in the caption, rather than pooling all words into one representation, gives a better cross-modal embedding for fine-grained actions. In JPoSE, each part of speech gets its own multi-modal embedding network (a PoS-MMEN): the verb branch is trained so that captions and videos sharing the same verb are relevant, and the noun branch is trained so that those sharing the same noun are relevant, regardless of the other component. The outputs of these branches are fused by an encoding function and projected through a final PoS-agnostic multi-modal embedding in which action retrieval is performed. Training is joint, combining the PoS-aware triplet losses with the PoS-agnostic loss, and on EPIC-Kitchens this produces the best video-to-text and text-to-video mAP among the compared approaches, on both seen and unseen test kitchens.

Load-bearing premise

EPIC's semantic verb and noun classes are used both to choose which video-caption pairs count as relevant during training and to compute the reported mAP; if those classes are noisy or incomplete, the scores partly measure how well the model reproduces that particular grouping rather than fine-grained action similarity itself.

Editorial extensions

If this is right

On EPIC-Kitchens, JPoSE achieves higher mAP than every compared single-embedding variant for both video-to-text and text-to-video retrieval, on both seen and unseen kitchens.
Within-modal search also improves: text-to-text and video-to-video mAP rise, which the paper reads as evidence that cross-modal training injects useful information into each modality's own ranking.
In the generalized zero-shot splits, where test captions contain verbs or nouns absent from training, JPoSE outperforms the caption-only MMEN, suggesting the PoS-specialized views help generalization to unseen actions.
On MSR-VTT, the adapted JPoSE that disentangles nouns from the rest of the caption consistently beats the full-caption single embedding on recall@k and median rank.
Ablations show the full gain requires joint training and a learned final embedding; independently trained PoS embeddings or simple pooling without the final MMEN perform worse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the verb and noun branches encode complementary views, JPoSE points toward a compositional search interface in which a user queries by one action component (the verb) or one object (the noun) and then refines with the other; the paper's supplementary verb- and noun-retrieval results already hint at this capability.
Since EPIC's semantic classes are used both to build training triplets and to compute the mAP relevance labels, part of the measured improvement may reflect a better fit to those class groupings; an evaluation with independently human-judged caption-video relevance would show whether the advantage transfers to open-ended similarity.
The MSR-VTT results, where nouns alone carry most of the signal and the best split is nouns versus the rest of the caption, suggest the optimal PoS split depends on caption statistics: noun-heavy descriptive captions favor isolating nouns, while terse action captions benefit from balanced verb and noun branches.
The same disentangling principle could be applied to other paired compositional modalities—for instance audio and visual streams—where the two branches are defined by semantic roles rather than linguistic tags, without requiring extra annotation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Plainly: the PoS-disentanglement idea is genuinely new and the paper is worth reading, but the main conceptual claim is not fully pinned down by the experiments. The missing ablation flagged in the stress-test is real, and the authors should add it before we can say the gains come from PoS-aware relevance rather than just a bigger model.

What's good: learning a separate embedding per part of speech, each with its own relevance definition (verb class for the verb embedding, noun class for the noun embedding), then stitching them with a final learned embedding, is clean and sensible. It needs no extra annotation beyond a PoS parser. The EPIC open-vocabulary retrieval benchmark is a useful addition, and the consistent improvements over the MMEN baselines on both EPIC and MSR-VTT suggest the approach has real value. The ablations that exist are careful, and the supplementary material is thorough.

The soft spots: the stress-test is on target. Every row in the ablation keeps the PoS-aware losses in the branches. There is no control with the same multi-branch architecture where all branches are trained on action-level relevance. So the jump from MMEN([Verb,Noun]) to JPoSE could be partly or fully due to extra parameters and multiple views, not the PoS-aware relevance itself. This is the one experiment that would make the central claim convincing; its absence is a significant gap, but not fatal—the method may still be useful even if the reason is partly architectural.

Other issues are minor by comparison. No error bars or significance tests on EPIC (the gaps are large enough that I'd be surprised if they vanished, but still). The 'first retrieval results' on EPIC is overstated given their own earlier BMVC paper on verb-only retrieval. The zero-shot section is a reasonable descriptive analysis, but comparing against a couple of standard ZSL methods would make the GZSL framing honest. No code released.

Who this is for: people working on text-video retrieval or compositional action understanding. It's a solid contribution from a good group, clearly written, with a new benchmark and an interesting method. It deserves a serious referee—the missing ablation is exactly what a good reviewer would ask for, and it's easy to run. My recommendation: engage with it, request the control experiment, and accept after that.

Referee Report

3 major / 4 minor

Summary. The paper proposes JPoSE, a cross-modal embedding architecture for fine-grained action retrieval between video and text. The method first parses captions into parts of speech (mainly verbs and nouns), learns a separate multi-modal embedding network (PoS-MMEN) for each PoS with PoS-specific relevance, and then combines the branch outputs through a final learned embedding trained with a PoS-agnostic action-level loss. The authors evaluate on EPIC for fine-grained action retrieval, including a generalized zero-shot analysis, and on MSR-VTT for standard video-caption retrieval. The reported experiments show consistent improvements over single-embedding MMEN variants and several baselines.

Significance. If the central claim holds, the paper makes a useful contribution: it introduces a simple and modular way to inject linguistic structure into cross-modal retrieval, and it reports the first fine-grained action retrieval results on EPIC using the open-vocabulary narrations. The experimental design is generally careful: the ablations vary pooling, joint versus independent training, and whether the final embedding is learned, and the MSR-VTT experiments include several PoS combinations. The paper is also honest about the distinction between the EPIC Seen/Unseen kitchen split and zero-shot classes. The main weakness is that the ablations do not isolate the specific contribution claimed in the abstract, namely the PoS-aware relevance inside each branch; without that control, the gains could be explained by added capacity and multiple views rather than by PoS disentanglement.

major comments (3)

[Sec. 3.3, Eq. (7); Table 4] The central novelty is the PoS-aware relevance, but the ablations never test it in isolation. In every row of Table 4 the branch losses Lk are built from PoS-specific relevance, and only the final loss L-hat is PoS-agnostic; Eq. (7) always contains the first term. There is no variant with the same multi-branch architecture (two MMENs, concatenation, learned f-hat/g-hat) in which all triplet losses are trained with the action-level relevance used for L-hat. Without such a control, the improvement of JPoSE over MMEN([Verb, Noun]) can be attributed to extra parameters and multiple views rather than to PoS disentanglement, which is exactly the claim in the abstract and conclusion. Please add this control, and if the result changes, rephrase the central claim.
[Sec. 4.1.2, Tables 2 and 3] No statistical significance or variance is reported for any EPIC retrieval result, although the supplementary reports averages over 10 runs for MSR-VTT. The main improvements (vt 23.2 vs 18.7, tv 15.8 vs 13.6 on Seen; smaller gaps on Unseen) need confidence intervals or a paired test over repeated training runs to establish they are not within run-to-run noise, particularly for the zero-shot rows in Table 6 where the improvement is also used to support a separate claim.
[Sec. 4.1.2, 'Zero-shot experiments' and Table 6] The generalized zero-shot claim is not supported by the current comparison set. The experiments compare only with MMEN(Caption), CCA, and random baselines; standard ZSL or generalized-ZSL embedding methods are not evaluated. Moreover, the paper itself notes that the EPIC Seen/Unseen split refers to kitchens, not classes, and the derived ZSV/ZSN subsets are based on vocabulary presence while evaluation relevance is based on the semantic verb/noun classes. As a result, Table 6 shows that JPoSE beats the included baselines on these subsets, but not that it is competitive in the GZSL setting as generally understood. Please either add standard GZSL baselines or limit the claim to retrieval on test instances with unseen verbs/nouns relative to the baselines considered.

minor comments (4)

[Sec. 4.1.2, text after Tables 2 and 3] The text refers to 'Table 11' and 'Table 12' for the main cross-modal and within-modal results; these should be Tables 2 and 3 in the main text.
[Sec. 4.2, Table 7 and Supplementary Table 13] The main table does not state whether the reported MSR-VTT numbers are averages over 10 runs as in the supplementary; please harmonize the two presentations and include standard deviations.
[Table 4, caption] The table headers use symbols such as '×', 'Id', and 'indep' versus 'joint' without a full definition in the caption; the reader has to infer their meaning from the surrounding text.
[Sec. 5, Conclusion] The phrase 'combine these in a principal manner' should read 'in a principled manner'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the method is evaluated on held-out benchmarks and the PoS-aware relevance is an external annotation convention, not a fitted input.

full rationale

The derivation chain is self-contained rather than circular. The JPoSE model (Eq. 7) combines PoS-aware losses Lk with a PoS-agnostic loss ÊˆL, all defined from video features, Word2Vec caption embeddings, and external verb/noun semantic classes from EPIC; no term in the objective is defined in terms of the retrieval result it is meant to explain. The evaluation uses held-out seen/unseen test splits and standard mAP, and the fact that the same semantic classes define both training triplets and test relevance is a consistent benchmark convention rather than a fitted-input-called-prediction step: no parameter is fit to the test set and the relevance classes come from the dataset annotations, not from the model. The self-citations that appear ([6] for the EPIC dataset and [38] for the authors' prior verb-only retrieval work) are related-work or data-source citations and are not load-bearing for the paper's central claim. The remaining concern that no ablation isolates PoS-aware relevance from added multi-branch capacity is an experimental underdetermination issue, not a circularity, because the reported gains are empirical comparisons on held-out data rather than consequences of the definitions.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The approach introduces no new physical or conceptual entities; it is an architectural combination of existing components (triplet loss, joint embedding, PoS parsing). The main external dependencies are the spaCy parser, the EPIC semantic classes, and the pre-trained visual and textual features, all treated as given inputs.

free parameters (3)

embedding dimension = 256
Set to 256, described as 'a dimension we found to be suitable across all settings'; no sensitivity analysis is shown, and the choice affects all embedding spaces.
triplet margin gamma = not reported
The equations use a constant margin gamma, but its numeric value is not stated in the paper, making it a hidden hyperparameter that affects the loss.
loss weights lambda and alpha = lambda_vv = lambda_tt = 0.1, lambda_vt = lambda_tv = 1.0, alpha_k = 1
Chosen without a reported grid search; these weights control the balance between PoS-aware and PoS-agnostic losses and could influence the final results.

assumptions (3)

domain assumption spaCy PoS parser correctly tags English captions
The method splits captions into parts of speech using the spaCy parser; any tagging errors propagate into the PoS-specific embeddings and the final retrieval space.
domain assumption EPIC verb and noun semantic classes correctly define retrieval relevance
Both the training triplets and the evaluation mAP are built on these semantic classes, as described in Sec. 4.1. If the grouping is inaccurate, the reported performance does not reflect true action retrieval.
domain assumption Word2Vec text features and TSN video features are sufficient input representations
The method operates on fixed pre-extracted features (100-dim Word2Vec and TSN BNInception features) and does not learn from raw words or pixels, so the quality of these features bounds the achievable performance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings." pith.science (2026). https://pith.science/paper/PBOSBLTL

@misc{pith2026190803477,
  author       = {Pith},
  title        = {Pith review of: Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PBOSBLTL}},
  note         = {Machine review of arXiv:1908.03477}
}

read the original abstract

We address the problem of cross-modal fine-grained action retrieval between text and video. Cross-modal retrieval is commonly achieved through learning a shared embedding space, that can indifferently embed modalities. In this paper, we propose to enrich the embedding by disentangling parts-of-speech (PoS) in the accompanying captions. We build a separate multi-modal embedding space for each PoS tag. The outputs of multiple PoS embeddings are then used as input to an integrated multi-modal space, where we perform action retrieval. All embeddings are trained jointly through a combination of PoS-aware and PoS-agnostic losses. Our proposal enables learning specialised embedding spaces that offer multiple views of the same embedded entities. We report the first retrieval results on fine-grained actions for the large-scale EPIC dataset, in a generalised zero-shot setting. Results show the advantage of our approach for both video-to-text and text-to-video action retrieval. We also demonstrate the benefit of disentangling the PoS for the generic task of cross-modal video retrieval on the MSR-VTT dataset.

Figures

Figures reproduced from arXiv: 1908.03477 by the authors.

**Figure 2.** Overview of the JPoSE model. We first disentangle a caption into its parts of speech (PoS) and learn a Multi-Modal Embedding [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Qualitative results for video-to-text (top) and text-to-video (bottom) action retrieval on EPIC. For several query videos (top) or [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Maximum activation examples for visual embedding in the noun (left) and the verb (right) PoS-MMEN. Examples of similar [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Qualitative results of text-to-video action retrieval on MSR-VTT. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 35 canonical work pages

[1]

English spaCy parser https://spacy.io/. 4

work page
[2]

Re-id done right: towards good practices for person re- identiﬁcation

Jon Almaz ´an, Bojana Gajic, Naila Murray, and Diane Lar- lus. Re-id done right: towards good practices for person re- identiﬁcation. CoRR, abs/1801.05339, 2018. 2

arXiv 2018
[3]

NetVLAD: CNN architecture for weakly supervised place recognition

Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pa- jdla, and Josef Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In CVPR, 2016. 2, 8

work page 2016
[4]

An empirical study and analysis of generalized zero- shot learning for object recognition in the wild

Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero- shot learning for object recognition in the wild. In ECCV,

work page
[5]

Beyond triplet loss: a deep quadruplet network for person re-identiﬁcation

Weihua Chen, Xiaotang Chen, Ianguo Zhang, and Kaiqi Huang. Beyond triplet loss: a deep quadruplet network for person re-identiﬁcation. In CVPR, 2017. 2

work page 2017
[6]

Scaling egocentric vision: The epic-kitchens dataset

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. 2, 4, 5

work page 2018
[7]

Predict- ing visual features from text for image and video caption re- trieval

Jianfeng Dong, Xirong Li, and Cees GM Snoek. Predict- ing visual features from text for image and video caption re- trieval. IEEE Trans. Multimed., 2018. 1

work page 2018
[8]

Dual Encoding for Zero-Example Video Retrieval

Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, and Xun Wang. Dual dense encoding for zero-example video re- trieval. CoRR, arXiv:1809.06181, 2018. 1, 2, 7

work page Pith review arXiv 2018

Show all 42 references

[9]

Improving image-sentence embeddings using large weakly annotated photo collections

Yunchao Gong, Liwei Wang, Micah Hodosh, Julia Hocken- maier, and Svetlana Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In ECCV, 2014. 2, 6

2014
[10]

Deep image retrieval: Learning global representations for image search

Albert Gordo, Jon Almaz ´an, J´erome Revaud, and Diane Lar- lus. Deep image retrieval: Learning global representations for image search. In ECCV, 2016. 2

2016
[11]

Beyond instance-level im- age retrieval: Leveraging captions to learn a global visual representation for semantic retrieval

Albert Gordo and Diane Larlus. Beyond instance-level im- age retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In CVPR, 2017. 2

2017
[12]

Something Something

Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The “Something Something” Video Dat...

2017
[13]

Ava: A video dataset of spatio-temporally localized atomic visual actions

Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Car- oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018. 2

2018
[14]

Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition

Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkar- nenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV, 2013. 2

2013
[15]

Meera Hahn, Andrew Silva, and James M. Rehg. Ac- tion2vec: A crossmodal embedding approach to action learn- ing. In BMVC, 2018. 1, 2

2018
[16]

In de- fense of the triplet loss for person re-identiﬁcation

Alexander Hermans, Lucas Beyer, and Bastian Leibe. In de- fense of the triplet loss for person re-identiﬁcation. CoRR, arXiv:1703.07737, 2017. 2

2017 arXiv
[17]

Deep metric learning using triplet network

Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In ICLR, 2015. 2

2015
[18]

Unifying visual-semantic embeddings with multimodal neu- ral language models

Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neu- ral language models. TACL, 2015. 2

2015
[19]

Learning two-branch neural networks for image-text match- ing tasks

Wang Liwei, Li Yin, Huang Jing, and Svetlana Lazebnik. Learning two-branch neural networks for image-text match- ing tasks. TPAMI, 2019. 3

2019
[20]

On the effec- tiveness of task granularity for transfer learning

Farzaneh Mahdisoltani, Guillaume Berger, Waseem Ghar- bieh, David Fleet, and Roland Memisevic. On the effec- tiveness of task granularity for transfer learning. CoRR, arXiv:1804.09235, 2018. 1, 2

2018 arXiv
[21]

Learnable pooling with context gating for video classiﬁcation

Antoine Miech, Ivan Laptev, and Josef Sivic. Learnable pooling with context gating for video classiﬁcation. CoRR, arXiv:1706.06905, 2017. 2

2017 arXiv
[22]

Learning a text-video embedding from incomplete and heterogeneous data

Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a text-video embedding from incomplete and heterogeneous data. CoRR, arXiv:1804.02516, 2018. 1, 2, 7, 8, 12

2018 arXiv
[23]

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV,
[24]

Learning joint embedding with multimodal cues for cross-modal video-text retrieval

Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In ICMR, 2018. 1, 2, 7

2018
[25]

Learning joint representations of videos and sentences with web image search

Mayu Otani, Yuta Nakashima, Esa Rahtu, Janne Heikkil ¨a, and Naokazu Yokoya. Learning joint representations of videos and sentences with web image search. In ECCV,
[26]

Enhancing video summarization via vision-language embed- ding

Bryan A Plummer, Matthew Brown, and Svetlana Lazebnik. Enhancing video summarization via vision-language embed- ding. In CVPR, 2017. 1

2017
[27]

CNN image retrieval learns from BoW: Unsupervised ﬁne-tuning with hard examples

Filip Radenovi ´c, Giorgos Tolias, and Ond ˇrej Chum. CNN image retrieval learns from BoW: Unsupervised ﬁne-tuning with hard examples. In ECCV, 2016. 2

2016
[28]

Recognizing ﬁne-grained and composite ac- tivities using hand-centric features and script data

Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Recognizing ﬁne-grained and composite ac- tivities using hand-centric features and script data. IJCV,
[29]

Facenet: A uniﬁed embedding for face recognition and clus- tering

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A uniﬁed embedding for face recognition and clus- tering. In CVPR, 2015. 2

2015
[30]

High order neural networks for video classiﬁcation

Jie Shao, Kai Hu, Yixin Bao, Yining Lin, and Xiangyang Xue. High order neural networks for video classiﬁcation. CoRR, arXiv:1811.07519, 2018. 1

2018 arXiv
[31]

Sigurdsson, G ¨ul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta

Gunnar A. Sigurdsson, G ¨ul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity under- standing. In ECCV, 2016. 2

2016
[32]

Improved deep metric learning with multi- class n-pair loss objective

Kihyuk Sohn. Improved deep metric learning with multi- class n-pair loss objective. In NIPS, 2016. 2

2016
[33]

Cross modal embeddings for video and audio retrieval

Didac Sur ´ıs, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Gir´oo-i Nieto. Cross modal embeddings for video and audio retrieval. In ECCVW, 2018. 2

2018
[34]

Learning language-visual embedding for movie understanding with natural-language

Atousa Torabi, Niket Tandon, and Leonid Sigal. Learning language-visual embedding for movie understanding with natural-language. CoRR, arXiv:1609.08124, 2016. 2

2016 arXiv
[35]

Learn- ing ﬁne-grained image similarity with deep ranking

Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learn- ing ﬁne-grained image similarity with deep ranking. In CVPR, 2014. 2

2014
[36]

Learning deep structure-preserving image-text embeddings

Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings. In CVPR,
[37]

Temporal segment networks: Towards good practices for deep action recogni- tion

Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, 2016. 5

2016
[38]

Learning visual actions using multiple verb-only labels

Michael Wray and Dima Damen. Learning visual actions using multiple verb-only labels. In BMVC, 2019. 2

2019
[39]

Msr-vtt: A large video description dataset for bridging video and language

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 2, 4, 7

2016
[40]

Jointly modeling deep video and compositional text to bridge vision and language in a uniﬁed framework

Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso. Jointly modeling deep video and compositional text to bridge vision and language in a uniﬁed framework. In AAAI, 2015. 2

2015
[41]

A joint se- quence fusion model for video question answering and re- trieval

Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint se- quence fusion model for video question answering and re- trieval. In ECCV, 2018. 7

2018
[42]

Zero-shot learning via semantic similarity embedding

Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity embedding. In ICCV, 2015. 2 EPIC SEEN vv vt tv tt Random Baseline 12.6 12.6 12.6 12.6 Features(Word2Vec) – – – 50.0 Features(Video) 21.0 – – – CCA Baseline 21.3 23.3 25.7 37.7 MMEN(Caption) 32.0 5...

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

English spaCy parser https://spacy.io/. 4

work page

[2] [2]

Re-id done right: towards good practices for person re- identiﬁcation

Jon Almaz ´an, Bojana Gajic, Naila Murray, and Diane Lar- lus. Re-id done right: towards good practices for person re- identiﬁcation. CoRR, abs/1801.05339, 2018. 2

arXiv 2018

[3] [3]

NetVLAD: CNN architecture for weakly supervised place recognition

Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pa- jdla, and Josef Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In CVPR, 2016. 2, 8

work page 2016

[4] [4]

An empirical study and analysis of generalized zero- shot learning for object recognition in the wild

Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero- shot learning for object recognition in the wild. In ECCV,

work page

[5] [5]

Beyond triplet loss: a deep quadruplet network for person re-identiﬁcation

Weihua Chen, Xiaotang Chen, Ianguo Zhang, and Kaiqi Huang. Beyond triplet loss: a deep quadruplet network for person re-identiﬁcation. In CVPR, 2017. 2

work page 2017

[6] [6]

Scaling egocentric vision: The epic-kitchens dataset

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. 2, 4, 5

work page 2018

[7] [7]

Predict- ing visual features from text for image and video caption re- trieval

Jianfeng Dong, Xirong Li, and Cees GM Snoek. Predict- ing visual features from text for image and video caption re- trieval. IEEE Trans. Multimed., 2018. 1

work page 2018

[8] [8]

Dual Encoding for Zero-Example Video Retrieval

Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, and Xun Wang. Dual dense encoding for zero-example video re- trieval. CoRR, arXiv:1809.06181, 2018. 1, 2, 7

work page Pith review arXiv 2018