{"id":"e5d1ea25-5ca5-4f17-ae4c-17a72f002d3c","arxiv_id":"1908.03477","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Disentangling captions by parts of speech into separate learned embeddings improves cross-modal text-video retrieval for fine-grained actions.","lead":"The paper proposes a cross-modal retrieval method that splits captions into parts of speech, learns a separate embedding space for each part of speech, and combines them for fine-grained action retrieval. It reports the first retrieval results on the EPIC dataset and shows gains over single-embedding baselines on EPIC and MSR-VTT.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core claim is underdetermined: no ablation isolates PoS-aware relevance from the added capacity of multiple embedding branches.","rationale":"I focused on isolating the causal mechanism because the paper's novelty lies in PoS-aware disentanglement. The missing ablation is a standard confound: capacity and multiple views versus the specific PoS-aware inductive bias. The paper does include useful ablations (pooling, end-to-end training, learned vs identity final embedding), so this is not a general lack of rigor, but the particular factor that defines the contribution is untested. The reader's concern about semantic-class relevance is valid but applies to the benchmark's external validity; it would not change the relative ranking of methods. The proposed test is cheap and directly settles whether the central mechanism matters, so the verdict remains CONDITIONAL as the reader stated.","tokens_in":15688,"tokens_out":7257,"duration_ms":78187,"concrete_test":"Train a JPoSE-agnostic control: keep the same architecture (verb branch, noun branch, concatenation, learned f-hat/g-hat, joint training, loss weights) but set all branch losses Lk to the same PoS-agnostic action-level triplets used for L-hat. Evaluate on EPIC Seen for vt and tv and compare to JPoSE (23.2/15.8) and MMEN([Verb,Noun]) (18.7/13.6). If the control reaches or exceeds JPoSE within about 1 mAP point, the PoS-aware loss is not the driver of the reported gains; if JPoSE remains clearly ahead by more than 2 points, the concern is resolved. Ideally repeat over at least 3 random seeds to rule out noise, since no error bars are currently reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that disentangling captions into PoS-specific embedding spaces with PoS-aware relevance improves retrieval. The experiments compare JPoSE (two PoS-MMENs with PoS-specific triplets plus a final MMEN) to single MMEN variants, but no variant has the same multi-branch architecture while using the same action-level relevance in every branch. Concretely, Eq. (7) combines Lk (PoS-aware) with L-hat (PoS-agnostic); the ablations in Table 4 vary pooling, joint vs independent training, and whether f-hat/g-hat are learned, but they keep Lk PoS-aware in all rows. A model with separate verb and noun branches, concatenation, and a learned final embedding, but with all branches trained on action-level relevance (i.e., Lk replaced by L-hat), is never reported. If that variant matches JPoSE's mAP, the improvement over MMEN([Verb,Noun]) comes from extra parameters and multiple views, not from PoS disentanglement. This matters because the abstract and conclusion attribute the gains to PoS-aware losses. The relevance-class concern raised by the reader is related but secondary: even if the EPIC classes are a noisy measure of fine-grained similarity, the relative comparison would still favor JPoSE unless the missing ablation shows otherwise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JPoSE, a cross-modal embedding architecture for fine-grained action retrieval between video and text. The method first parses captions into parts of speech (mainly verbs and nouns), learns a separate multi-modal embedding network (PoS-MMEN) for each PoS with PoS-specific relevance, and then combines the branch outputs through a final learned embedding trained with a PoS-agnostic action-level loss. The authors evaluate on EPIC for fine-grained action retrieval, including a generalized zero-shot analysis, and on MSR-VTT for standard video-caption retrieval. The reported experiments show consistent improvements over single-embedding MMEN variants and several baselines.","tokens_in":15900,"tokens_out":5120,"duration_ms":55174,"significance":"If the central claim holds, the paper makes a useful contribution: it introduces a simple and modular way to inject linguistic structure into cross-modal retrieval, and it reports the first fine-grained action retrieval results on EPIC using the open-vocabulary narrations. The experimental design is generally careful: the ablations vary pooling, joint versus independent training, and whether the final embedding is learned, and the MSR-VTT experiments include several PoS combinations. The paper is also honest about the distinction between the EPIC Seen/Unseen kitchen split and zero-shot classes. The main weakness is that the ablations do not isolate the specific contribution claimed in the abstract, namely the PoS-aware relevance inside each branch; without that control, the gains could be explained by added capacity and multiple views rather than by PoS disentanglement.","major_comments":[{"comment":"The central novelty is the PoS-aware relevance, but the ablations never test it in isolation. In every row of Table 4 the branch losses Lk are built from PoS-specific relevance, and only the final loss L-hat is PoS-agnostic; Eq. (7) always contains the first term. There is no variant with the same multi-branch architecture (two MMENs, concatenation, learned f-hat/g-hat) in which all triplet losses are trained with the action-level relevance used for L-hat. Without such a control, the improvement of JPoSE over MMEN([Verb, Noun]) can be attributed to extra parameters and multiple views rather than to PoS disentanglement, which is exactly the claim in the abstract and conclusion. Please add this control, and if the result changes, rephrase the central claim.","section":"Sec. 3.3, Eq. (7); Table 4"},{"comment":"No statistical significance or variance is reported for any EPIC retrieval result, although the supplementary reports averages over 10 runs for MSR-VTT. The main improvements (vt 23.2 vs 18.7, tv 15.8 vs 13.6 on Seen; smaller gaps on Unseen) need confidence intervals or a paired test over repeated training runs to establish they are not within run-to-run noise, particularly for the zero-shot rows in Table 6 where the improvement is also used to support a separate claim.","section":"Sec. 4.1.2, Tables 2 and 3"},{"comment":"The generalized zero-shot claim is not supported by the current comparison set. The experiments compare only with MMEN(Caption), CCA, and random baselines; standard ZSL or generalized-ZSL embedding methods are not evaluated. Moreover, the paper itself notes that the EPIC Seen/Unseen split refers to kitchens, not classes, and the derived ZSV/ZSN subsets are based on vocabulary presence while evaluation relevance is based on the semantic verb/noun classes. As a result, Table 6 shows that JPoSE beats the included baselines on these subsets, but not that it is competitive in the GZSL setting as generally understood. Please either add standard GZSL baselines or limit the claim to retrieval on test instances with unseen verbs/nouns relative to the baselines considered.","section":"Sec. 4.1.2, 'Zero-shot experiments' and Table 6"}],"minor_comments":[{"comment":"The text refers to 'Table 11' and 'Table 12' for the main cross-modal and within-modal results; these should be Tables 2 and 3 in the main text.","section":"Sec. 4.1.2, text after Tables 2 and 3"},{"comment":"The main table does not state whether the reported MSR-VTT numbers are averages over 10 runs as in the supplementary; please harmonize the two presentations and include standard deviations.","section":"Sec. 4.2, Table 7 and Supplementary Table 13"},{"comment":"The table headers use symbols such as '×', 'Id', and 'indep' versus 'joint' without a full definition in the caption; the reader has to infer their meaning from the surrounding text.","section":"Table 4, caption"},{"comment":"The phrase 'combine these in a principal manner' should read 'in a principled manner'.","section":"Sec. 5, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the idea is interesting. My recommendation is driven by the missing control experiment: the authors should train a multi-branch model with the same capacity and architecture but with PoS-agnostic branch losses. If that control matches JPoSE, the central claim about PoS disentanglement would need to be substantially softened. I have no concerns about attribution or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: the PoS-disentanglement idea is genuinely new and the paper is worth reading, but the main conceptual claim is not fully pinned down by the experiments. The missing ablation flagged in the stress-test is real, and the authors should add it before we can say the gains come from PoS-aware relevance rather than just a bigger model.\n\nWhat's good: learning a separate embedding per part of speech, each with its own relevance definition (verb class for the verb embedding, noun class for the noun embedding), then stitching them with a final learned embedding, is clean and sensible. It needs no extra annotation beyond a PoS parser. The EPIC open-vocabulary retrieval benchmark is a useful addition, and the consistent improvements over the MMEN baselines on both EPIC and MSR-VTT suggest the approach has real value. The ablations that exist are careful, and the supplementary material is thorough.\n\nThe soft spots: the stress-test is on target. Every row in the ablation keeps the PoS-aware losses in the branches. There is no control with the same multi-branch architecture where all branches are trained on action-level relevance. So the jump from MMEN([Verb,Noun]) to JPoSE could be partly or fully due to extra parameters and multiple views, not the PoS-aware relevance itself. This is the one experiment that would make the central claim convincing; its absence is a significant gap, but not fatal—the method may still be useful even if the reason is partly architectural.\n\nOther issues are minor by comparison. No error bars or significance tests on EPIC (the gaps are large enough that I'd be surprised if they vanished, but still). The 'first retrieval results' on EPIC is overstated given their own earlier BMVC paper on verb-only retrieval. The zero-shot section is a reasonable descriptive analysis, but comparing against a couple of standard ZSL methods would make the GZSL framing honest. No code released.\n\nWho this is for: people working on text-video retrieval or compositional action understanding. It's a solid contribution from a good group, clearly written, with a new benchmark and an interesting method. It deserves a serious referee—the missing ablation is exactly what a good reviewer would ask for, and it's easy to run. My recommendation: engage with it, request the control experiment, and accept after that.","headline":"The PoS-disentanglement idea is genuinely new and the paper is worth reading, but the central claim is underdetermined: no ablation isolates PoS-aware relevance from the added capacity of multiple branches.","tokens_in":16424,"tokens_out":4599,"would_cite":true,"duration_ms":42241,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learning separate embedding spaces for verbs and nouns improves fine-grained video-action retrieval.","keywords":["cross-modal retrieval","fine-grained action retrieval","parts-of-speech embedding","video-text retrieval","zero-shot action retrieval","multi-modal embedding","EPIC-Kitchens","MSR-VTT"],"falsifier":"Train JPoSE and the single-embedding MMEN on EPIC with all settings identical except that the verb/noun branch split is replaced by a random partition of caption words into two fixed groups; if the random-split model matches the real JPoSE's mAP, the gains come from the two-branch architecture and joint training rather than from linguistic parts of speech.","tokens_in":15483,"feed_emoji":"🎬","tokens_out":9317,"duration_ms":86008,"temperature":0.7,"pith_summary":"Cross-modal retrieval between video and text usually learns one shared embedding space for both modalities. This paper claims that splitting the caption by part of speech—learning a separate multi-modal embedding for verbs and another for nouns—before fusing these views into a final shared space produces better fine-grained action retrieval than single-space baselines. The proposed model, JPoSE, is trained end-to-end with both PoS-aware and PoS-agnostic triplet losses. On EPIC-Kitchens, it reports the first fine-grained action retrieval results and improves both video-to-text and text-to-video mAP over the compared methods, including generalized zero-shot settings. On MSR-VTT, disentangling nouns from the rest of the caption likewise improves general video retrieval.","feed_headline":"One embedding per part of speech beats one shared space for action search","feed_subtitle":"Verbs and nouns get their own embedding spaces, then are fused; the method beats single-space baselines on two video benchmarks.","key_machinery":"The load-bearing mechanism is the PoS-MMEN, a separate multi-modal embedding network per part of speech whose positive and negative sets are defined within that part of speech. For example, the verb branch treats 'cut tomato' and 'cut carrots' as relevant while the noun branch treats 'cut tomato' and 'take tomato' as relevant; the same video features therefore get multiple views, one specialized to actions and one to objects. The branch embeddings are combined with concatenation, max, or average pooling, and the best configuration learns a final embedding with a PoS-agnostic MMEN. All components are trained jointly with the combined loss $\\hat{L}+\\sum_k \\alpha_k L_k$, where $L_k$ is the PoS-aware loss for branch $k$ and $\\hat{L}$ is the final action-retrieval loss.","core_discovery":"The paper's central claim is that disentangling parts of speech in the caption, rather than pooling all words into one representation, gives a better cross-modal embedding for fine-grained actions. In JPoSE, each part of speech gets its own multi-modal embedding network (a PoS-MMEN): the verb branch is trained so that captions and videos sharing the same verb are relevant, and the noun branch is trained so that those sharing the same noun are relevant, regardless of the other component. The outputs of these branches are fused by an encoding function and projected through a final PoS-agnostic multi-modal embedding in which action retrieval is performed. Training is joint, combining the PoS-aware triplet losses with the PoS-agnostic loss, and on EPIC-Kitchens this produces the best video-to-text and text-to-video mAP among the compared approaches, on both seen and unseen test kitchens.","pith_inferences":["Because the verb and noun branches encode complementary views, JPoSE points toward a compositional search interface in which a user queries by one action component (the verb) or one object (the noun) and then refines with the other; the paper's supplementary verb- and noun-retrieval results already hint at this capability.","Since EPIC's semantic classes are used both to build training triplets and to compute the mAP relevance labels, part of the measured improvement may reflect a better fit to those class groupings; an evaluation with independently human-judged caption-video relevance would show whether the advantage transfers to open-ended similarity.","The MSR-VTT results, where nouns alone carry most of the signal and the best split is nouns versus the rest of the caption, suggest the optimal PoS split depends on caption statistics: noun-heavy descriptive captions favor isolating nouns, while terse action captions benefit from balanced verb and noun branches.","The same disentangling principle could be applied to other paired compositional modalities—for instance audio and visual streams—where the two branches are defined by semantic roles rather than linguistic tags, without requiring extra annotation."],"forward_implications":["On EPIC-Kitchens, JPoSE achieves higher mAP than every compared single-embedding variant for both video-to-text and text-to-video retrieval, on both seen and unseen kitchens.","Within-modal search also improves: text-to-text and video-to-video mAP rise, which the paper reads as evidence that cross-modal training injects useful information into each modality's own ranking.","In the generalized zero-shot splits, where test captions contain verbs or nouns absent from training, JPoSE outperforms the caption-only MMEN, suggesting the PoS-specialized views help generalization to unseen actions.","On MSR-VTT, the adapted JPoSE that disentangles nouns from the rest of the caption consistently beats the full-caption single embedding on recall@k and median rank.","Ablations show the full gain requires joint training and a learned final embedding; independently trained PoS embeddings or simple pooling without the final MMEN perform worse."],"supporting_citations":[{"why":"Supplies EPIC-Kitchens with open-vocabulary narrations and the semantic verb/noun classes that define all relevance sets for training triplets and for the mAP evaluations.","marker":"[6]"},{"why":"Supplies MSR-VTT, the dataset used for the general video-retrieval adaptation and its recall@k and median-rank protocol.","marker":"[39]"},{"why":"Supplies the Mixture-of-Experts Embeddings used as the core network and baseline on MSR-VTT, along with the pre-extracted visual features and the pair loss adapted for JPoSE.","marker":"[22]"},{"why":"Provides the structure-preserving cross-modal embedding formulation whose cross-modal and within-modal triplet losses form the MMEN training loss.","marker":"[36]"},{"why":"Provides the two-branch neural-network and triplet-loss setup for image-text matching that the MMEN modules build on.","marker":"[19]"},{"why":"Supplies NetVLAD aggregation, used to encode video and caption features in the MSR-VTT experiments.","marker":"[3]"},{"why":"Supplies the TSN BNInception appearance and flow features that represent videos in the EPIC experiments.","marker":"[37]"},{"why":"Supplies the automatic part-of-speech parser used to split captions into verb, noun, and other tags for both datasets.","marker":"[1]"}],"fun_headline_variants":["Each verb, noun gets its own space for sharper action search","Parts-of-speech split boosts cross-modal action retrieval","Separate PoS embeddings outdo shared space for action retrieval","Disentangled PoS embeddings yield finer action retrieval","Splitting captions by PoS improves fine-grained action search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"EPIC's semantic verb and noun classes are used both to choose which video-caption pairs count as relevant during training and to compute the reported mAP; if those classes are noisy or incomplete, the scores partly measure how well the model reproduces that particular grouping rather than fine-grained action similarity itself.","fun_headline_variants_meta":{"raw":{"variants":["Each verb, noun gets its own space for sharper action search","Parts-of-speech split boosts cross-modal action retrieval","Separate PoS embeddings outdo shared space for action retrieval","Disentangled PoS embeddings yield finer action retrieval","Splitting captions by PoS improves fine-grained action search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1203,"prompt_tokens":916,"completion_tokens":287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":532,"tokens_out":287,"duration_ms":3232,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:11:04.590978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train JPoSE and the single-embedding MMEN on EPIC with all settings identical except that the verb/noun branch split is replaced by a random partition of caption words into two fixed groups; if the random-split model matches the real JPoSE's mAP, the gains come from the two-branch architecture and joint training rather than from linguistic parts of speech.","supporting_citations":[{"cited_title":"Scaling egocentric vision: The epic-kitchens dataset","cited_arxiv_id":null,"evidence_quote":"Supplies EPIC-Kitchens with open-vocabulary narrations and the semantic verb/noun classes that define all relevance sets for training triplets and for the mAP evaluations."},{"cited_title":"Msr-vtt: A large video description dataset for bridging video and language","cited_arxiv_id":null,"evidence_quote":"Supplies MSR-VTT, the dataset used for the general video-retrieval adaptation and its recall@k and median-rank protocol."},{"cited_title":"Learning deep structure-preserving image-text embeddings","cited_arxiv_id":null,"evidence_quote":"Provides the structure-preserving cross-modal embedding formulation whose cross-modal and within-modal triplet losses form the MMEN training loss."},{"cited_title":"Learning two-branch neural networks for image-text match- ing tasks","cited_arxiv_id":null,"evidence_quote":"Provides the two-branch neural-network and triplet-loss setup for image-text matching that the MMEN modules build on."},{"cited_title":"NetVLAD: CNN architecture for weakly supervised place recognition","cited_arxiv_id":null,"evidence_quote":"Supplies NetVLAD aggregation, used to encode video and caption features in the MSR-VTT experiments."},{"cited_title":"Temporal segment networks: Towards good practices for deep action recogni- tion","cited_arxiv_id":null,"evidence_quote":"Supplies the TSN BNInception appearance and flow features that represent videos in the EPIC experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the automatic part-of-speech parser used to split captions into verb, noun, and other tags for both datasets."}],"review_version":1}