{"id":"7010c18c-b651-446d-8709-22bcdb73063b","arxiv_id":"2411.15628","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ACE fine-tunes video-language models with stochastically sampled action synonyms and shadow negatives, improving zero-shot classification of unseen procedural actions by up to 16 percent harmonic mean on ATA, IKEA, and GTEA.","lead":"This paper introduces a fine-tuning trick that makes video-language models more robust to different ways of describing the same action, like 'spin block' versus 'rotate block'. The method, ACE, randomly swaps action labels with their synonyms during training and adds misleading 'shadow' labels, improving zero-shot recognition of unseen actions on cooking and assembly video benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GTEA Synonym Robustness Test shows no accuracy gain over ProcVLR, contradicting the claimed universal improvement.","rationale":"The reader identified GPT-4 synonym quality as the weakest assumption, citing Table 4. This is a legitimate concern, but the paper itself already acknowledges sensitivity to synonym quality, and ACE still outperforms baselines with GPT-4 synonyms on ATA and IKEA. A more load-bearing issue is that the paper's own Table 2 shows no SRT accuracy improvement over the direct backbone baseline ProcVLR on GTEA, directly contradicting the stated claim of 'higher mean accuracy and lower standard deviation' on all datasets. This is not a matter of external consensus but an internal inconsistency in the reported results. The paper otherwise provides valuable evidence: three datasets, ablations, and a direct baseline comparison. The conditional verdict remains appropriate, but the authors must correct the overclaim and provide statistical support for the GTEA results. The proposed test would settle whether the GTEA finding is a real effect or noise.","tokens_in":17754,"tokens_out":8421,"duration_ms":74569,"concrete_test":"Recompute the GTEA SRT results from Table 2 and run a paired significance test (e.g., Wilcoxon signed-rank or paired t-test) over the 10 synonym runs, or over the 4 cross-validation folds, comparing ACE against ProcVLR. If the accuracy difference is not significant (p>0.05), the claim of higher mean SRT accuracy on GTEA should be retracted and the abstract revised. Also report standard deviations or confidence intervals for the default-label GTEA unseen accuracy to assess whether the 3.4-point gain over Text4Vis is meaningful.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in the abstract and Section 4.2 is that ACE achieves higher mean accuracy and lower standard deviation on the Synonym Robustness Test across all datasets. Table 2 (GTEA) contradicts this: ProcVLR, the one-to-one baseline sharing the same TimeSformer backbone and Howto100M pretraining, obtains SRT unseen accuracy 45.0 ±16.9, while ACE obtains 45.0 ±16.8. Accuracy is identical; only F1 improves (28.8 vs 32.4). This directly undermines the claim that ACE enhances synonym robustness on GTEA. The default-label unseen gain over the best baseline (Text4Vis) is only 3.4 accuracy points (67.2 vs 63.8), with no reported error bars over cross-validation folds, so the GTEA evidence for the headline improvement is weak. The paper does not report a significance test for this difference. Because GTEA is the only egocentric and cooking dataset, the failure there indicates the method's benefit is dataset-dependent and the universal claim is not supported by the paper's own results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ACE, a fine-tuning technique for vision-language models applied to procedural video action classification. ACE builds per-verb synonym trees using GPT-4, then during training stochastically replaces ground-truth action labels with randomly sampled synonyms in an auxiliary classification loss and augments the classifier with object-matched \"shadow negatives\" that pair the correct object with a wrong verb. The authors evaluate on ATA, IKEA, and GTEA in a base-to-novel zero-shot setting, reporting consistent gains over existing VLM baselines on seen and unseen default labels, as well as a new Synonym Robustness Test (SRT) that measures accuracy/F1 over 10 random synonym sets for unseen actions. They also provide ablations, an analysis of fine-tuned layers, a manual-synonym comparison, and TSNE visualizations of the learned concept spaces.","tokens_in":17988,"tokens_out":3878,"duration_ms":35242,"significance":"If the reported results hold, ACE is a simple and practical recipe for improving VLM robustness to label variation in procedural domains, with potential application to cooking and assembly assistants. The paper's strengths include a clear training-time augmentation scheme, evaluation across three datasets with multiple baselines, component ablations, and a stated intention to release code. The experiments with human-annotated synonyms (Table 4) are a particularly valuable sanity check of the LLM-generated synonym quality. However, the paper's claim of universal improvement in synonym robustness is contradicted by its own GTEA SRT results, where unseen accuracy exactly matches a baseline; the lack of variance estimates for default-label results further weakens the \"significantly beats\" language. These issues need to be addressed before the central claims can be fully accepted.","major_comments":[{"comment":"The GTEA Synonym Robustness Test shows ACE's unseen accuracy as 45.0±16.8, numerically identical to ProcVLR's 45.0±16.9. This directly contradicts the text's statement that ACE \"achieves the highest mean acc and F1 while maintaining low std in all datasets\" and the claim that ACE \"enhances the action concept understanding of ProcVLR consistently on all datasets.\" The only SRT improvement on GTEA is in F1 (32.4 vs. 28.8), and no significance test is reported. Since GTEA is the only egocentric and cooking dataset in the study, the universal robustness claim is not supported; the authors should either provide statistical evidence for a meaningful difference or revise the claim to reflect the dataset-dependent nature of the gains.","section":"Section 4.2, Table 2"},{"comment":"Default-label zero-shot results are reported as single point estimates without standard deviations or significance tests across cross-validation folds, although the experimental setup states that IKEA uses 5 splits and GTEA uses 4-fold cross-validation. For GTEA, the unseen-accuracy gain over Text4Vis is only 3.4 points (67.2 vs. 63.8), and the F1 is actually lower (41.0 vs. 47.4). Without error bars or a paired test, the claim that ACE \"significantly beats the SoTA in zero-shot classification\" is not statistically supported, especially on the small GTEA test set. The authors should report per-fold results or at least mean±std for the default-label experiments.","section":"Section 4.1/4.2, Tables 1 and 2"},{"comment":"The Synonym Robustness Test uses GPT-4-generated synonyms, the same LLM that supplies training-time augmentations. Because ACE explicitly trains the encoder to align with GPT-4's synonym distribution, SRT may overstate concept understanding relative to human-defined synonyms. Table 4 shows that the GTEA SRT accuracy jumps from 45.0 with GPT test trees to 63.6 with manual test trees, and drops to 41.1 when training on manual trees but testing on GPT trees, confirming a strong sensitivity to the synonym source. The paper should explicitly acknowledge this as a limitation of SRT and, if feasible, include a human-evaluated SRT on at least one additional dataset to demonstrate generalization beyond the LLM's distribution.","section":"Section 3.2/4.3, Table 4"}],"minor_comments":[{"comment":"The notation for the synonym set is confusing: the text defines v+ = Synonyms(v) ∪ {v} and then uses M as the number of children, but the equation averages over M items without clarifying that this includes the parent node. Please define M consistently and state whether the parent replication is counted in M.","section":"Section 3.2, Eq. (3)"},{"comment":"The number of synonym children M varies across datasets (2 for IKEA, 9 for GTEA, 11 for ATA) without a stated criterion. Since M is a free parameter that directly affects the amount of augmentation, the authors should justify this choice or show sensitivity to M on more than the ATA dataset.","section":"Section 4.1, Implementation Details"},{"comment":"There are typographical errors in the table captions and section title: \"GPT-genrated\", \"Robustnss\", and \"genrated\" should be corrected to \"GPT-generated\" and \"Robustness\".","section":"Appendix, Tables 7-9"},{"comment":"The comparison with LaVila in Table 6 is acknowledged as not entirely fair due to different pretraining data. It would be helpful to state explicitly in the main text that LaVila's Ego4D pretraining provides an advantage on egocentric GTEA, even though the appendix does mention this.","section":"Section 4.2, LaVila comparison"}],"recommendation":"major_revision","confidential_remarks":"The identical GTEA SRT accuracy with ProcVLR is a notable inconsistency with the paper's own textual claims, and the lack of error bars for default-label results is a recurring weakness. The authors should be asked to either add statistical evidence or temper the universal-robustness claim. The manual-synonym result (Table 4) is a strong positive that partially mitigates concerns about LLM circularity, but the paper would benefit from an explicit limitations paragraph. The method is simple and the ATA/IKEA gains are substantial, so a major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. ACE is a simple, sensible idea: during fine-tuning, stochastically replace action labels with LLM-generated synonyms, compute similarity by averaging over a synonym leaf, and add a per-class object-matched negative. The paper does the work of showing each piece matters, and on ATA and IKEA the gains are real and large—harmonic mean up 16 and 11 points over the next best baseline. The SRT protocol, testing robustness to 10 random synonym sets, is a useful addition to the toolkit.\n\nNow the soft spots. The paper claims in the abstract and in Section 4.2 that ACE achieves higher SRT accuracy on all datasets. Table 2 contradicts this on GTEA: unseen SRT accuracy is exactly 45.0 for both ACE and ProcVLR, with standard deviation 16.8 vs 16.9. F1 does improve (32.4 vs 28.8), so there is a real but much weaker effect. Similarly, the default-label unseen gain over Text4Vis on GTEA is only 3.4 accuracy points (67.2 vs 63.8) with no error bars over the four folds. Given GTEA is the only egocentric and cooking dataset, the evidence suggests the method's benefit is dataset-dependent, not universal.\n\nThere are minor complaints too. The synonym tree sizes differ widely across datasets (2, 9, 11), and the paper doesn't explain how these numbers were chosen beyond saying they were adjusted. The novel class selection by least frequent verbs is reasonable but makes cross-dataset comparison of gains tricky. The authors promise to release code and data; that matters here because the LLM-generated trees are central to reproducibility. The manual synonym test (Table 4) is honest and exposes a real limitation—synonym quality strongly affects the result—but it also shows the method can still beat SoTA with GPT synonyms.\n\nOn balance, the central mechanism is plausible, the ablations are informative, and the ATA/IKEA results are genuinely strong. The GTEA overclaim is a fixable problem, not a fatal one. This is exactly the kind of paper that deserves serious refereeing, with a request to correct the universal claim and add error bars or significance tests on GTEA. I'd bring it to reading group, and I'd cite the SRT protocol if I were working on label augmentation for VLMs.","headline":"Useful fine-tuning recipe with strong ATA/IKEA gains, but the universal SRT claim is contradicted by the paper's own GTEA numbers.","tokens_in":18464,"tokens_out":2519,"would_cite":true,"duration_ms":21122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ACE fine-tuning stochastically replaces fixed action labels with LLM-generated synonyms and shadow negatives, improving zero-shot classification of unseen procedural actions.","keywords":["action concept enhancement","zero-shot action recognition","procedural videos","synonym robustness","vision-language models","fine-tuning","action synonyms","shadow negatives"],"falsifier":"Run ACE with the same training pipeline but replace each LLM synonym with a randomly drawn verb from a general dictionary (keeping the object fixed). If zero-shot robustness on the Synonym Robustness Test stays at the same high level, the gains are due to label noise rather than to the semantic relation encoded in the synonyms; if it collapses, the concept-semantics explanation is supported.","tokens_in":17594,"feed_emoji":"🎬","tokens_out":6137,"duration_ms":50140,"temperature":0.7,"pith_summary":"Vision-language models can recognize actions they were not trained on, but they overfit to the exact wording of action labels and fail when the same procedural step is described by a synonym. This paper proposes Action Concept Enhancement (ACE), a fine-tuning method that continually swaps fixed action labels for LLM-generated synonyms and adds object-matched 'shadow negatives' during training. ACE reports large gains on zero-shot classification of unseen actions in cooking and assembly videos, with harmonic mean accuracy up to 16, 11, and 4 percent above the second-best baselines on ATA, IKEA, and GTEA, while staying competitive on seen actions. The paper also introduces a Synonym Robustness Test that measures mean accuracy and variance over ten randomly chosen synonym sets, and ACE shows higher mean and lower variance than the baselines there. If the method holds, robustness to unseen action wording becomes a trainable property rather than a fixed limitation of pretrained video-language models.","feed_headline":"Synonym shuffling lifts zero-shot action recognition up to 16%","feed_subtitle":"ACE swaps fixed labels for LLM synonyms and shadow negatives, gaining up to 16% harmonic mean accuracy.","key_machinery":"The central object is the verb synonym tree: each action is decomposed into a verb-object pair, and the verb is expanded into a tree of LLM-generated synonyms, with the parent repeated as a child at every level to preserve semantic continuity. The argument runs on three mechanisms built from these trees: leaf augmentation in the similarity measure (average over a node's synonym children), randomized action synonyms in an auxiliary classification loss ($L_{rand}$) that creates up to $M^C$ label combinations, and shadow negatives that share the true object but pair it with a wrong verb, extending classification to $C+1$ classes. The fixed root-label loss $L_{fixed}$ anchors the concept while the stochastic loss prevents overfitting to any single surface form.","core_discovery":"The paper's central claim is that stochastic replacement of action labels during fine-tuning teaches a video-language model the action concept behind the label, not just the label string. ACE builds a synonym tree for each action verb, where the root is the dataset's annotated verb and descendants are LLM-generated synonyms up to second order. During each training iteration, every video is classified twice: once against the fixed root labels and once against a freshly randomized set of verb synonyms assigned to the same objects. A 'shadow negative' category, which pairs the true object with a wrong verb, is added to force the model to attend to verbs rather than objects. At inference, the model classifies by the average similarity between the video and all synonyms of a candidate action. On ATA, IKEA, and GTEA, this recipe improves unseen-action accuracy and synonym robustness over fixed-label fine-tuning baselines while keeping seen-action performance competitive.","pith_inferences":["Beyond the paper: because ACE operates purely on the text side of the model, the same stochastic synonym recipe could be applied to other action taxonomies, such as medical procedures or sports, whenever an LLM can produce in-domain synonyms; the paper does not test these settings.","Beyond the paper: Table 4 suggests human-curated synonym trees outperform LLM-only trees, so a hybrid pipeline with light human review of generated synonyms may yield further gains than increasing synonym count.","Beyond the paper: the shadow-negative design points to a general cure for object bias in video-language fine-tuning; analogous object-matched wrong-verb negatives could be constructed for any verb-heavy classification task outside procedural videos."],"forward_implications":["ACE improves zero-shot classification of unseen procedural actions without changing the model architecture or pretraining data, by changing only the fine-tuning loss.","The Synonym Robustness Test provides a reproducible protocol for measuring whether a VLM is invariant to unseen action synonyms, not just to unseen labels.","Both the fixed root-label loss and the randomized synonym loss are necessary: removing $L_{rand}$ or leaf augmentation degrades robustness, and removing shadow negatives reintroduces object bias.","The approach transfers across cooking (GTEA) and assembly (ATA, IKEA) domains, suggesting it is tied to the training recipe rather than to a single dataset.","Using more synonyms does not monotonically help; the quality of the generated synonyms and the number of fine-tuned encoder layers need to be balanced."],"supporting_citations":[{"why":"Supplies the base-to-novel fine-tuning setup and the main image-based baseline that ACE is compared against.","marker":"[50]"},{"why":"Provides the Howto100M-pretrained video encoder that ACE fine-tunes and the one-to-one ProcVLR competitor.","marker":"[66]"},{"why":"The LLM used to generate the synonym trees that define the action concepts in the paper.","marker":"[1]"},{"why":"Text-augmentation baseline that also uses LLM-generated attributes and is compared in the main tables.","marker":"[61]"},{"why":"Strong image-based baseline with high seen accuracy but lower unseen generalization, used for harmonic-mean comparison.","marker":"[60]"},{"why":"The instructional-video pretraining corpus for the video encoder shared by ACE and ProcVLR.","marker":"[37]"}],"fun_headline_variants":["Synonym shuffling lifts zero-shot action recognition by 16%","ACE swaps labels with synonyms to teach action concepts","Random synonym flips boost unseen action accuracy in VLMs","Fine-tune with LLM synonyms for 16% gain on unseen actions","Procedural video actions: ACE improves zero-shot with synonym trees"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the LLM-generated synonyms are semantically valid and contextually appropriate for the procedural domain; if they are poor or off-distribution, the stochastic training signal can mislead the encoders and the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Synonym shuffling lifts zero-shot action recognition by 16%","ACE swaps labels with synonyms to teach action concepts","Random synonym flips boost unseen action accuracy in VLMs","Fine-tune with LLM synonyms for 16% gain on unseen actions","Procedural video actions: ACE improves zero-shot with synonym trees"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1553,"prompt_tokens":889,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":577}},"tokens_in":505,"tokens_out":664,"duration_ms":6631,"temperature":1.0,"reasoning_tokens":577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:05:08.978657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ACE with the same training pipeline but replace each LLM synonym with a randomly drawn verb from a general dictionary (keeping the object fixed). If zero-shot robustness on the Synonym Robustness Test stays at the same high level, the gains are due to label noise rather than to the semantic relation encoded in the synonyms; if it collapses, the concept-semantics explanation is supported.","supporting_citations":[{"cited_title":"Fine-tuned clip models are efficient video learners","cited_arxiv_id":null,"evidence_quote":"Supplies the base-to-novel fine-tuning setup and the main image-based baseline that ACE is compared against."},{"cited_title":"Learning procedure-aware video represen- tation from instructional videos and their narrations","cited_arxiv_id":null,"evidence_quote":"Provides the Howto100M-pretrained video encoder that ACE fine-tunes and the one-to-one ProcVLR competitor."},{"cited_title":"Bidirectional cross- modal knowledge exploration for video recognition with pre-trained vision-language models","cited_arxiv_id":null,"evidence_quote":"Text-augmentation baseline that also uses LLM-generated attributes and is compared in the main tables."},{"cited_title":"Revisiting clas- sifier: Transferring vision-language models for video recog- nition","cited_arxiv_id":null,"evidence_quote":"Strong image-based baseline with high seen accuracy but lower unseen generalization, used for harmonic-mean comparison."},{"cited_title":"Howto100m: Learning a text-video embedding by watching hundred million narrated video clips, 2019","cited_arxiv_id":null,"evidence_quote":"The instructional-video pretraining corpus for the video encoder shared by ACE and ProcVLR."}],"review_version":1}