{"id":"ecff6245-c85b-4f23-bf76-c89a09ae7cfd","arxiv_id":"2411.17406","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Chain-of-Action, a five-step prompting pipeline, improves zero-shot generation of multiple open-vocabulary semantic labels for images by using the model's own captions, self-correction, appearance, and relationship steps.","lead":"A prompting method called Chain-of-Action asks a vision-language model to describe an image, check each object it mentions, describe appearances and relationships, and then output a final set of semantic labels. It reports large gains over prior baselines for open-ended, multi-label image tagging, which matters for search, annotation, and scene understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Macc as written in §4.1 rewards output length rather than accuracy, so the headline +11.56% advantage may be an artifact of CoA producing more labels; the central claim of improved label accuracy is not yet established.","rationale":"The reader's weakest assumption points at RAM and CLIP as black-box arbiters; I agree broadly and sharpen the concern to a specific internal inconsistency in the printed Macc formula, which makes the length-bias problem concrete and testable. This is the most load-bearing issue because it directly controls the abstract's central quantitative claim: if Macc does not penalize incorrect labels, then CoA's multi-action pipeline is predisposed to score higher regardless of whether its extra labels are correct. The ablations and cross-VLM generalization experiments in §4.4–4.5 demonstrate that the action chain changes outputs, but they inherit the same metric, so they cannot independently validate accuracy. I do not see fraud or misconduct; the fix is a re-evaluation, which is exactly what a conditional decision should require. Because the reader already assigned CONDITIONAL, my recommendation is UNCHANGED: the verdict stays conditional pending the corrected metric check. If the corrected Macc recomputation collapses the advantage, the appropriate outcome would move toward REJECT for the current version of the claims; if the advantage persists, the concern does not land and the paper's quantitative evidence would be substantially strengthened.","tokens_in":16750,"tokens_out":7451,"duration_ms":76805,"concrete_test":"Using the released code, recompute the VOC results in Table 1 with a corrected Macc that follows the prose in §4.1: add 1 for each predicted label above RAM's confidence threshold and subtract 1 for each predicted label below it (or, better, compute standard per-image precision/recall/F1 against the dataset object labels). If CoA's +11.56% Macc margin over the best caption baseline drops below about 3 points or reverses, the headline improvement is an output-length artifact rather than evidence of label accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest quantitative claim (CoA beats the second-best method by +11.56% in Macc and +6.01% in Mcom on VOC) is supported only by the two metrics defined in §4.1. The Macc definition is internally inconsistent: the prose states that one point is added for each correct prediction and one point subtracted for each incorrect prediction, but the displayed formula is AS_i = Σ_j Sigmoid(RAM(Y_j^i)), a sum of positive sigmoid scores over all predicted labels with no penalty term. As written, Macc increases whenever more labels are emitted, even if RAM gives those labels near-zero confidence, because every sigmoid is positive. CoA is exactly the configuration that emits more labels (caption, self-correct, appearance, relationship, final actions), so part or all of the reported gap could be a length effect rather than a quality effect. Mcom has a related weakness: it awards a point whenever a CLIP prompt built from the predicted list is more similar to the image than a prompt built from the dataset's own labels, which the paper itself describes as 'relatively limited', so under-annotated references make longer lists systematically favored. Neither metric has been validated against human judgments, so the central claim that CoA guides VLMs to generate 'accurate and comprehensive semantic labels' is not established by the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Generative Semantic Labels (GSLs), a task requiring a vision-language model to produce an open-ended set of composite semantic labels for an image, and proposes Chain-of-Action (CoA), a zero-shot five-step prompting pipeline (caption, self-correction, appearance, relationship, final integration) that progressively enriches context. The authors evaluate CoA on VOC, COCO, NUS-WIDE, and a self-collected OVD dataset, using two proposed metrics: Mcom (CLIP-based comparison of predicted vs. human prompt similarity) and Macc (RAM-confidence sum over predicted labels). They report consistent gains over VQA and captioning baselines across several VLMs, with headline improvements of +11.56% in Macc and +6.01% in Mcom on VOC, and an ablation table that attributes the gains to each action.","tokens_in":16946,"tokens_out":11064,"duration_ms":83966,"significance":"If the evaluation were reliable, the paper would make a useful contribution: GSLs is a sensible task formalization, the prompting strategy is simple and model-agnostic, the code is released, and improvements are reported across multiple base models and datasets. The idea of chaining caption, verification, appearance, and relationship actions is a reasonable way to elicit more structured outputs from VLMs. However, the two proposed metrics are not validated as measures of label correctness or comprehensiveness, and the numeric claims contain inconsistencies with the tables, so the central contribution is currently not established.","major_comments":[{"comment":"The Macc metric is internally inconsistent: the prose states that 'one point is added for each correct prediction, and one point is subtracted for each incorrect prediction,' but the displayed formula AS_i = Σ_j Sigmoid(RAM(Y_j^i)) is a sum of positive sigmoid scores with no penalty term. As written, every emitted label adds a positive quantity, so Macc increases with the number of predicted labels regardless of their correctness; CoA is exactly the configuration that emits the most labels. The headline +11.56% Macc advantage (Section 4.2) may therefore reflect a length effect, and the central claim of improved label accuracy is not established. Please provide a corrected formula that implements the stated scoring rule, or validate the sigmoid sum against human judgments.","section":"4.1"},{"comment":"The reported headline improvements are not consistent with the tables. For VOC, the text claims '+11.56% in Macc and +6.01% in Mcom' over the second-best method, but Table 1(a) shows the best baseline Macc is 75.34 (Caption LLaVA) and the best baseline Mcom is 70.89 (Caption InstructBLIP); the corresponding differences are 2.15 and 11.56 respectively. For COCO, the text reports '+10.77% in Macc and +6.21% in Mcom', but Table 1(b) yields +10.77% in Mcom and +6.21% in Macc. For NUS, the text reports '+13.4% in Macc and +4.45% in Mcom', while Table 2 yields +12.05% in Mcom and +4.45% in Macc. Please re-check the calculations and state precisely which baseline is being used.","section":"4.2"},{"comment":"The ablation table does not support the claim that each added action contributes improvements. In Table 3, row (iv) (Actions 1+2+3+5) decreases Vehicles Mcom from 80.14 in row (iii) to 79.63, and row (v) (full CoA) decreases Vehicles Macc from 75.84 in row (iv) to 75.18; these negative changes are not reported in the parenthetical deltas. The statement in Section 4.4 that 'each added component contributes to significant improvements' is therefore not supported by the displayed numbers. Please report all deltas and provide a statistical or consistency analysis.","section":"4.4 (Table 3)"},{"comment":"The results for the real-world open-vocabulary dataset appear to be identical to the NUS-WIDE Nature subset in Table 2 (e.g., VQA BLIP-2 is 70.76/60.54, VQA InstructBLIP is 64.40/62.70, Caption LLaVA is 74.76/61.71). Moreover, the claimed gains do not match the table: from Table 5, the best baseline Mcom is 77.84 (Caption MiniGPT-4), giving a CoA advantage of 5.46 rather than 6.7%, and the best baseline Macc is 74.76 (Caption LLaVA), giving an advantage of 8.54 rather than 7.73%. Please clarify the provenance of the OVD numbers and correct the claims.","section":"4.3 (Table 5)"}],"minor_comments":[{"comment":"In the formula for AS_i, 'Sigmod' should be 'Sigmoid'.","section":"4.1"},{"comment":"The text says Mcom 'assess[es] the coverage of predicted labels,' but the metric is a relative comparison of CLIP similarities between the predicted prompt and the manual-annotation prompt; this should be described as a relative preference, not an absolute coverage score.","section":"4.1"},{"comment":"The term 'second-best method' is ambiguous; specify whether it means the best baseline (excluding CoA) or the second-best of all methods, and use that definition consistently throughout.","section":"4.2"},{"comment":"The header of Table 3 contains a typo: 'datatset' should be 'dataset'.","section":"4.4"},{"comment":"The 'filtering strategy' that extracts the initial object list from the caption is not described; provide the extraction algorithm or examples for reproducibility.","section":"3.3"},{"comment":"References [49] and [50] are the same paper (MiniGPT-4) and should be merged.","section":"References"},{"comment":"The limitation statement discusses coarse- and fine-grained labels but does not mention the reliance on RAM and CLIP as proxy ground truth; add a discussion of the limitations of the proposed evaluation metrics.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The metric inconsistency in Section 4.1 and the mismatches between the text and the tables in Sections 4.2 and 4.3 are serious and need to be resolved before the paper can be considered for acceptance. The apparent duplication of NUS-WIDE Nature values in the OVD table (Table 5) is particularly concerning and should be verified by the authors. The paper's central claim of improved label accuracy and comprehensiveness is not yet supported by the evidence as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper to know about: it defines Generative Semantic Labels (GSLs), an open-ended multi-label tagging task, and proposes Chain-of-Action (CoA), a zero-shot five-step prompting recipe (caption, self-correct, appearance, relationship, final). The method is sensible, and the ablation in Table 3 suggests each step contributes something. That part is fine.\n\nWhat is genuinely new: GSLs formalizes a real task—tagging images without a closed label set—and CoA is a clean way to approach it by leveraging the VLM's own generated context. The paper also shows the approach transfers across several base VLMs, and it ships code and prompt templates. The observation that caption-based baselines beat VQA-based ones is useful.\n\nThe problem is the evaluation. The Macc metric as written in Section 4.1 is AS_i = sum_j Sigmoid(RAM(Y_j^i)), a sum of positive scores. The prose says one point is added for correct and subtracted for incorrect predictions, but the displayed formula has no penalty term. As a result, emitting more labels always increases Macc, even if those labels receive near-zero confidence from RAM. CoA is exactly the configuration that emits the most labels, so the headline +11.56% Macc gain may be substantially a length effect, not evidence of accuracy. Mcom has a related issue: it awards a point when a CLIP prompt built from the predicted list beats a prompt built from the dataset's own annotations, which the paper itself calls 'relatively limited'—under-annotated references make longer lists systematically favored. Neither metric is validated against human judgments.\n\nThere are smaller soft spots: the final action outputs only object names, while the task definition promises scenes, attributes, and relationships; the caption prompt is selected on an undescribed auxiliary dataset; there are no error bars; and standard multi-label metrics like mAP or F1 against ground-truth labels are absent. These are all addressable.\n\nMy take: the method is not circular, and the thinking is clear. But the central claim—that CoA generates more accurate and comprehensive labels—is not yet established by the reported numbers. The paper deserves serious review because the task and method are worth discussing, but it needs major revision: human evaluation, standard metrics, error bars, and full disclosure of prompt selection and filtering.\n\nWho should read it: people working on open-vocabulary tagging, VLM prompting, and zero-shot multi-label recognition. I would not cite it in its current form, but I'd bring it to a reading group to discuss the metric-design trap.\n\nRecommendation: send it to review, with the clear expectation that the authors redo the evaluation.","headline":"CoA is a plausible prompting pipeline for a useful new task, but the two proposed metrics both reward label quantity over label quality, so the headline accuracy gains are not yet established.","tokens_in":17541,"tokens_out":3544,"would_cite":false,"duration_ms":34005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Chain-of-Action (CoA), a five-step zero-shot prompting pipeline, makes vision-language models generate broader and more accurate semantic labels without any predefined label set.","keywords":["Generative Semantic Labels","Chain-of-Action","Vision-Language Model","Zero-shot prompting","Open-vocabulary classification","Multi-label image recognition","Vocabulary-unknown label spaces","Composite semantic labels"],"falsifier":"Take a random sample of images from VOC and COCO, have independent human annotators judge each predicted label produced by CoA and by the best caption baseline as present or absent in the image, and compare precision and recall. If CoA's margin over the baseline disappears or reverses under human scoring, the reported Macc/Mcom advantages are artifacts of trusting RAM and CLIP as the arbiters of correctness and coverage.","tokens_in":78,"feed_emoji":"🏷️","tokens_out":5157,"duration_ms":92308,"temperature":0.7,"pith_summary":"The paper defines a new task, Generative Semantic Labels (GSLs): given an image, produce a comprehensive set of semantic labels—objects, scenes, attributes, relationships—without any predefined label vocabulary. It then proposes Chain-of-Action (CoA), a zero-shot prompting pipeline that breaks label generation into five sequential steps: caption the image, self-correct the entity list, describe appearances, infer relationships, and synthesize final labels. The paper claims CoA consistently outperforms VQA-based and caption-based baselines built on BLIP-2, InstructBLIP, LLaVA, and MiniGPT-4 across VOC, COCO, NUS, and a collected open-vocabulary dataset. If true, the result matters because it offers a training-free way to make vision-language models usable in open-ended domains where label sets are unknown and often composite. The paper also proposes two evaluation metrics, semantic accuracy and semantic comprehensiveness, as a reusable protocol for this task.","feed_headline":"Five-step prompt chain lifts VLM multi-label accuracy 11.6%","feed_subtitle":"A zero-shot five-step CoA pipeline generates broader, more accurate semantic labels without predefined label sets.","key_machinery":"The load-bearing mechanism is the Chain-of-Action (CoA) prompt sequence, a five-step zero-shot protocol: (1) Caption Action generates a one-sentence image summary and an initial entity list; (2) Self-Correct Action filters that list with targeted yes/no questions; (3) Appearance Action extracts per-entity attributes; (4) Relationship Action infers spatial and interaction links; (5) Final Action integrates the accumulated context into the output labels. Each action hands its enriched context to the next, so the VLM is never asked to infer everything from a single prompt. The paper's ablations treat the five-action chain as the unit under test and attribute the performance gain to the progressive accumulation of context.","core_discovery":"The central discovery, stated on the paper's own terms, is that a vision-language model can be guided to 'see the undefined' by decomposing label generation into a chain of progressively enriched actions rather than asking for labels in one shot. CoA first obtains a broad one-sentence caption and extracts an initial object list, then verifies each entity with a yes/no self-correction query, then gathers appearance details and inter-entity relationships, and finally merges all of it into the label set. The paper reports that this full chain exceeds the second-best method by +11.56% in semantic accuracy and +6.01% in semantic comprehensiveness on VOC, with similar gains on COCO, NUS, and a real-world open-vocabulary dataset, and that the gains hold when the base model is changed to LLaVA-v1.6-7B. The paper also shows that multiple interactions beat a single merged interaction, and that caption-first strategies beat direct VQA queries.","pith_inferences":["Inference: The yes/no self-correction step probably does heavy lifting against caption hallucination; the paper does not isolate it, but a per-action hallucination audit would settle it.","Inference: The Mcom metric may reward longer label lists if CLIP similarity rises with prompt length, so a length-controlled variant (same number of labels per comparison) would be a sturdier evaluation.","Inference: CoA's context chaining could transfer to video captioning or embodied agents, where previous-frame descriptions play the role of the caption step; the paper does not test those settings."],"forward_implications":["Zero-shot prompting alone can substantially improve multi-label generation across multiple VLM families, making the approach usable without training or external label databases.","Decomposing an open-ended generation task into caption, verification, attribute, and relationship sub-questions is a transferable recipe for other vision-language tasks.","Caption-based grounding outperforms direct VQA queries for label generation, suggesting broad scene description should precede fine-grained interrogation.","The proposed Macc and Mcom metrics give future GSLs work a shared evaluation protocol, even though the protocol relies on automatic scorers."],"supporting_citations":[{"why":"LLaVA-1.5-7B is the base model on which CoA is implemented and ablated.","marker":"[22]"},{"why":"BLIP-2 serves as a VQA and captioning baseline that CoA must beat.","marker":"[17]"},{"why":"InstructBLIP serves as a VQA and captioning baseline and an alternative VLM.","marker":"[8]"},{"why":"CLIP provides the text and image encoders used to compute the semantic comprehensiveness metric Mcom.","marker":"[26]"},{"why":"RAM provides the confidence scores and thresholding used to compute the semantic accuracy metric Macc.","marker":"[43]"},{"why":"PASCAL VOC 2012 supplies the primary benchmark and the human-annotation label set for Mcom.","marker":"[11]"},{"why":"MS-COCO 2014 supplies a second multi-label benchmark used in the main comparison.","marker":"[19]"},{"why":"LLaVA-v1.6-7B is used to test CoA's generalizability across different vision-language model versions.","marker":"[21]"}],"fun_headline_variants":["Chain-of-Action prompts boost VLM semantic labels 11.6%","Multi-step action chain lifts VLM open-label accuracy","+11.6% semantic accuracy via chain-of-action for VLMs","Action-chained prompting enriches VLM label generation"],"cache_read_input_tokens":19584,"weakest_assumption_plain":"The central claim rests on trusting RAM's automatic tagger and CLIP's text-image similarity as faithful proxies for human judgments of label correctness and completeness.","fun_headline_variants_meta":{"raw":{"variants":["Chain-of-Action prompts boost VLM semantic labels 11.6%","Multi-step action chain lifts VLM open-label accuracy","+11.6% semantic accuracy via chain-of-action for VLMs","Action-chained prompting enriches VLM label generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1751,"prompt_tokens":1009,"completion_tokens":742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":671}},"tokens_in":625,"tokens_out":742,"duration_ms":7037,"temperature":1.0,"reasoning_tokens":671,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:11:54.885984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of images from VOC and COCO, have independent human annotators judge each predicted label produced by CoA and by the best caption baseline as present or absent in the image, and compare precision and recall. If CoA's margin over the baseline disappears or reverses under human scoring, the reported Macc/Mcom advantages are artifacts of trusting RAM and CLIP as the arbiters of correctness and coverage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA-1.5-7B is the base model on which CoA is implemented and ablated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLIP-2 serves as a VQA and captioning baseline that CoA must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"InstructBLIP serves as a VQA and captioning baseline and an alternative VLM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP provides the text and image encoders used to compute the semantic comprehensiveness metric Mcom."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PASCAL VOC 2012 supplies the primary benchmark and the human-annotation label set for Mcom."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MS-COCO 2014 supplies a second multi-label benchmark used in the main comparison."}],"review_version":1}