{"id":"25347c77-b9a4-4a44-931e-120a0cfccd65","arxiv_id":"2412.11025","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"CapAgent converts simple captioning requests into professional multi-constraint instructions and uses a tool-using agent to generate captions, but the report provides no experiments.","lead":"CapAgent is a proposed image captioning system that takes simple user instructions, expands them into detailed professional instructions using GPT-4o plus web search, and then uses an agent with tools such as object detection and depth estimation to generate captions that respect those instructions. The paper is a technical report with only hand-picked examples and no quantitative evaluation, so its effectiveness claims are not yet supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No constraint-verification step or evaluation supports the claim that CapAgent 'ensures' combinatorial caption control; sequential tool edits can break earlier constraints and the stopping criterion is the planner's self-assessment.","rationale":"The paper is a clearly written technical report describing a plausible agent pipeline, and the provided code link is a real asset. The strongest claim is that CapAgent automatically turns simple instructions into professional instructions and ensures captions adhere to combinatorial constraints. For this claim to hold, the system must (i) evolve instructions without introducing false context, (ii) plan correct tool calls, and (iii) produce final captions satisfying all constraints jointly. The weakest link is (iii): the only check is the LLM planner's own decision to stop, and there is no constraint-satisfaction module. This is not merely a missing experiment — the architecture as described cannot certify compliance because tools operate independently and sequentially, with no global verification. The paper provides no baselines, no metrics, and no ablations; the visualizations are selected successes. A single benchmark with automatic constraint checks would settle whether the control loop works. If it does, the report could be revised as a system demonstration; as it stands, the central claim is unsupported. I agree with the reader's rejection, though my concern is more specifically about the absent verification/enforcement mechanism than about retrieval precision alone.","tokens_in":7917,"tokens_out":4282,"duration_ms":38141,"concrete_test":"Create a benchmark of 100 images and user instructions that each specify 2–4 constraints from Table 1, e.g., include keywords K, length in [Lmin, Lmax], sentiment S, and format F. Run CapAgent end-to-end and a strong baseline (GPT-4o prompted with the same professional instruction) on identical inputs. Compute per-constraint satisfaction automatically: keyword substring match, length interval membership, format parse (paragraph vs. bullets), and sentiment agreement with a pretrained classifier or human annotations. Record the joint satisfaction rate (all constraints met) and an interaction matrix showing how often one tool's edit breaks another constraint. If CapAgent's joint satisfaction is not substantially above baseline, or if sequential tool edits show a high breakage rate, the central claim of ensured combinatorial control is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract states that CapAgent 'ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting.' The architecture in §2.3 provides no mechanism that can establish this. The ReAct loop (§2.3.1) plans tool calls and stops when the planner believes 'all user requirements are met,' but there is no objective constraint checker in the loop and no final verification step. Each tool in §2.3.3 edits one dimension in isolation — sentiment modification, expansion, condensation, keyword inclusion — yet the tools are applied sequentially, so later edits can silently break earlier constraints. For example, condensing to meet a length limit can drop a required keyword or flatten the target sentiment, while expanding for length may introduce off-focus content. Because the stopping criterion is the same MLLM's self-assessment, the loop can terminate with unmet constraints, and no measurement of constraint satisfaction is reported. Additionally, retrieval-augmented planning (§2.3.2) selects top-N chains by cosine similarity of the user instruction only, not the image or the specific constraint combination, and neither retrieval precision nor planning accuracy is evaluated. The 'ensures' claim therefore rests on six illustrative examples in Figures 4–5 rather than a validated control loop.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CapAgent, an agent-based image captioning system with two main components: an instruction-evolution module that uses GPT-4o plus web context from Google Lens and Google Search to convert simple user instructions into detailed professional instructions, and a ReAct-style agent loop (Section 2.3.1) that plans actions, executes Python code, and observes results to control caption generation. The agent uses retrieval-augmented planning (Section 2.3.2) with BGE-M3 embeddings and a suite of tools including VQA, sentiment modification, caption expansion/condensation, object counting, and spatial relation (Section 2.3.3). The paper claims that CapAgent 'ensures' captions adhere to user-specified constraints (sentiment, keywords, focus, formatting) and demonstrates the system with six visualization examples (Figures 4-5). No quantitative evaluation is provided.","tokens_in":8156,"tokens_out":4904,"duration_ms":43874,"significance":"If the central claims were validated, CapAgent would be a useful contribution: it offers a transparent, tool-based pipeline for controllable image captioning, lowers the burden of writing complex prompts for non-expert users, and ships open-source code. The use of retrieval-augmented planning and a modular tool suite is a plausible design, and the paper contains no fitted parameters or circular derivation. However, the paper's significance is entirely contingent on empirical evidence, which is absent. The system's core promise is 'ensuring' constraint satisfaction, yet no measurements of constraint compliance, retrieval quality, or planning accuracy are reported. The six hand-picked visualizations are illustrative but cannot support the paper's definitive claims. As a technical report the paper may be of interest to practitioners, but as a research contribution it falls short of the standard needed to establish the claimed guarantees.","major_comments":[{"comment":"The abstract states that CapAgent 'ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting.' The architecture described in Section 2.3 provides no mechanism that can substantiate this claim. The ReAct loop stops when the planner believes all user requirements are met, but there is no objective constraint checker or final verification step. The tools in Section 2.3.3 edit constraints sequentially, so later edits can silently break earlier ones: for instance, the caption condensation tool could remove a required keyword or flatten the target sentiment while shortening the text, and the expansion tool could introduce off-focus content. The stopping criterion is the same MLLM's self-assessment, so the loop can terminate with unmet constraints. This is a load-bearing issue because 'ensures' is the central claim of the paper.","section":"Abstract; Section 2.3.1; Section 2.3.3"},{"comment":"The only empirical evidence consists of six hand-picked visualization examples in Figures 4 and 5. The paper reports no measurements of constraint satisfaction (e.g., keyword inclusion, sentiment match, length compliance, format adherence), no comparisons to baselines such as direct GPT-4o prompting with the professional instruction, no ablation of the retrieval or tool components, and no user study. The conclusion that the system 'delivers contextually accurate, sentiment-aligned, and detail-rich descriptions' is therefore unsupported. For a paper whose central contribution is controllable caption generation, this absence of evaluation is a load-bearing gap.","section":"Section 3"},{"comment":"Retrieval-augmented planning selects the top N thought-chain examples by cosine similarity of the user instruction embedding only (Eq. 1), with no use of the image content or the specific combination of constraints. The paper provides no evaluation of retrieval precision, no analysis of how the retrieved chains affect planning quality, and no discussion of failure cases. Since the entire agent control loop depends on the retrieved examples to produce correct actions, this unvalidated dependency is a load-bearing weakness rather than a minor detail.","section":"Section 2.3.2"},{"comment":"The instruction-evolution step relies on external context from Google Lens and Google Search to identify entities such as 'Tesla Cybercab,' 'Venom 3,' and 'King Charles III.' The paper does not verify the accuracy of this web-derived context or measure the effect of incorrect or irrelevant context on the evolved instruction and final caption. Given that the examples in Figures 4-5 are explicitly tied to such externally sourced facts, the risk of hallucination or context drift is real and unaddressed. This further undermines the 'context-aware' and 'ensures' claims.","section":"Section 2.2"}],"minor_comments":[{"comment":"The paper is labeled 'A technical report' on page 2, but the abstract and conclusion use definitive language such as 'ensures' and 'delivers.' The claims should be aligned with the level of evidence provided, or the evidence should be added.","section":"General"},{"comment":"The row for 'Format' reads 'Some images have strong emotions, and it is a better choice to describe the image with the corresponding emotional tone,' which appears to describe Sentiment rather than Format. This is likely a copy-and-paste error and should be corrected.","section":"Table 1"},{"comment":"The text in the visualization figures is very small and dense, making it difficult to read the user instructions, professional instructions, and captions. Consider presenting these examples in a larger format or in a table.","section":"Figures 4-5"},{"comment":"The correctness criterion in Section 2.2 cites [14] (MIA-Bench), but the connection is not explained in the text. Please clarify how MIA-Bench informs the 'describable by humans' criterion.","section":"References"},{"comment":"The caption expansion and condensation tools are described without specifying the prompts or the mechanism that preserves other constraints such as keywords or sentiment. Please provide more detail on how these tools interact with the full set of user requirements.","section":"Section 2.3.3"}],"recommendation":"reject","confidential_remarks":"This is a system/technical report with no experimental evaluation. The central 'ensures' claim is not only unverified but is contradicted by the architecture, which lacks any objective constraint-checking mechanism. The paper would need a substantial redesign of the loop (e.g., adding a verifier) and a comprehensive evaluation to become a viable journal submission. The authors may be encouraged to pursue that direction, but the current manuscript is not ready for publication in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clearly written technical report describing CapAgent, an agent pipeline that turns simple captioning instructions into more elaborate ones and then uses a ReAct loop with specialized tools to satisfy them. What's actually new is the composition—instruction evolving with web context via Google Lens/Search, retrieval-augmented planning over thought-action-observation chains with BGE-M3, and a set of caption-editing tools on top of GPT-4o, GroundingDINO, and DepthAnything v2. Each component is known, and the paper cites the relevant prior work; the contribution is the specific integration for controllable captioning. The writing is straightforward and the system design is plausible.\n\nThe soft spot is not subtle: there is no experimental evaluation, no baseline, and no quantitative constraint-satisfaction metric. The abstract says CapAgent 'ensures' captions adhere to specified guidelines, and the conclusion repeats that. The only evidence is six hand-picked visualizations. The stress-test note is right that the architecture has no objective constraint checker: tools edit one dimension at a time sequentially, and a later edit (e.g., condensation) can break an earlier constraint (e.g., keyword inclusion). The stopping criterion is the planner's own self-assessment, so 'all requirements met' is unverified. Retrieval quality and planning accuracy are also unmeasured. These aren't fatal to the idea, but they make the central claim unsupported.\n\nI agree with the reader's reject verdict and high confidence. The paper is honest enough to call itself a technical report, and the code link may let someone evaluate it later. But as a submission, it's a system demo without measurements. For a serious venue, this would need at least a constraint-satisfaction evaluation against baselines (e.g., direct GPT-4o prompting, other controllable captioning methods) and ideally human ratings. If the authors add that, the composed system could be a useful contribution for accessible captioning tools.\n\nWho is this for? Readers building agent-based controllable generation systems, particularly for captioning. It does not deserve to be published in its current form. My recommendation: desk reject or send back with invitation to add evaluation; do not spend referee time on the current manuscript unless the venue explicitly publishes technical reports without experiments.","headline":"Plausible agent architecture for controllable captioning, but the central 'ensures' claim rests on six cherry-picked examples; without any evaluation this is a technical report, not a validated system.","tokens_in":8679,"tokens_out":2051,"would_cite":false,"duration_ms":17623,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CapAgent turns simple prompts into controlled professional captions.","keywords":["image captioning","controllable captioning","instruction evolving","multimodal large language models","retrieval-augmented planning","ReAct agents","external tools","combinatorial constraints"],"falsifier":"Run a controlled evaluation on a set of images with human-verified constraint labels: count how many of CapAgent's final captions contain every required keyword, stay within the required length, and give correct object counts. If the compliance rate is low, or if replacing the retrieved examples with random ones does not lower compliance, then the central claim of controllable captioning is falsified.","tokens_in":7739,"feed_emoji":"🖼️","tokens_out":7134,"duration_ms":55796,"temperature":0.7,"pith_summary":"CapAgent is a system designed to let ordinary users give a short, vague caption instruction like 'describe this cab' and still receive a caption that follows professional-grade constraints on sentiment, keywords, focus, length, format, and genre. It works in two stages: first, a multimodal language model evolves the user's simple instruction into a detailed, context-aware professional instruction, enriching it with external context gathered from web image and text search; second, an agent equipped with specialized tools plans and executes steps to follow that instruction, checking objects, spatial relations, counts, sentiment, and length. The paper's claim is that this decomposition makes image captioning truly controllable and transparent, because every reasoning step and tool call is visible. If the claim holds, non-expert users gain the ability to produce customized, high-quality image descriptions without learning prompt engineering.","feed_headline":"CapAgent turns simple prompts into controlled professional captions","feed_subtitle":"Two-stage system evolves your request with web context, then enforces each constraint with vision tools.","key_machinery":"The central object is the CapAgent agent loop itself, a ReAct-style planner-executor cycle that combines retrieval-augmented planning with a suite of caption-control tools. The planner is GPT-4o, which at each step emits a thought and a Python code action; the code is executed on a local Jupyter server, and the observation is fed back to generate the next thought and action. Retrieval-augmented planning uses BGE-M3 embeddings to select the top N most similar thought-action-observation chains from a curated database, based on cosine similarity with the user instruction, and places them in the system prompt to guide the planner. The tools—question answering, sentiment rewriting, expansion, condensation, object counting, and spatial relation description (built on GroundingDINO and DepthAnything v2)—provide concrete levers for enforcing the combinatorial constraints from the evolved professional instruction. The instruction evolving module, also built on GPT-4o, supplies that professional instruction by combining the user's request, the image, and external web context.","core_discovery":"The central claim is that the gap between simple user instructions and professional image captions can be closed by decomposing caption generation into instruction evolution plus tool-guided agent execution. The instruction evolution module, built on GPT-4o, takes the user's instruction and the image, pulls in additional context from Google Lens image search and Google Search text search, and outputs an evolved instruction that specifies several constraint dimensions at once—sentiment, keywords, focus content, length, format, genre, and viewpoint. The agent then follows that instruction through a ReAct-style thought-action-observation loop: it generates a thought and a Python code action, executes the code, observes the result, and repeats until the caption satisfies the constraints. To keep the planning precise, the agent retrieves the most similar worked examples from a database using cosine similarity in BGE-M3 embedding space and injects them into the prompt. The tools available to the agent include a visual question answering tool, a sentiment modification tool, caption expansion and condensation tools, an object counting tool, and a spatial relation tool built on object detection and depth estimation, allowing it to verify and adjust each aspect of the caption. The paper presents qualitative demonstrations of this process across historical events, movie posters, product photos, cat images, living rooms, and photography, showing that the evolved professional instructions and final captions match the user's intent.","pith_inferences":["The practical value of CapAgent will depend heavily on the reliability of its external dependencies: if Google Lens/Search return irrelevant context or the retrieval step selects poor examples, the quality of the evolved instruction and the agent's plan will degrade, so the method needs a quantitative sensitivity analysis.","A direct extension is to measure constraint satisfaction as a metric—checking keyword presence, length bounds, object counts, and sentiment alignment—on a benchmark dataset; this would turn the qualitative demonstrations into a testable claim.","The same instruction-evolving plus tool-using-agent architecture could be transferred to other multimodal generation tasks such as controllable video description or document summarization, where combinatorial constraints also matter.","The instruction evolving step and the agent execution step can be evaluated independently to see which one contributes more to final quality; the paper's design implicitly assumes both are necessary."],"forward_implications":["Users can combine multiple types of constraints—format, semantic, lexical, and utility—in a single request and have an agent enforce them simultaneously.","The instruction evolution step can identify entities and context that are not obvious from the image alone by consulting web search, such as recognizing a 'Tesla Cybercab' from a photo of a car.","Running the captioning process as a visible sequence of thoughts, actions, and observations gives users a way to check why a caption was produced, supporting trust in the output.","The two-stage design separates the question of what to say (professional instruction) from the question of how to say it (tool-guided execution), which is the core mechanism for control.","For users, the system removes the need to write long, carefully worded prompts; a one-line request can yield a caption that obeys strict constraints."],"supporting_citations":[{"why":"Supplies the ReAct thought-action-observation prompting pattern that drives the agent loop.","marker":"[20]"},{"why":"Provides the BGE-M3 text embedding model used to vectorize and retrieve similar thought-chain examples.","marker":"[2]"},{"why":"Establishes the retrieval-augmented generation paradigm that motivates selecting examples by similarity.","marker":"[10]"},{"why":"Gives the retrieval-augmented planning template for using retrieved context to guide multimodal LLM agents.","marker":"[9]"},{"why":"Serves as both the instruction evolving module and the planner model that generates thoughts and code actions.","marker":"[8]"},{"why":"Supplies the open-set object detection capability behind the object counting and spatial relation tools.","marker":"[11]"},{"why":"Provides depth estimation that enables the spatial relation tool to describe distances and layout.","marker":"[19]"},{"why":"Defines the constraint dimensions (format, semantic, lexical, utility) that the instruction evolving step targets.","marker":"[18]"},{"why":"Motivates the correctness criterion for evolved instructions by requiring that they be answerable and grounded in the image.","marker":"[14]"}],"fun_headline_variants":["CapAgent: simple prompts in, professional captions out","Agent evolves your caption request with web context and tools","From a few words to a full professional caption, automatically","CapAgent turns simple requests into controlled, tool-checked captions","Your simple caption idea becomes a professional one via CapAgent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"CapAgent assumes that the similar examples it retrieves and the web context it gathers are reliable guides for planning the right tool actions on a new image; if that retrieval or context is wrong, the whole captioning process goes wrong.","fun_headline_variants_meta":{"raw":{"variants":["CapAgent: simple prompts in, professional captions out","Agent evolves your caption request with web context and tools","From a few words to a full professional caption, automatically","CapAgent turns simple requests into controlled, tool-checked captions","Your simple caption idea becomes a professional one via CapAgent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1611,"prompt_tokens":936,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":552,"tokens_out":675,"duration_ms":5579,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:21:54.629034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled evaluation on a set of images with human-verified constraint labels: count how many of CapAgent's final captions contain every required keyword, stay within the required length, and give correct object counts. If the compliance rate is low, or if replacing the retrieved examples with random ones does not lower compliance, then the central claim of controllable captioning is falsified.","supporting_citations":[{"cited_title":"St Edward's Crown","cited_arxiv_id":null,"evidence_quote":"Establishes the retrieval-augmented generation paradigm that motivates selecting examples by similarity."},{"cited_title":"Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents, 2024","cited_arxiv_id":null,"evidence_quote":"Gives the retrieval-augmented planning template for using retrieved context to guide multimodal LLM agents."},{"cited_title":"Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Alan Hayes","cited_arxiv_id":null,"evidence_quote":"Serves as both the instruction evolving module and the planner model that generates thoughts and code actions."},{"cited_title":"Depth anything v2, 2024","cited_arxiv_id":null,"evidence_quote":"Provides depth estimation that enables the spatial relation tool to describe distances and layout."}],"review_version":1}