From Simple to Professional: A Combinatorial Controllable Image Captioning Agent

Baoteng Li; Haiwen Zhang; Kongming Liang; Muxi Diao; Xinran Wang; Zhanyu Ma

REVIEW 4 major objections 5 minor 20 references

From Simple to Professional: A Combinatorial Controllable Image Captioning Agent

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CapAgent turns simple prompts into controlled professional captions.

desk verdict Plausible agent architecture for controllable captioning, but the central 'ensures' claim rests on six cherry-picked examples; without any evaluation this is a technical report, not a validated system. read the letter →

arxiv 2412.11025 v1 pith:QR2MLIBL submitted 2024-12-15 cs.CV cs.AI

Xinran Wang , Muxi Diao , Baoteng Li , Haiwen Zhang , Kongming Liang , Zhanyu Ma This is my paper · ORCID

classification cs.CVcs.AI

keywords imagecaptioningcontrollableinstructionevolvingmultimodallargelanguagemodelsretrieval-augmentedplanningReActagentsexternaltoolscombinatorialconstraints

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CapAgent is a system designed to let ordinary users give a short, vague caption instruction like 'describe this cab' and still receive a caption that follows professional-grade constraints on sentiment, keywords, focus, length, format, and genre. It works in two stages: first, a multimodal language model evolves the user's simple instruction into a detailed, context-aware professional instruction, enriching it with external context gathered from web image and text search; second, an agent equipped with specialized tools plans and executes steps to follow that instruction, checking objects, spatial relations, counts, sentiment, and length. The paper's claim is that this decomposition makes image captioning truly controllable and transparent, because every reasoning step and tool call is visible. If the claim holds, non-expert users gain the ability to produce customized, high-quality image descriptions without learning prompt engineering.

What carries the argument

The central object is the CapAgent agent loop itself, a ReAct-style planner-executor cycle that combines retrieval-augmented planning with a suite of caption-control tools. The planner is GPT-4o, which at each step emits a thought and a Python code action; the code is executed on a local Jupyter server, and the observation is fed back to generate the next thought and action. Retrieval-augmented planning uses BGE-M3 embeddings to select the top N most similar thought-action-observation chains from a curated database, based on cosine similarity with the user instruction, and places them in the system prompt to guide the planner. The tools—question answering, sentiment rewriting, expansion, condensation, object counting, and spatial relation description (built on GroundingDINO and DepthAnything v2)—provide concrete levers for enforcing the combinatorial constraints from the evolved professional instruction. The instruction evolving module, also built on GPT-4o, supplies that professional instruction by combining the user's request, the image, and external web context.

What would settle it

Run a controlled evaluation on a set of images with human-verified constraint labels: count how many of CapAgent's final captions contain every required keyword, stay within the required length, and give correct object counts. If the compliance rate is low, or if replacing the retrieved examples with random ones does not lower compliance, then the central claim of controllable captioning is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the gap between simple user instructions and professional image captions can be closed by decomposing caption generation into instruction evolution plus tool-guided agent execution. The instruction evolution module, built on GPT-4o, takes the user's instruction and the image, pulls in additional context from Google Lens image search and Google Search text search, and outputs an evolved instruction that specifies several constraint dimensions at once—sentiment, keywords, focus content, length, format, genre, and viewpoint. The agent then follows that instruction through a ReAct-style thought-action-observation loop: it generates a thought and a Python code action, executes the code, observes the result, and repeats until the caption satisfies the constraints. To keep the planning precise, the agent retrieves the most similar worked examples from a database using cosine similarity in BGE-M3 embedding space and injects them into the prompt. The tools available to the agent include a visual question answering tool, a sentiment modification tool, caption expansion and condensation tools, an object counting tool, and a spatial relation tool built on object detection and depth estimation, allowing it to verify and adjust each aspect of the caption. The paper presents qualitative demonstrations of this process across historical events, movie posters, product photos, cat images, living rooms, and photography, showing that the evolved professional instructions and final captions match the user's intent.

Load-bearing premise

CapAgent assumes that the similar examples it retrieves and the web context it gathers are reliable guides for planning the right tool actions on a new image; if that retrieval or context is wrong, the whole captioning process goes wrong.

Editorial extensions

If this is right

Users can combine multiple types of constraints—format, semantic, lexical, and utility—in a single request and have an agent enforce them simultaneously.
The instruction evolution step can identify entities and context that are not obvious from the image alone by consulting web search, such as recognizing a 'Tesla Cybercab' from a photo of a car.
Running the captioning process as a visible sequence of thoughts, actions, and observations gives users a way to check why a caption was produced, supporting trust in the output.
The two-stage design separates the question of what to say (professional instruction) from the question of how to say it (tool-guided execution), which is the core mechanism for control.
For users, the system removes the need to write long, carefully worded prompts; a one-line request can yield a caption that obeys strict constraints.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The practical value of CapAgent will depend heavily on the reliability of its external dependencies: if Google Lens/Search return irrelevant context or the retrieval step selects poor examples, the quality of the evolved instruction and the agent's plan will degrade, so the method needs a quantitative sensitivity analysis.
A direct extension is to measure constraint satisfaction as a metric—checking keyword presence, length bounds, object counts, and sentiment alignment—on a benchmark dataset; this would turn the qualitative demonstrations into a testable claim.
The same instruction-evolving plus tool-using-agent architecture could be transferred to other multimodal generation tasks such as controllable video description or document summarization, where combinatorial constraints also matter.
The instruction evolving step and the agent execution step can be evaluated independently to see which one contributes more to final quality; the paper's design implicitly assumes both are necessary.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a clearly written technical report describing CapAgent, an agent pipeline that turns simple captioning instructions into more elaborate ones and then uses a ReAct loop with specialized tools to satisfy them. What's actually new is the composition—instruction evolving with web context via Google Lens/Search, retrieval-augmented planning over thought-action-observation chains with BGE-M3, and a set of caption-editing tools on top of GPT-4o, GroundingDINO, and DepthAnything v2. Each component is known, and the paper cites the relevant prior work; the contribution is the specific integration for controllable captioning. The writing is straightforward and the system design is plausible.

The soft spot is not subtle: there is no experimental evaluation, no baseline, and no quantitative constraint-satisfaction metric. The abstract says CapAgent 'ensures' captions adhere to specified guidelines, and the conclusion repeats that. The only evidence is six hand-picked visualizations. The stress-test note is right that the architecture has no objective constraint checker: tools edit one dimension at a time sequentially, and a later edit (e.g., condensation) can break an earlier constraint (e.g., keyword inclusion). The stopping criterion is the planner's own self-assessment, so 'all requirements met' is unverified. Retrieval quality and planning accuracy are also unmeasured. These aren't fatal to the idea, but they make the central claim unsupported.

I agree with the reader's reject verdict and high confidence. The paper is honest enough to call itself a technical report, and the code link may let someone evaluate it later. But as a submission, it's a system demo without measurements. For a serious venue, this would need at least a constraint-satisfaction evaluation against baselines (e.g., direct GPT-4o prompting, other controllable captioning methods) and ideally human ratings. If the authors add that, the composed system could be a useful contribution for accessible captioning tools.

Who is this for? Readers building agent-based controllable generation systems, particularly for captioning. It does not deserve to be published in its current form. My recommendation: desk reject or send back with invitation to add evaluation; do not spend referee time on the current manuscript unless the venue explicitly publishes technical reports without experiments.

Referee Report

4 major / 5 minor

Summary. The paper presents CapAgent, an agent-based image captioning system with two main components: an instruction-evolution module that uses GPT-4o plus web context from Google Lens and Google Search to convert simple user instructions into detailed professional instructions, and a ReAct-style agent loop (Section 2.3.1) that plans actions, executes Python code, and observes results to control caption generation. The agent uses retrieval-augmented planning (Section 2.3.2) with BGE-M3 embeddings and a suite of tools including VQA, sentiment modification, caption expansion/condensation, object counting, and spatial relation (Section 2.3.3). The paper claims that CapAgent 'ensures' captions adhere to user-specified constraints (sentiment, keywords, focus, formatting) and demonstrates the system with six visualization examples (Figures 4-5). No quantitative evaluation is provided.

Significance. If the central claims were validated, CapAgent would be a useful contribution: it offers a transparent, tool-based pipeline for controllable image captioning, lowers the burden of writing complex prompts for non-expert users, and ships open-source code. The use of retrieval-augmented planning and a modular tool suite is a plausible design, and the paper contains no fitted parameters or circular derivation. However, the paper's significance is entirely contingent on empirical evidence, which is absent. The system's core promise is 'ensuring' constraint satisfaction, yet no measurements of constraint compliance, retrieval quality, or planning accuracy are reported. The six hand-picked visualizations are illustrative but cannot support the paper's definitive claims. As a technical report the paper may be of interest to practitioners, but as a research contribution it falls short of the standard needed to establish the claimed guarantees.

major comments (4)

[Abstract; Section 2.3.1; Section 2.3.3] The abstract states that CapAgent 'ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting.' The architecture described in Section 2.3 provides no mechanism that can substantiate this claim. The ReAct loop stops when the planner believes all user requirements are met, but there is no objective constraint checker or final verification step. The tools in Section 2.3.3 edit constraints sequentially, so later edits can silently break earlier ones: for instance, the caption condensation tool could remove a required keyword or flatten the target sentiment while shortening the text, and the expansion tool could introduce off-focus content. The stopping criterion is the same MLLM's self-assessment, so the loop can terminate with unmet constraints. This is a load-bearing issue because 'ensures' is the central claim of the paper.
[Section 3] The only empirical evidence consists of six hand-picked visualization examples in Figures 4 and 5. The paper reports no measurements of constraint satisfaction (e.g., keyword inclusion, sentiment match, length compliance, format adherence), no comparisons to baselines such as direct GPT-4o prompting with the professional instruction, no ablation of the retrieval or tool components, and no user study. The conclusion that the system 'delivers contextually accurate, sentiment-aligned, and detail-rich descriptions' is therefore unsupported. For a paper whose central contribution is controllable caption generation, this absence of evaluation is a load-bearing gap.
[Section 2.3.2] Retrieval-augmented planning selects the top N thought-chain examples by cosine similarity of the user instruction embedding only (Eq. 1), with no use of the image content or the specific combination of constraints. The paper provides no evaluation of retrieval precision, no analysis of how the retrieved chains affect planning quality, and no discussion of failure cases. Since the entire agent control loop depends on the retrieved examples to produce correct actions, this unvalidated dependency is a load-bearing weakness rather than a minor detail.
[Section 2.2] The instruction-evolution step relies on external context from Google Lens and Google Search to identify entities such as 'Tesla Cybercab,' 'Venom 3,' and 'King Charles III.' The paper does not verify the accuracy of this web-derived context or measure the effect of incorrect or irrelevant context on the evolved instruction and final caption. Given that the examples in Figures 4-5 are explicitly tied to such externally sourced facts, the risk of hallucination or context drift is real and unaddressed. This further undermines the 'context-aware' and 'ensures' claims.

minor comments (5)

[General] The paper is labeled 'A technical report' on page 2, but the abstract and conclusion use definitive language such as 'ensures' and 'delivers.' The claims should be aligned with the level of evidence provided, or the evidence should be added.
[Table 1] The row for 'Format' reads 'Some images have strong emotions, and it is a better choice to describe the image with the corresponding emotional tone,' which appears to describe Sentiment rather than Format. This is likely a copy-and-paste error and should be corrected.
[Figures 4-5] The text in the visualization figures is very small and dense, making it difficult to read the user instructions, professional instructions, and captions. Consider presenting these examples in a larger format or in a table.
[References] The correctness criterion in Section 2.2 cites [14] (MIA-Bench), but the connection is not explained in the text. Please clarify how MIA-Bench informs the 'describable by humans' criterion.
[Section 2.3.3] The caption expansion and condensation tools are described without specifying the prompts or the mechanism that preserves other constraints such as keywords or sentiment. Please provide more detail on how these tools interact with the full set of user requirements.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper contains no fitted parameters, no derivation chain equivalent to its inputs, and no load-bearing self-citation; its claims are under-evaluated but not circular.

full rationale

This paper is a system description and qualitative visualization report rather than a derivation-based method. The load-bearing claims are that instruction evolving converts simple instructions into professional instructions and that the CapAgent agent controls caption generation via planning, retrieval-augmented planning, and tool use. None of these components is fitted to the paper's own outputs, and no numerical result is predicted from a fitted parameter. The retrieved chain examples are curated by the authors, but retrieval is explicitly by cosine similarity in BGE-M3 embedding space (Eq. 1) and is not claimed to be derived from the final captions. The visualization examples are produced by the system itself, but they are presented as illustrations rather than as evidence obtained by derivation. The references include prior work, including [18] for constraint types and [20] for ReAct, but these are external and do not constitute a self-citation chain that forces the central claim. The abstract's word 'ensures' for constraint adherence is not supported by any verification mechanism or evaluation, and the stopping criterion relies on the planner's self-assessment; however, this is an evaluation and validation gap, not circular reasoning. There is no equation in which an input is defined in terms of an output, no fitted input renamed as a prediction, and no uniqueness theorem imported from the authors' own prior work. Under the rule that only explicit reduction to inputs counts as circularity, the paper merits a score of zero.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its free parameters are two unspecified retrieval counts, N and K, that influence the system's behavior. The central assumptions are domain-level beliefs about the reliability of GPT-4o, embedding-based retrieval, web search context, and the captioning tools, none of which are empirically validated in the report.

free parameters (2)

N (number of retrieved thought-chain examples) = not specified
The paper defines Cs = TopN of similar examples but never gives a value for N. This choice affects planning quality and is not derived from data.
K (number of similar images retrieved by Google Lens) = not specified
The paper says Google Lens searches top K similar images but does not specify K. This context retrieval step influences the professional instruction and the final caption.

assumptions (4)

domain assumption GPT-4o can reliably perform instruction evolution and generate correct tool-using actions from retrieved examples.
The entire system depends on GPT-4o's instruction-following and planning ability, asserted in Sections 2.2 and 2.3 without evaluation.
domain assumption Cosine similarity in BGE-M3 embedding space selects relevant and useful planning examples.
Section 2.3.2 assumes that similar user demands correspond to similar thought processes, but no retrieval accuracy or downstream planning success is measured.
domain assumption Google Lens and Google Search provide accurate and helpful context about the image.
Section 2.2 uses web summaries as image context c, but the paper does not assess whether these summaries are correct or beneficial.
domain assumption The suite of tools (VQA, sentiment modification, expansion, condensation, counting, spatial relation) can enforce the specified constraints as expected.
Section 2.3.3 describes each tool's purpose, but no experiment verifies that tool outputs actually satisfy the constraints in the professional instruction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Simple to Professional: A Combinatorial Controllable Image Captioning Agent." pith.science (2026). https://pith.science/paper/QR2MLIBL

@misc{pith2026241211025,
  author       = {Pith},
  title        = {Pith review of: From Simple to Professional: A Combinatorial Controllable Image Captioning Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QR2MLIBL}},
  note         = {Machine review of arXiv:2412.11025}
}

read the original abstract

The Controllable Image Captioning Agent (CapAgent) is an innovative system designed to bridge the gap between user simplicity and professional-level outputs in image captioning tasks. CapAgent automatically transforms user-provided simple instructions into detailed, professional instructions, enabling precise and context-aware caption generation. By leveraging multimodal large language models (MLLMs) and external tools such as object detection tool and search engines, the system ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting. CapAgent transparently controls each step of the captioning process, and showcases its reasoning and tool usage at every step, fostering user trust and engagement. The project code is available at https://github.com/xin-ran-w/CapAgent.

Figures

Figures reproduced from arXiv: 2412.11025 by the authors.

**Figure 2.** We first use Google Lens to search top K similar images on the web. Then, we use the LLM to generate a summary of the phrases or sentences that need to be further searched according to titles of similar images. Finally, we use the LLM summary of the Google Search results and Google Lens results as the image context c. We combine user instruction s and image x with c together to prompt MLLM to generate the profession… view at source ↗

**Figure 2.** Process of instruction evolution with context information extracted from the webpage by [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: , like most general agents, CapAgent’s workflow includes three main steps: planning, tool usage, and observation. When the user inputs an image and a caption query, the CapAgent will generate a thought and a corresponding action to tackle the user request. In each acti…

**Figure 4.** Figure 4: Professional instruction and caption examples of CapAgent (Page 1). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Professional instruction and caption examples of CapAgent (Page 2). [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages

[1]

Good news, everyone! context driven entity-aware captioning for news images

Ali Furkan Biten, Lluis Gomez, Marçal Rusinol, and Dimosthenis Karatzas. Good news, everyone! context driven entity-aware captioning for news images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12466–12475, 2019

work page 2019
[2]

Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

2024
[3]

Microsoft coco captions: Data collection and evaluation server

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

arXiv 2015
[4]

Benchmarking and improving detail image caption

Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092, 2024

arXiv 2024
[5]

Flex- cap: Generating rich, localized, and flexible captions in images, 2024

Debidatta Dwibedi, Vidhi Jain, Jonathan Tompson, Andrew Zisserman, and Yusuf Aytar. Flex- cap: Generating rich, localized, and flexible captions in images, 2024

work page 2024
[6]

Im- ageinwords: Unlocking hyper-detailed image descriptions

Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Ya- sumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Im- ageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024

arXiv 2024
[7]

Visual sketchpad: Sketching as a visual chain of thought for multimodal language models

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.09403, 2024

arXiv 2024
[8]

Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Alan Hayes

Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Alan Hayes. Gpt-4o system card, 2024

work page 2024

Show all 20 references

[9]

Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents, 2024

Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents, 2024

2024
[10]

St Edward's Crown

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems...

2020
[11]

Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024

2024
[12]

Senticap: Generating image descriptions with sentiments

Alexander Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descriptions with sentiments. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

2016
[13]

Docci: Descriptions of connected and contrasting images

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. arXiv preprint arXiv:2404.19753, 2024

2024 arXiv
[14]

Mia-bench: Towards better instruction following evaluation of multimodal llms

Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia-bench: Towards better instruction following evaluation of multimodal llms. arXiv preprint arXiv:2407.01509, 2024

2024 arXiv
[15]

Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

2018
[16]

Transform and tell: Entity-aware news image captioning

Alasdair Tran, Alexander Mathews, and Lexing Xie. Transform and tell: Entity-aware news image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13035–13045, 2020

2020
[17]

Caption anything: Interactive image description with diverse multimodal controls, 2023

Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao. Caption anything: Interactive image description with diverse multimodal controls, 2023

2023
[18]

Benchmarking complex instruction-following with multiple constraints composition

Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, et al. Benchmarking complex instruction-following with multiple constraints composition. arXiv preprint arXiv:2407.03978, 2024

2024 arXiv
[19]

Depth anything v2, 2024

Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2, 2024

2024
[20]

React: Synergizing reasoning and acting in language models

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 9

2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Good news, everyone! context driven entity-aware captioning for news images

Ali Furkan Biten, Lluis Gomez, Marçal Rusinol, and Dimosthenis Karatzas. Good news, everyone! context driven entity-aware captioning for news images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12466–12475, 2019

work page 2019

[2] [2]

Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

2024

[3] [3]

Microsoft coco captions: Data collection and evaluation server

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

arXiv 2015

[4] [4]

Benchmarking and improving detail image caption

Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092, 2024

arXiv 2024

[5] [5]

Flex- cap: Generating rich, localized, and flexible captions in images, 2024

Debidatta Dwibedi, Vidhi Jain, Jonathan Tompson, Andrew Zisserman, and Yusuf Aytar. Flex- cap: Generating rich, localized, and flexible captions in images, 2024

work page 2024

[6] [6]

Im- ageinwords: Unlocking hyper-detailed image descriptions

Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Ya- sumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Im- ageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024

arXiv 2024

[7] [7]

Visual sketchpad: Sketching as a visual chain of thought for multimodal language models

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.09403, 2024

arXiv 2024

[8] [8]

Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Alan Hayes

Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Alan Hayes. Gpt-4o system card, 2024

work page 2024