Pith. sign in

REVIEW 4 cited by

Altogether: Image Captioning via Re-aligning Alt-text

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.17251 v3 pith:SLQNQOQ4 submitted 2024-10-22 cs.CV cs.CL

Hu Xu , Po-Yao Huang , Xiaoqing Ellen Tan , Ching-Feng Yeh , Jacob Kahn , Christine Jou , Gargi Ghosh , Omer Levy
show 5 more authors
Luke Zettlemoyer Wen-tau Yih Shang-Wen Li Saining Xie Christoph Feichtenhofer
This is my paper · ORCID
classification cs.CVcs.CL
keywords imagedataexistingalt-textaltogethercaptionsimagesalt-texts
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners' training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

    cs.CV 2025-04 conditional novelty 7.0 of 10

    PerceptionLM releases 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions, plus models and a new benchmark, arguing that human data, not just synthetic data, is needed for detailed video understanding.

  2. ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ScaleCap is an inference-time captioning pipeline that enriches captions with heuristic questions and filters hallucinations via offline contrastive sentence rating, yielding a 450K dataset that improves LVLM pretrain...

  3. RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction

    cs.CV 2025-05 conditional novelty 6.0 of 10

    RICO refines image captions by reconstructing them into images with a text-to-image model and asking GPT-4o to fix discrepancies against the original, iteratively, with a DPO-distilled fast variant.

  4. CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Training-free CorrCLIP reconstructs patch correlations in CLIP with SAM masks and DINO similarity, raising averaged mIoU across eight benchmarks from 48.6 to 53.6.

Pith tools