Pith. sign in

REVIEW 3 major objections 273 cited by

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

T0 review · 3 major / 0 minor · reviewed 2026-05-12 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CLIP embeddings can score how well a generated caption matches its image without any human reference captions and match human judgments better than metrics that require them.

desk verdict CLIPScore is a practical reference-free metric that beats CIDEr and SPICE on standard tests but shows clear limits on context-heavy domains. read the letter →

arxiv 2104.08718 v3 pith:U6DHRDMH submitted 2021-04-18 cs.CV cs.CL

Jack Hessel , Ari Holtzman , Maxwell Forbes , Ronan Le Bras , Yejin Choi This is my paper · ORCID
classification cs.CVcs.CL
keywords imagecaptioningevaluationmetricsreference-freeCLIPmodelhumancorrelationmultimodalsimilarityautomatic
checked against Cost.FunctionalEquationFoundation.DAlembert.Inevitability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper demonstrates that a cross-modal model trained on hundreds of millions of web image-caption pairs can measure caption quality directly through image-text similarity. This reference-free approach removes the need to collect multiple human-written descriptions for comparison. Across several captioning datasets, the resulting CLIPScore correlates more strongly with human ratings than established reference-based metrics. The method focuses tightly on visual-textual fit and proves complementary to text-only similarity measures, yielding an improved hybrid when references are available. This finding matters because reference collection is costly and limits rapid iteration in captioning research.

What carries the argument

CLIPScore, the direct cosine similarity between a CLIP model's image embedding and caption embedding that quantifies image-text compatibility without references.

What would settle it

A new human rating study on a held-out captioning dataset in which CLIPScore shows lower Pearson or Spearman correlation with the ratings than CIDEr or SPICE would falsify the central performance claim.

Watch

Extended reading notes

Core claim

CLIPScore is computed from the cosine similarity between CLIP image and text embeddings and achieves higher correlation with human judgments of caption quality than reference-based metrics such as CIDEr and SPICE on multiple corpora. A reference-augmented variant called RefCLIPScore further improves correlation by incorporating text-text similarity as well. The approach performs strongly on literal description tasks and domains such as clip-art but shows relative weakness on captions that require external contextual knowledge.

Load-bearing premise

That CLIP's web-pretrained image and text representations already encode a general, transferable signal of caption quality that holds across domains without task-specific retraining.

Editorial extensions

If this is right

  • Captioning systems can be evaluated automatically in settings where reference captions are unavailable or expensive to collect.
  • Hybrid reference-plus-CLIP metrics become preferable when references exist, as they capture both visual fit and textual fluency.
  • The metric remains reliable on literal visual descriptions but requires caution on tasks that demand world knowledge beyond the image.
  • Evaluation pipelines can now incorporate CLIPScore as a fast, scalable complement to slower human studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Captioning models could be trained end-to-end by treating CLIPScore as a differentiable reward signal instead of relying solely on cross-entropy or CIDEr optimization.
  • The same reference-free idea may extend to evaluating other image-text outputs such as visual question answering answers or story generation from images.
  • Domains where CLIPScore underperforms, such as news images, point to the need for additional knowledge sources that current web-pretrained embeddings do not supply.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper introduces CLIPScore, a reference-free metric for image captioning evaluation that computes cosine similarity between CLIP embeddings of an image and a generated caption from a fixed, publicly pretrained model. It claims this metric achieves the highest correlation with human judgments across several corpora, outperforming reference-based metrics such as CIDEr and SPICE, while information-gain experiments show complementarity to text-text similarity metrics. A reference-augmented variant (RefCLIPScore) is also presented that yields even higher correlations. Case studies highlight strong performance on clip-art and alt-text tasks but weaker results on news captions requiring external context.

Significance. If the empirical correlations hold, the work would be significant for establishing a simple, parameter-free, reference-free evaluation method that aligns better with human judgments than standard n-gram or scene-graph metrics. The absence of any fitting to evaluation datasets and the explicit complementarity analysis are strengths that could shift evaluation practices in vision-language research toward leveraging large pretrained multimodal models.

major comments (3)
  1. [Abstract] Abstract and experimental results: the claim of consistent outperformance and highest correlation with human judgments lacks reported exact Pearson/Spearman values, confidence intervals, or statistical significance tests comparing CLIPScore to CIDEr and SPICE; without these, the superiority assertion cannot be fully evaluated.
  2. [Case studies] Case studies section: weaker performance on news captions is noted as requiring richer contextual knowledge, but no domain-stratified splits, ablation on CLIP variants, or controls for distribution shift are described; this directly challenges the robustness claim for the web-pretrained embeddings across captioning domains.
  3. [Experiments] Human judgment collection: potential confounds (e.g., annotation instructions, inter-annotator agreement details, or selection bias in the corpora) are not addressed, which is load-bearing for validating that CLIPScore's image-text compatibility signal truly tracks quality rather than artifacts of the judgment process.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the presentation of our empirical claims and limitations. We address each major point below and have revised the manuscript accordingly where feasible.

read point-by-point responses

  1. Referee: [Abstract] Abstract and experimental results: the claim of consistent outperformance and highest correlation with human judgments lacks reported exact Pearson/Spearman values, confidence intervals, or statistical significance tests comparing CLIPScore to CIDEr and SPICE; without these, the superiority assertion cannot be fully evaluated.

    Authors: The manuscript reports Pearson and Spearman correlations in Tables 2 and 3 for multiple datasets, showing CLIPScore outperforming CIDEr and SPICE. However, we agree that the abstract and main text do not highlight exact values, confidence intervals, or significance tests. In revision, we will update the abstract with key correlation figures and add bootstrap-derived 95% confidence intervals plus paired significance tests (e.g., Williams test) to the experimental section. This will allow direct evaluation of the outperformance claims without altering the underlying results. revision: yes

  2. Referee: [Case studies] Case studies section: weaker performance on news captions is noted as requiring richer contextual knowledge, but no domain-stratified splits, ablation on CLIP variants, or controls for distribution shift are described; this directly challenges the robustness claim for the web-pretrained embeddings across captioning domains.

    Authors: The case studies are qualitative illustrations of domain differences rather than a comprehensive robustness study; the primary claims rest on the aggregate results across standard captioning benchmarks. We explicitly flag the news-caption limitation in the manuscript. To address the concern, the revision will include a short discussion of potential distribution shift between web-pretraining data and news domains, plus a note that future work could explore CLIP variants or fine-tuning. No new ablations or stratified splits are added, as the focus remains on the fixed public model, but the limitation is now stated more prominently. revision: partial

  3. Referee: [Experiments] Human judgment collection: potential confounds (e.g., annotation instructions, inter-annotator agreement details, or selection bias in the corpora) are not addressed, which is load-bearing for validating that CLIPScore's image-text compatibility signal truly tracks quality rather than artifacts of the judgment process.

    Authors: The human judgments are taken from previously published evaluation datasets whose collection protocols are described in the cited source papers. We will expand the experimental setup section in revision to summarize the key details of annotation instructions, reported inter-annotator agreement, and corpus construction from those references. This addition will make explicit that CLIPScore is evaluated against the same human signals used by prior metrics, while acknowledging any known limitations of the original judgment processes. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CLIPScore is a fixed function of an external pretrained model; correlations are measured empirically.

full rationale

The paper defines CLIPScore as a direct cosine similarity computation in the fixed CLIP embedding space (pretrained on 400M web pairs, no parameters tuned on caption evaluation data). The reported correlations with human judgments are post-hoc empirical measurements on standard corpora, not quantities fitted or defined in terms of the target results. No self-citation chain, ansatz smuggling, or renaming of known results is load-bearing for the central claim. The derivation is self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the empirical performance of the fixed pretrained CLIP model applied to caption evaluation; no new free parameters are introduced and no new entities are postulated.

assumptions (1)
  • domain assumption CLIP embeddings capture semantic image-text compatibility sufficiently well to serve as a proxy for human caption quality judgments.
    Invoked when defining CLIPScore as the cosine similarity between image and text features without further justification or calibration on the target task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIPScore: A Reference-free Evaluation Metric for Image Captioning." pith.science (2026). https://pith.science/paper/U6DHRDMH

@misc{pith2026210408718,
  author       = {Pith},
  title        = {Pith review of: CLIPScore: A Reference-free Evaluation Metric for Image Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U6DHRDMH}},
  note         = {Machine review of arXiv:2104.08718}
}
read the original abstract

Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 273 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 273 Pith citations

  1. A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems

    cs.CV 2026-08 conditional novelty 7.0 of 10

    By asking a unified multimodal model to caption an image, regenerate it, and answer questions about the regenerated image, SGU consistently exposes accuracy drops that separate benchmarks do not reveal.

  2. MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.

  3. MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Distilling frozen Motion-JEPA features into a compact 32-D latent whose geometry is coupled to the decoder lets a standard non-autoregressive flow-matching DiT reach state-of-the-art text-to-motion quality on HumanML3...

  4. JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models

    cs.LG 2026-07 conditional novelty 7.0 of 10

    JAGG replaces per-step gradient backpropagation in diffusion GRPO with two endpoint backward passes joined by timestep-weighted interpolation, giving ~2x backward-pass savings at modest quality cost.

  5. UniNDM: A Unified Noise-driven Detection and Mitigation Framework Against Sexual Content in Text-to-Image Generation

    cs.CV 2026-07 conditional novelty 7.0 of 10

    UniNDM detects sexual intent from early-stage diffusion noise and mitigates it via LLM-generated negative prompts and initial-noise optimization, across U-Net and DiT models.

  6. C3-Bench: A Context-Aware Change Captioning Benchmark

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    C3-Bench supplies a multi-domain dataset and LLM-based evaluation protocol that exposes systematic failures in existing change captioning models outside their training regimes.

  7. SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SafeGen-Bench is a benchmark with 10 malicious categories that evaluates conditional T2V models on paired start frames and text prompts, finding unsafety scores up to 44.5 and 80% guardrail failure rate.

  8. Garment Particles: A 2D--3D Symmetric Garment Representation for Generation and Editing

    cs.GR 2026-05 unverdicted novelty 7.0 of 10

    Garment Particles is a 5D point cloud representation jointly encoding 2D sewing patterns and 3D geometry, supporting rectified flow generation from high-level inputs and diffusion-based editing of patterns or shapes.

  9. DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DiffusionOPD applies online policy distillation from per-task teachers to a unified diffusion student, with a derived closed-form per-step KL objective that unifies SDE and ODE sampling via mean matching.

  10. D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    D-OPSD formulates supervised fine-tuning of step-distilled diffusion models as on-policy self-distillation by minimizing distribution differences between a text-only student and a multimodal teacher on the student's o...

  11. 1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation

    cs.CV 2026-04 conditional novelty 7.0 of 10

    1.x-Distill achieves better quality and diversity than prior few-step distillation methods at 1.67 and 1.74 effective NFEs on SD3 models with up to 33x speedup.

  12. Reflective Flow Sampling Enhancement

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    RF-Sampling enhances flow matching models by implicitly performing gradient ascent on text-image alignment scores via linear textual combinations and flow inversion.

  13. Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks

    cs.CV 2025-09 unverdicted novelty 7.0 of 10

    Neural-MedBench reveals sharp performance drops in state-of-the-art VLMs on reasoning-intensive neurology tasks compared to conventional classification benchmarks, with reasoning failures dominating errors.

  14. Towards Multimodal Active Learning: Efficient Learning with Limited Paired Data

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Introduces the first active learning framework for unaligned multimodal data that selects alignments using uncertainty and diversity to cut annotation costs by up to 40% on benchmarks while preserving accuracy.

  15. DiffusionNFT: Online Diffusion Reinforcement with Forward Process

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    DiffusionNFT performs online RL for diffusion models on the forward process via flow matching and positive-negative contrasts, delivering up to 25x efficiency gains and rapid benchmark improvements over prior reverse-...

  16. MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation

    cs.CV 2025-09 conditional novelty 7.0 of 10

    A new 340K-image human-annotated dataset, a trained vision-language assessor, and an automated benchmark reveal that even state-of-the-art T2I models produce artifacts in roughly a third of output images.

  17. FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark

    cs.CV 2025-09 conditional novelty 7.0 of 10

    The authors build a 6M-image, 20M-caption reasoning dataset with generation chain-of-thought and a 7-track VLM-judged benchmark, then rank 19 text-to-image models.

  18. Durian: Dual Reference Image-Guided Portrait Animation with Attribute Transfer

    cs.CV 2025-09 conditional novelty 7.0 of 10

    Durian introduces a dual-reference diffusion model trained via self-reconstruction on video frames to enable cross-identity attribute transfer in portrait animations, supporting multi-attribute composition and interpolation.

  19. MultiRef: Controllable Image Generation with Multiple Visual References

    cs.CV 2025-08 conditional novelty 7.0 of 10

    MultiRef-bench shows that current image generators that accept multiple visual references still fail to combine them reliably, with the best tested model OmniGen reaching only 66.6% synthetic and 79.0% real-world alig...

  20. LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs

    cs.CV 2025-07 conditional novelty 7.0 of 10

    A large human-annotated benchmark of AI-edited images (EBench-18K) plus a fine-tuned LMM metric (LMM4Edit) that predicts human preference scores across three dimensions and answers editing-specific questions.

  21. A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation

    cs.CV 2025-06 conditional novelty 7.0 of 10

    Layer-normalized averaging of all decoder-only LLM hidden states, rather than last-layer embeddings, improves text-to-image compositional alignment and beats T5 on GenAI-Bench.

  22. Your Demands Deserve More Bits: Referring Semantic Image Compression at Ultra-low Bitrate

    eess.IV 2025-05 conditional novelty 7.0 of 10

    RSIC allocates bits to user-specified image regions via a grounding model and guides a pretrained diffusion decoder with the compressed latent, boosting local fidelity at ultra-low rates.

  23. R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A 3,068-prompt benchmark with per-instance Q&A scoring shows that current text-to-image models, including reasoning-enhanced ones, handle reasoning-driven prompts poorly, with mathematical reasoning near zero.

  24. D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples

    cs.CV 2025-05 conditional novelty 7.0 of 10

    Mask-guided self-attention fusion creates well-aligned target images that stay visually close to poorly-aligned base images, with full denoising trajectories, and DPO on these pairs improves alignment.

  25. LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A new benchmark with 58,500 AI videos and 2.6 million human annotations, plus an LMM-based metric that predicts perceptual quality, prompt-video correspondence, and task-specific correctness.

  26. Towards Understanding Camera Motions in Any Video

    cs.CV 2025-04 conditional novelty 7.0 of 10

    CameraBench is a new expert-annotated benchmark with a 50-primitive camera motion taxonomy, revealing that SfM and VLMs fail in complementary ways and that SFT on the benchmark boosts VLM performance.

  27. Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editing

    cs.CV 2025-04 conditional novelty 7.0 of 10

    ELECT selects the best random seed for instruction-guided image editing by scoring background consistency from early diffusion latents, reducing inference cost by about 41 percent and turning about 40 percent of previ...

  28. WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    Text-to-image models show significant limitations in integrating world knowledge, as measured by the new WISE benchmark and WiScore metric across 20 models.

  29. Latent Swap Joint Diffusion for 2D Long-Form Latent Generation

    cs.SD 2025-02 conditional novelty 7.0 of 10

    A training-free latent swap method that replaces averaging with binary swapping in joint diffusion, improving long-form audio spectrum and panorama generation.

  30. Towards Affordance-Aware Articulation Synthesis for Rigged Objects

    cs.CV 2025-01 conditional novelty 7.0 of 10

    A training-free optimization framework synthesizes affordance-aware articulation for arbitrary rigged 3D objects by aligning them with 2D diffusion-inpainted references.

  31. Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution

    cs.CV 2024-12 conditional novelty 7.0 of 10

    CrossFlow turns text directly into images, and images into text, depth, and higher resolution, by flowing between modality latents without a noise prior or cross-attention.

  32. IDEA-Bench: How Far are Generative Models from Professional Designing?

    cs.CV 2024-12 conditional novelty 7.0 of 10

    IDEA-Bench measures generative models on 100 professional design tasks and finds the best tested system scores only 22.48 out of 100.

  33. EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM

    cs.CV 2024-12 conditional novelty 7.0 of 10

    EasyRef uses a multimodal LLM to encode multiple reference images and a text prompt into learned tokens, enabling zero-shot multi-reference personalization of Stable Diffusion XL without per-group fine-tuning.

  34. T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts

    cs.CV 2024-12 unverdicted novelty 7.0 of 10

    T2I-FactualBench is a new three-tier benchmark for factuality of knowledge-intensive concepts in T2I models, using multi-round VQA evaluation to show SOTA models need improvement.

  35. Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression

    cs.CV 2024-11 conditional novelty 7.0 of 10

    A masked-autoregressive diffusion model trained on a compact essential-feature latent space claims state-of-the-art text-to-motion generation under a new essential-dimension evaluation protocol.

  36. Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

    cs.CV 2023-10 unverdicted novelty 7.0 of 10

    Latent Consistency Models enable high-fidelity text-to-image generation in 2-4 steps by directly predicting solutions to the probability flow ODE in latent space, distilled from pre-trained LDMs.

  37. LayeringDiff: Layered Image Synthesis via Generation, then Disassembly with Generative Knowledge

    cs.CV 2025-01 conditional novelty 6.5 of 10

    LayeringDiff synthesizes layered images by generating a composite with a pretrained diffusion model and then decomposing it into foreground and background layers using small fine-tuned networks.

  38. KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using learned 32×32 Kronecker block transforms as an online activation smoother improves W4A4 image quality of PixArt-Sigma, SANA, and FLUX.1-schnell over SVDQuant and LoRaQ, with a kernel up to 14% faster than SmoothQuant.

  39. GroupVideo: Multi-Identity Customized Text-to-Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    GroupVideo generates multi-person videos from reference photos plus text, using multimodal identity alignment and ID localization to keep each person's identity consistent.

  40. Importance-Aware OBS Pruning for Diffusion Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Injecting spatial importance maps (e.g., CFG delta) into the OBS Hessian improves subject preservation in pruned diffusion models at high sparsity, but gains over the baseline are small and without error bars.

  41. Optimizing Visual Generative Models via Distribution-wise Rewards

    cs.LG 2026-07 unverdicted novelty 6.0 of 10

    Distribution-wise rewards with subset-replace strategy and post-hoc merging improve FID-50K on SiT (8.30 to 5.77) and EDM2 (3.74 to 3.52) while preserving diversity.

  42. Histogram-constrained Image Generation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    HIG enforces exact histogram constraints on diffusion-generated images by modeling the control task as an optimal transport problem and applying guidance transformations during sampling.

  43. Lighting-Consistent Object Transfer Across Radiance Fields

    cs.GR 2026-06 unverdicted novelty 6.0 of 10

    Diffusion-based per-view harmonization for lighting-consistent object transfer between 3DGS scenes, using heterogeneous training data and final 3D consolidation.

  44. Customizing Video Portraits via Identity-ActionDecoupling

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Proposes IaD framework with Identity Decoupling Loss and Text Alignment Loss for richer, identity-consistent IPT2V without subject-specific fine-tuning.

  45. Polaris: Scaling Up Instruction-Guided Image Generation Towards Millions of Personalized Style Needs

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Polaris retrieves and integrates relevant models from a large library of checkpoints and adapters to enable scalable instruction-guided image generation and editing without additional training.

  46. Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    A constrained optimization framework for diffusion model unlearning via KL and likelihood constraints, with duality results and reported better retention-unlearning tradeoffs than weight-based baselines.

  47. Broken Memories: Detecting and Mitigating Memorization in Diffusion Models with Degraded Generations

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Proposes stability regions based on latent update norms to detect and mitigate memorization in diffusion models, reporting AUC over 0.999 and zero memorization rate after mitigation on Stable Diffusion 1.4.

  48. LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    LIFT decomposes distillation into coarse linear alignment then fine refinement while PLACE adds error-based local adaptation, allowing stable training of 1.3M-parameter students (1.6% teacher size) to FID 15.73 across...

  49. ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    ClaimDiff-RL replaces holistic scalar rewards with reference-conditioned atomic claim differences verified by a multimodal judge to improve the hallucination-missing-fact tradeoff in long-form image captioning.

  50. LESA: Learnable Stage-Aware Predictors for Diffusion Model Acceleration

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A learned, stage-segmented KAN predictor for feature caching accelerates diffusion transformers by 5-6.25x while preserving more image/video fidelity than prior training-free forecasters.

  51. RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    RL-RIG uses a generate-reflect-edit loop with reinforcement learning to improve spatial accuracy in image generation, reporting up to 11% gains over prior open-source models on scene-graph metrics.

  52. Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Prototypicality bias: common text-to-image metrics systematically prefer plausible-but-wrong images over correct non-prototypical ones; PROTOSCORE mitigates but does not eliminate the failure.

  53. EmoFeedback$^2$: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedback

    cs.CV 2025-11 reject novelty 6.0 of 10

    A closed-loop emotional image generation system uses a fine-tuned vision-language model both as a reinforcement-learning reward and as an iterative prompt refiner, claiming improved valence-arousal fidelity.

  54. Distribution Matching Distillation Meets Reinforcement Learning

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Combining DMD distillation with RL during training produces few-step text-to-image models that outperform their multi-step teacher on several benchmarks.

  55. Rethinking Robust Adversarial Concept Erasure in Diffusion Models

    cs.CV 2025-10 conditional novelty 6.0 of 10

    S-GRACE generates semantically guided adversarial prompts and fine-tunes only the text encoder, reporting stronger concept-erasure robustness and ~90% lower training time than prior adversarial erasure methods.

  56. StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback

    cs.CV 2025-10 conditional novelty 6.0 of 10

    StableSketcher improves text-to-sketch generation by fine-tuning a diffusion VAE and adding a VQA-based RL reward, while releasing the SketchDUO dataset of sketches with captions and QA pairs.

  57. NoiseShift: Resolution-Aware Noise Recalibration for Better Low-Resolution Image Generation

    cs.CV 2025-10 unverdicted novelty 6.0 of 10

    NoiseShift learns a resolution-specific mapping from scheduler noise to conditioning noise via lightweight calibration to restore consistency and improve low-resolution generation quality in models like SD3 and Flux.

  58. A Unified Framework for Diffusion Model Unlearning with f-Divergence

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Diffusion model unlearning is generalized from KL/MSE to any f-divergence, with closed-form Hellinger and chi-square losses and a variational min-max form.

  59. VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis

    cs.CV 2025-09 unverdicted novelty 6.0 of 10

    VC-Inspector introduces a lightweight open-source LMM and a controllable factual-error generation framework that achieves state-of-the-art correlation with human judgments on reference-free video caption evaluation.

  60. Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration

    cs.AI 2025-09 conditional novelty 6.0 of 10

    A multi-agent prompt-refinement system using pairwise AI judging and targeted edit signals outperforms prior automated methods on complex text-to-image tasks.

See all 273 Pith citations

Reference graph

Works this paper leans on

70 extracted references · 70 canonical work pages · cited by 273 Pith papers (see all)

  1. [1]

    Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos. 2015. From images to sentences through scene description graphs using commonsense reasoning and knowledge. arXiv preprint arXiv:1511.03292

    work page Pith review arXiv 2015

  2. [2]

    Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating CLIP : Towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02818

    work page arXiv 2021

  3. [3]

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In ECCV. Springer

    work page 2016

  4. [4]

    Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, 7:597--610

    work page 2019

  5. [5]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop on Evaluation Measures for MT and Summarization

    work page 2005

  6. [6]

    Berg, Tamara L

    Alexander C. Berg, Tamara L. Berg, Hal Daumé III, Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, and Kota Yamaguchi. 2012. Understanding and predicting importance in images. In CVPR

    work page 2012

  7. [7]

    Ali Furkan Biten, Lluis Gomez, Mar c al Rusinol, and Dimosthenis Karatzas. 2019. Good news, everyone! context driven entity-aware captioning for news images. In CVPR

    work page 2019

  8. [8]

    John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In COLING

    work page 2004

Show all 70 references
  1. [9]

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In ECCV

    2020

  2. [10]

    Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In CVPR

    2018

  3. [11]

    Bo Dai and Dahua Lin. 2017. Contrastive learning for image captioning. In NeurIPS

    2017

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL

    2019

  5. [13]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognit...

    2021

  6. [14]

    Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In ACL

    2014

  7. [15]

    Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris, Kris M Kitani, and Jeffrey P Bigham. 2019. ``it's almost like they're trying to hide it": How user-provided image descriptions have failed to make twitter accessible. In WWW

    2019

  8. [16]

    Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M Kitani, and Jeffrey P Bigham. 2020. Twitter a11y: A browser extension to make twitter images accessible. In CHI

    2020

  9. [17]

    Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV), pages 771--787

    2018

  10. [18]

    Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 47:853--899

    2013

  11. [19]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML

    2021

  12. [20]

    Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. 2019. TIGEr: text-to-image grounding for image caption evaluation. In EMNLP

    2019

  13. [21]

    Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT : An empirical study. In ICLR

    2020

  14. [22]

    Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020. NUBIA : N e U ral based interchangeability assessor for text generation. In 1st Workshop on Evaluating NLG Evaluation

    2020

  15. [23]

    Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In EACL

    2017

  16. [24]

    Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Trung Bui, and Kyomin Jung. 2021. UMIC: an unreferenced metric for image captioning via contrastive learning. In ACL

    2021

  17. [25]

    Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. Vilbertscore: Evaluating image caption using vision-and-language bert. In First Workshop on Evaluation and Comparison of NLP Systems

    2020

  18. [26]

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In ECCV

    2018

  19. [27]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out

    2004

  20. [28]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft COCO : Common objects in context. In ECCV. Springer

    2014

  21. [29]

    Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. 2018. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In ECCV

    2018

  22. [30]

    Chi-kiu Lo. 2019. Yisi-a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. In Fourth Conference on Machine Translation

    2019

  23. [31]

    Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267--300

    2013

  24. [32]

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT : Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS

    2019

  25. [33]

    Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In CVPR

    2020

  26. [34]

    Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. NewsCLIPpings: automatic generation of out-of-context multimodal media. arXiv preprint arXiv:2104.05893

    2021

  27. [35]

    Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. In CVPR

    2018

  28. [36]

    Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's experiences with computer-generated captions of social media images. In CHI

    2017

  29. [37]

    Pranava Madhyastha, Josiah Wang, and Lucia Specia. 2019. VIFIDEL : Evaluating the visual fidelity of image descriptions. In ACL

    2019

  30. [38]

    Yashar Mehdad, Matteo Negri, and Marcello Federico. 2012. Match without a referee: evaluating mt adequacy without reference translations. In Seventh Workshop on Statistical Machine Translation

    2012

  31. [39]

    Shikib Mehri and Maxine Eskenazi. 2020. USR : An unsupervised and reference free evaluation metric for dialog generation. In ACL

    2020

  32. [40]

    Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In FAccT

    2019

  33. [41]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  34. [42]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL

    2002

  35. [43]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. JMLR, 12

    2011

  36. [44]

    Maxime Peyrard and Iryna Gurevych. 2018. Objective function learning to match human judgements for optimization-based summarization. In NAACL

    2018

  37. [45]

    Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT ? In ACL

    2019

  38. [46]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision

    2021

  39. [47]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  40. [48]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In EMNLP

    2018

  41. [49]

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. IJCV

    2017

  42. [50]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL

    2016

  43. [51]

    Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aur \'e lie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017. FOIL it! find one mismatch between image and language caption. In ACL

    2017

  44. [52]

    Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In CVPR

    2019

  45. [53]

    Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS

    2016

  46. [54]

    Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. Machine translation, 24(1):39--50

    2010

  47. [55]

    Lucia Specia and Kashif Shah. 2018. Machine translation quality estimation: Applications and future perspectives. In Translation Quality Assessment, pages 201--235. Springer

    2018

  48. [56]

    Abigale Stangl, Meredith Ringel Morris, and Danna Gurari. 2020. ``person, shoes, tree. is the person naked?" what people with vision impairments want in image descriptions. In CHI

    2020

  49. [57]

    Simeng Sun and Ani Nenkova. 2019. The feasibility of embedding based automatic evaluation for single document summarization. In EMNLP

    2019

  50. [58]

    Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In AAAI

    2018

  51. [59]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS

    2017

  52. [60]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR

    2015

  53. [61]

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. TPAMI, 39(4):652--663

    2016

  54. [62]

    Sijin Wang, Ziwei Yao, Ruiping Wang, Zhongqin Wu, and Xilin Chen. 2021. FAIEr : Fidelity and adequacy ensured image caption evaluation. In CVPR

    2021

  55. [63]

    Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT . In EMNLP

    2019

  56. [64]

    Elizaveta Yankovskaya, Andre T \"a ttar, and Mark Fishel. 2019. Quality estimation and translation metrics via pre-trained word and sentence embeddings. In Fourth Conference on Machine Translation

    2019

  57. [65]

    Yanzhi Yi, Hangyu Deng, and Jinglu Hu. 2020. Improving image captioning evaluation by considering inter references variance. In ACL

    2020

  58. [66]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67--78

    2014

  59. [67]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore : Evaluating text generation with BERT . In ICLR

    2020

  60. [68]

    Wei Zhao, Goran Glava s , Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation. In ACL

    2020

  61. [69]

    C Lawrence Zitnick and Devi Parikh. 2013. Bringing semantics into focus using visual abstraction. In CVPR

    2013

  62. [70]

    Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In EMNLP

    2020

Pith tools

Reviewed May 12, 2026 · model on record in the stance chip above.