Towards Diverse and Accurate Image Captions via Reinforcing Determinantal Point Process

Antoni B. Chan; Qingzhong Wang

REVIEW 3 major objections 5 minor 246 references

Towards Diverse and Accurate Image Captions via Reinforcing Determinantal Point Process

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reinforcement-learning objective built on a determinantal point process generates image captions that are simultaneously more accurate and more diverse than noise-conditioned baselines.

desk verdict R-DPP is a simple, empirically promising diversity-accuracy training reward for captioning, but the claimed equivalence to DPP training rests on an invalid gradient derivation and should be treated as heuristic. read the letter →

arxiv 1908.04919 v1 pith:J5RVHMNY submitted 2019-08-14 cs.CV cs.CL

Qingzhong Wang , Antoni B. Chan This is my paper · ORCID

classification cs.CVcs.CL

keywords imagecaptioningdiversedeterminantalpointprocessreinforcementlearningCIDErself-CIDErdiversitybeamsearch

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes R-DPP, a reinforcement-learning objective that combines a determinantal point process (DPP) with a CIDEr-based reward to train image captioning models. The authors aim to show that a single training objective can deliver both high-quality and diverse captions, solving the trade-off that noise-conditioned models (GANs, VAEs) manage poorly. If the claim holds, captioning systems can generate a diverse set of descriptions without giving up the ability to pick a single best caption by beam search.

What carries the argument

The DPP L-ensemble is the load-bearing mechanism: a probability distribution over subsets of items in which P(C) ∝ det(L_C) for a positive semidefinite kernel matrix L. The paper sets L_ij = q_i q_j s_ij with q_i = CIDEr(c_i, C_GT) and s_ij = CIDEr(c_i,c_j) (self-CIDEr), so the determinant rewards sets whose members are individually high quality and mutually dissimilar. The training reward is R(θ) = Σ_ij sign( (L_C)^{-1}_{ij} ) L_ij p_θ(c_i)p_θ(c_j), and its policy gradient is derived from the symmetry of L_C; the sign of the inverse matrix elements tells the model whether increasing or decreasing each pair's joint probability will enlarge det(L_C), which is the DPP likelihood.

What would settle it

Compute L_C from real MSCOCO caption samples using the paper's own CIDEr quality and self-CIDEr similarity definitions and check whether the matrix is positive semidefinite; a single sampled set with a negative eigenvalue (or negative determinant) would break the DPP interpretation. Alternatively, train R-DPP with the sign term removed and compare diversity: if the diversity gain persists, the inverse-matrix sign is not the active mechanism.

Watch

Extended reading notes

Core claim

The paper's central discovery is that maximizing the determinant of a pairwise caption-quality matrix, through a policy-gradient reward, trains a captioning model to produce sets of captions that are simultaneously more accurate and more diverse than those from noise-conditioned baselines. The reward uses the CIDEr score as quality and self-CIDEr as similarity, forming a matrix L whose determinant scores a sampled set; the resulting R-DPP model improves CIDEr from 1.114 to 1.222 against an SCST baseline for single-caption beam search, and achieves the highest oracle scores when sampling 20 captions. Unlike GAN and VAE approaches, which spread probability mass and blur the mode, R-DPP preserves the modes of the learned distribution, so beam search still finds one accurate caption.

Load-bearing premise

The load-bearing premise is that the CIDEr-based matrix L, and its restriction L_C to sampled captions, behaves like a valid DPP kernel — positive semidefinite, and effectively constant with respect to the model parameters during gradient estimation — so that the reward and its policy gradient actually maximize a DPP probability.

Editorial extensions

If this is right

Any captioning model can be fine-tuned with the R-DPP reward and improve both its set-level diversity and its single-caption beam-search accuracy without architectural changes.
The training sample count m is a direct diversity-accuracy dial; larger m yields more diverse sets at the cost of some single-caption accuracy and higher compute.
R-DPP preserves the modes of the learned distribution, so the diversity gain does not come at the price of losing the ability to output one best caption.
The same objective transfers to other structured output tasks such as dialog generation or machine translation, where a trade-off between fluency and diversity exists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the DPP reward only needs a quality score and a pairwise dissimilarity, replacing CIDEr with a learned similarity function could give a training signal less tied to n-gram overlap and better aligned with human notions of caption diversity.
If mode preservation is robust, one can test whether R-DPP's probability mass stays concentrated near top-ranked captions while still spreading across diverse alternatives, in contrast to GAN and VAE models that flatten the peak.
The same determinant-reward idea could apply to other domains where a system must output a set of high-quality, mutually distinct options, such as document summarization, dialogue response generation, or recommendation lists.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a paper worth knowing about, but the theory is weaker than the abstract suggests. The new idea is simple: define a DPP-style reward over a sampled set of captions using CIDEr for both quality and self-similarity, then fine-tune a captioning model with something that looks like a policy gradient. It is model-agnostic, needs no extra branches, and the empirical results are respectable. Self-CIDEr diversity roughly doubles relative to the Att2in baseline, single-caption beam search CIDEr improves from 1.114 to 1.222 over SCST, and the oracle numbers improve too. The oracle and single-caption results are not purely circular, so the empirical claim has legs.

The main soft spot is the gradient derivation. The supplement differentiates the realized reward with respect to θ while treating L_C as constant and omits the likelihood-ratio term that accounts for the sampling distribution changing with θ. So Eq. (8) is not an unbiased estimator of ∇θ E[log det(L_C)] or ∇θ E[R]. The 'equivalent to DPP training' claim is not supported. The method can still be read as heuristic reward shaping, and empirically it seems to work, but the paper should either prove unbiasedness or explicitly label it approximate.

The PSD concern is less serious than it looks. CIDEr similarity is an average of cosine similarities over TF-IDF vectors, so the similarity matrix is a Gram matrix, and L = diag(q) S diag(q) is PSD by congruence. The paper should say this rather than leave it unstated. The self-CIDEr diversity metric overlaps with the training reward, so part of the diversity gain is by construction; the single-caption accuracy and oracle results help, but not completely. Also no code and no error bars are provided, which makes the comparisons hard to trust fully.

Who is this for? People working on diverse text generation or RL for sequence models. The idea is simple and potentially reusable. It deserves peer review: a referee should ask for a corrected derivation or an explicit heuristic framing, release code, and report variance across seeds. I would not cite it as a theoretically grounded RL-DPP method, but as a heuristic it is worth knowing.

Referee Report

3 major / 5 minor

Summary. The paper proposes R-DPP, a training method for image captioning that combines determinantal point processes (DPP) with reinforcement learning (RL). It defines a reward in Eq. (7) that uses pairwise CIDEr similarities and quality scores, derives the policy gradient in Eq. (8), and argues that maximizing this reward is equivalent to DPP training. Experiments on MSCOCO report improved diversity (self-CIDEr) and improved or competitive accuracy (CIDEr) relative to baselines such as SCST and XE+λCIDEr, including oracle results and single-caption beam search results.

Significance. If the theoretical claim were valid, R-DPP would be a simple, model-agnostic way to improve both set-level diversity and single-caption accuracy without adding model components or noise-based control signals. The empirical results, especially the oracle improvements and the beam-search single-caption scores, are potentially interesting. However, the central theoretical derivation of the policy gradient is flawed, and the diversity evaluation metric is the same as the similarity measure used in the training reward, so the paper's contributions are not yet fully established.

major comments (3)

[Supplement Eqs. (9)-(14) and Section 3 Eq. (8)] The policy gradient derivation is not a valid REINFORCE gradient for the stated objective. The reward R(θ) depends on θ both through the sampling distribution and through the explicit factors pθ(ci)pθ(cj). For the expected reward J(θ)=E_C[R(C,θ)], the gradient is E_C[∇θ R(C,θ)] + E_C[R(C,θ) ∇θ log pθ(C)]. The supplement computes only the first term for a fixed sample C, treating L_C as constant, and entirely omits the likelihood-ratio (score-function) term. Consequently, Eq. (8) is not an unbiased estimator of ∇θ J(θ) nor of ∇θ E[log det(L_C)]. The claim that maximizing Eq. (7) via Eq. (8) is equivalent to DPP training is therefore unsupported.
[Section 3, paragraph after Eq. (4)] The paper states that because L is constant w.r.t. θ, the denominator of the DPP likelihood can be ignored when maximizing the likelihood. Even if the denominator is constant, the objective to be maximized is the expected log-likelihood over sets C sampled from the model, and the reward in Eq. (7) is not det(L_C) or log det(L_C). The paper does not prove that the specific reward in Eq. (7) is a valid surrogate for det(L_C), nor does it define the exact objective function whose gradient Eq. (8) estimates. This gap is load-bearing for the central claim of equivalence to DPP training.
[Section 4, Fig. 1 and Eq. (3)] The diversity metric used for evaluation (self-CIDEr) is identical to the similarity measure sij used inside the training reward in Eq. (3). Improvements in self-CIDEr diversity are therefore partly by construction. The oracle and single-caption results in Tables 1 and 2 provide external evidence that the method does not simply overfit to the reward, but the paper should explicitly acknowledge this circularity and, ideally, also report an independent diversity metric.

minor comments (5)

[Section 3, Eq. (4)] The notation L = q T q ⊙ S is unclear. It should be written as L_{ij} = q_i q_j s_{ij} or L = diag(q) S diag(q).
[Supplement, after Eq. (13)] There is a typo: 'mtrices' should be 'matrices'.
[Supplement, Eq. (9) and paper Eq. (8)] The supplement says 'the policy gradient in Eq. (11) in our paper', but the policy gradient is Eq. (8). The cross-reference is incorrect.
[Section 2, DPP background] The paper does not mention that the CIDEr-based similarity matrix S is a Gram matrix, so L = diag(q) S diag(q) is positive semidefinite by the Schur product theorem. Stating this would justify the use of det(L_C) as a DPP.
[Section 4, Fig. 1 caption] The caption mentions 'CGAN-{1,10}' with standard deviations 1 and 10, but the figure does not clearly distinguish these points from other markers, making the comparison hard to follow.

Circularity Check

1 steps flagged · score 4.0 of 10

Diversity gains are partly built into the reward via self-CIDEr; quality and oracle results remain external.

fitted input called prediction [Section 3, Eqs. (2)-(4) and Eq. (7); Section 4, Fig. 1 caption]
"We define the similarity between captions as (i.e., 'self-CIDEr' in (Wang and Chan, 2019)), sij = CIDEr(ci,cj). ... L = qTq ⊙ S ... R(θ)=∑mi=1∑mj=1sign( ˆLCij)LCijpθ(ci)pθ(cj) ... The diversity metric is self-CIDEr diversity, which is shown to be more correlated to human judgment (Wang and Chan, 2019)."

The reward matrix LC is built from qi=CIDEr(ci,CGT) and sij=CIDEr(ci,cj), so the off-diagonal entries are pairwise self-CIDEr similarities—the same function that defines the reported self-CIDEr diversity metric. With the sign of the inverse, the update in Eq. (8) directly suppresses large off-diagonal LCij (similarity) while promoting diagonal quality terms. Thus the diversity improvement over GAN/VAE/SCST baselines is, in part, an optimization target rather than an independent DPP-derived prediction. The accuracy and oracle results are anchored to external human CIDEr/SPICE and are therefore not circular.

full rationale

The central method is transparently built to maximize a set-level reward containing self-CIDEr similarity, so the reported diversity gain is partly by construction; this is the main circularity. However, the paper's strongest independent content is (i) single-caption beam-search CIDEr/SPICE, (ii) oracle upper bounds, and (iii) comparison against XE+lambda-CIDEr at matched diversity, none of which reduces to the training reward. The authors' prior self-CIDEr metric is cited for human-judgment correlation; that is an external, falsifiable anchor, so the self-citation alone is not load-bearing circularity. Separately, the supplement's gradient derivation (Supplement A, 'Note that only p_theta(.) is a function of theta') omits the score-function term for the sampling distribution, so Eq. (8) is not an unbiased policy gradient of E[log det(LC)]; I flag this as a correctness or validity defect, not as a circularity. Overall, the diversity claim is partially circular, while the quality and oracle claims retain independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. It relies on the DPP mathematical framework (standard), the assumption that CIDEr-based L is PSD (unverified), the constancy of the DPP denominator, and the heuristic equivalence of the proposed reward to DPP training. The sample count m is a free hyperparameter chosen on validation.

free parameters (2)

m (number of sampled captions per training step) = 5 recommended (range 2 to 7 tested)
Controls the diversity-accuracy trade-off. The authors select m=5 based on oracle and diversity-accuracy results, which is a post hoc choice rather than a predicted value.
epsilon (invertibility regularizer added to L_C) = not reported
Added to ensure L_C is invertible for computing L_C^-1; the value affects the sign and scale of the reward, but is not disclosed.

assumptions (4)

domain assumption The L matrix L = q^T q (Hadamard product) S is positive semidefinite, so det(L_C) is a valid DPP probability kernel.
The paper uses CIDEr as q_i and self-CIDEr as s_ij without proving PSD. Negative CIDEr values can make qq^T indefinite, and the Schur product theorem requires both factors PSD.
standard math The denominator det(L + I) is constant with respect to model parameters theta for fixed dictionary D and caption length lm.
Follows from L being defined over a fixed finite caption space; used to drop the denominator in maximizing log det(L_C).
ad hoc to paper Maximizing the reward in Eq. (7) is equivalent to DPP training, i.e., to maximizing the expected determinant or DPP likelihood.
The paper asserts equivalence but only provides a sign-based heuristic; no proof links the quadratic reward to log det(L_C).
ad hoc to paper The policy gradient in Eq. (8) is an unbiased estimator of the gradient of the chosen objective.
The reward depends on model probabilities p_theta(c_j) and on the sampled set C through L_C; the derivation treats these as constants and omits the dependence of the sampling distribution, so unbiasedness is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Diverse and Accurate Image Captions via Reinforcing Determinantal Point Process." pith.science (2026). https://pith.science/paper/J5RVHMNY

@misc{pith2026190804919,
  author       = {Pith},
  title        = {Pith review of: Towards Diverse and Accurate Image Captions via Reinforcing Determinantal Point Process},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5RVHMNY}},
  note         = {Machine review of arXiv:1908.04919}
}

read the original abstract

Although significant progress has been made in the field of automatic image captioning, it is still a challenging task. Previous works normally pay much attention to improving the quality of the generated captions but ignore the diversity of captions. In this paper, we combine determinantal point process (DPP) and reinforcement learning (RL) and propose a novel reinforcing DPP (R-DPP) approach to generate a set of captions with high quality and diversity for an image. We show that R-DPP performs better on accuracy and diversity than using noise as a control signal (GANs, VAEs). Moreover, R-DPP is able to preserve the modes of the learned distribution. Hence, beam search algorithm can be applied to generate a single accurate caption, which performs better than other RL-based models.

Figures

Figures reproduced from arXiv: 1908.04919 by the authors.

**Figure 2.** [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

246 extracted references · 80 canonical work pages

[1]

a man sits in a diner photograph- ing his meal
[2]

photographer taking a picture of a meal in a small restaurant
[3]

a man taking a photo of food on a table
[4]

a man takes a picture of his food in a restaurant
[5]

a man taking a picture of his meal at a diner table Att2in-XE:
[6]

a man in the waiting taking while woman
[7]

a man sits in a chair holding a white and a pair of wine sitting around
[8]

a person looking at himself and sitting on a table

Show all 246 references

[9]

the man takes a picture in front on a bowl
[10]

a man taking a selfie of a view of a white pizza
[11]

person with close photo of a personal reflection on the pizza
[12]

lady in a taking in picture in front of
[13]

a woman takes a camera of a very glass window
[17]

a guy looking at something grilled black topping pizza
[18]

a man taking a photo of a small slice of pizza GMM-CVAE:
[20]

a man taking a bite of a pizza with a fork
[21]

a man is eating a meal at a restaurant
[22]

a man is eating a piece of bread
[23]

a man is taking a bite of a pizza
[24]

a man is taking a picture of a man in a white shirt
[25]

a man is holding a bowl of food on a table
[26]

a man in a red shirt is looking at a pizza
[27]

a man sitting at a table with a plate of food
[28]

a large plate of food on a table SCST:
[33]

a woman taking a picture of a pizza
[38]

a man taking a picture of a pizza" XE+5CIDEr:
[40]

a man taking a picture of a pizza with a slice of pizza
[42]

a man taking a picture of pizza on a table
[43]

a man taking a picture of pizza on a plate
[45]

a man taking a picture of a pizza with a
[47]

a man taking a picture of a pizza with a pizza
[48]

a man taking a picture of a pizza with a pizza XE+10CIDEr:
[50]

a man taking a picture of pizza on a table with a camera
[51]

a man taking a picture of a pizza with a camera
[52]

a man taking a picture of pizza with a persons of
[53]

a man taking a picture of a pizza with a knife
[56]

a man taking a picture of a pizza with a salad
[57]

a person taking a picture of a pizza with a salad
[58]

a person taking a camera of a pizza with a salad R-DPP(m=5):
[60]

a man taking a photo of a pizza in a restaurant
[61]

a woman taking a selfie in front of a pizza
[62]

a close up of a person taking a picture of a pizza
[63]

a person taking a picture of a pizza in a restaurant
[64]

a man taking a picture of a pizza on a table
[65]

a woman taking a picture of a pizza in the camera
[68]

a woman taking a picture of a pizza in a restaurant R-DPP(m=7):
[69]

a man taking a photo of a pizza with his side
[70]

a man taking a picture of a pizza
[72]

a man taking a selfie at a restaurant
[73]

a man taking a picture of a pizza on a plate
[74]

a person holding a camera in front of a pizza on a plate
[75]

a man taking a photo of a pizza
[76]

a man taking a photo of a pizza on a plate
[77]

a man taking a photo of a pizza while
[78]

a man in a suit holding a camera in front of a pizza with wine Figure 2: Human annotations:
[79]

different assortment of noodles and vegetables sitting in a pot and tray
[80]

a pot of pasta cooking on the stove next to a tray
[81]

a large pot of water and pasta sits on top of a stove
[82]

an image of food items on top of the stove
[83]

a pot of noodles is being cooked with more ingredients beside it Att2in-XE:
[84]

a trey of water and some fruit on it that
[85]

an food is cup sits on a table
[86]

the large meal during even a red mixture
[87]

hot plate displays a built into a bowl of sauces and a wood in top of
[88]

a piled next to a spoon and giving a fruit in a spoon
[89]

a countertop meal on a fork over in and sink
[90]

some bowl of stuff with pudding and man
[91]

a closeup is into a table that is left as has mustard and
[92]

a plate of food with some mexican
[93]

a measuring bowl of with front of a bowls and a GMM-CVAE:
[94]

a pot of food is being cooked in a pan
[95]

a pot of food is cooking
[96]

a tray of food cooking on a stove
[97]

print a pot of soup and a pot with a pot of food
[98]

a bowl of soup with a bowl of food
[99]

alight from a window a stove a stove and a metal ladle
[100]

a pot of food is on a stove
[101]

a pot of food is cooking on a stove SCST:
[103]

a bowl of fruit and sitting on a table
[106]

a bowl of food and sitting on a stove
[108]

a bowl of food and sitting on a counter
[110]

a bowl of food and sitting on a table
[111]

a bowl of food and sitting on a table XE+5CIDEr:
[112]

a bowl of food and cooking on a counter with utensils
[113]

a bowl of food and apples on a pan with a pot
[114]

a bowl of food and spoons on a pan with a pan
[115]

a spoon of food and cups on a pan with dishes
[116]

a pot of ingredients and an on a stove with a spoon
[117]

a bowl of food and vanilla dishes on a stove
[118]

a bowl of food and dishes on a pan with bowls
[119]

a bowl of food and fruit on a stove with a spoon
[120]

a bowl of food and fruit on a pan with a pot
[121]

a bowl of food and spoons on a stove with a bowl XE+10CIDEr:
[122]

a bowl of food and spoon on a wooden shelf with a spoon
[123]

a bowl of food carrots and a pan on a stove
[124]

a bowl of food and fruit on a pan with utensils
[125]

a bowl of food sitting pan on a pan with food bowl
[126]

a bowl of food bowls and dishes on a pan
[127]

a bowl of food rice and spoons on a pan with spoons
[128]

a bowl of food dishes and other dishes on a pan
[129]

a bowl of food and spoon on a pan with bowls
[130]

a bowl of food and fruit on a pan with fruit
[131]

a bowl of food and strawberries on a tray with food R-DPP(m=5):
[132]

a bowl of food is sitting on a table
[133]

a toaster oven next to a bowl of food on a table
[134]

a bowl of food with a spoon on a table
[135]

a bowl of food sitting on a table next to a UNK
[136]

a tray of food with a spoon on a table
[137]

a plate of food with a spoon on top of a table
[138]

a bowl of food on a table with a bowl of food
[139]

a plate of food sitting on a table with a spoon
[140]

a bowl of food is sitting on a counter
[141]

a bowl of food on a table next to a bowl of fruit R-DPP(m=7):
[142]

a bowl with a bowl of food and a stove
[143]

a bowl of food sitting on a table next to a spoon
[144]

a bowl of food and a spoon on a table
[145]

a bowl of food and rice on a kitchen counter
[147]

a person sitting on the stove with a bowl of food
[148]

a bowl of fruit and a spoon on a kitchen counter
[149]

a bowl of cereal and a spoon on a table
[150]

a close up of a bowl of food on a table
[151]

a bowl of food and a microwave on a counter Figure 3: Human annotations:
[152]

a table with a sandwich and two cups of coffee
[153]

sliced sandwich with tomatoes on a plate on a table
[154]

theres a ham sandwich and coffee for lunch on the table
[155]

a sandwich on a lace table cloth with coffee cups
[156]

two mugs next to a white plate with a sandwich on it Att2in-XE:
[157]

a cheese and ice cream sit on a table
[158]

a sandwich is cut at and a styrofoam cup of coffee the
[159]

two pieces of beer on on a plate with coffee and cake cup
[160]

cups of coffee with a cup of coffee a coffee mug in one and tiny cup
[161]

a plate of a pizza sitting next to a cup of coffee on a white table
[162]

a table with papers and cup of coffee
[163]

a slice of ice tea is and a beverage of coffee
[164]

a white sandwich with egg and eggs on and a cup
[165]

a baked with white plate top a cup a coffee cup of coffee
[166]

a coffee on a plate with some tea and and two cups GMM-CVAE:
[169]

a table with a sandwich and a cup of coffee
[170]

a meal of eggs and a sandwich
[172]

a close up of a plate of food
[173]

a table with a plate of food and a cup of coffee
[174]

a couple of plates that are sitting on a table
[175]

a plate with a sandwich and a cup of coffee
[176]

a table with a sandwich and a cup of coffee SCST:
[180]

a sandwich and sitting on a plate with a cup of coffee
[181]

a sandwich and sitting on a table with a cup
[182]

a plate of food and sitting on a table
[184]

a cup of coffee and sitting on a table
[185]

a sandwich and sitting on a table with a cup of coffee
[186]

a cup of coffee and sitting on a table XE+5CIDEr:
[194]

a sandwich and coffee sitting on a table with a cup of coffee
[195]

a sandwich and coffee on a table with a cup of coffee
[196]

a sandwich and coffee on a table with a cup of coffee XE+10CIDEr:
[197]

a plate of breakfast and breakfast coffee on a table with coffee
[198]

a breakfast and coffee breakfast on a table with a cup of coffee
[199]

a white plate with a sandwich and coffee coffee on a table
[200]

a plate of breakfast and coffee on a table with cups
[201]

a breakfast plate of breakfast and coffee coffee on a table
[202]

a plate with breakfast plates of coffee and coffee on a table
[203]

a breakfast of breakfast and coffee on a table with coffee coffee
[204]

a breakfast plate with breakfast and drinks on a table next coffee cups
[205]

a plate of breakfast and coffee coffee on a table with coffee
[206]

a breakfast with breakfast breakfast on a table with coffee cup and coffee R-DPP(m=5):
[207]

a sandwich on a plate with a cup of coffee on it
[208]

a plate of food that is sitting on a table
[209]

a sandwich sitting on top of a table next to a cup of coffee
[210]

a sandwich sits on a table next to a cup of coffee
[211]

a sandwich sitting on a desk next to a cup of coffee
[212]

a sandwich and a cup of coffee on a table
[213]

a plate of food sitting next to a cup of coffee
[214]

a sandwich on a table with a cup of coffee on it
[215]

a plate of breakfast are sitting on a table with a cup of coffee
[216]

a plate of food on top of a table R-DPP(m=7):
[217]

a plate of food is sitting on a table
[218]

a sandwich on a plate next to a coffee cup of coffee
[219]

a sandwich is sitting on a plate on a table
[220]

a plate with a sandwich and a cup of coffee on it
[221]

a sandwich that is sitting on a plate
[222]

a large plate of food on a table
[223]

a plate of food and a coffee mug on the table
[224]

a sandwich is on a plate on a table
[225]

a sandwich sitting on top of a table with a cup of coffee
[226]

an egg and a plate of food on a wooden table Figure 4: Human annotations:
[227]

a man and a woman cross country skiing on a snow covered trail with mountain peaks in the background
[228]

a couple of skiers are going down a snowy mountain
[229]

a lady skiing looking back at a man skiing
[230]

two snow skiers coming down a snowy hill
[231]

a man and a woman cross country skiing in deep snow Att2in-XE:
[232]

two people riding skis on a snowy surface
[233]

a couple of people standing on top of a snow covered slope slope
[234]

two skiers cross country on beneath a mountain
[235]

two skiers are carrying their snow on a snowy slope
[236]

the people are racing skiing in the snow
[237]

two skiers on poles a to finish the run down the mountain way
[238]

two a number of skiers in snow with umbrellas
[239]

three people are skiing across a mountain covered land
[240]

two people skiing skis down on snowy mountain
[241]

a group and skiers are gathered their the snow GMM-CVAE:
[242]

a group of three men standing on top of a snow covered slope
[243]

a couple of men standing on top of a snow covered slope
[244]

a couple of people on a snowy mountain
[245]

users of a group of people on skis
[247]

three people standing on a snow covered slope
[248]

three men on skis are standing in the snow
[249]

a group of people standing on top of a snow covered slope
[250]

a group of three men standing next to each other on a snow covered slope
[251]

a group of people standing on a snowy surface SCST:
[256]

two people are on skis in the snow
[257]

two people are skiing on skis in the snow
[259]

a group of people on skis in the snow
[260]

a couple of people on skis in the snow
[261]

a couple of people on skis in the snow XE+5CIDEr:
[266]

two people cross country skiing on a snow covered mountain
[267]

two women standing on skis in the snow covered mountain
[270]

two people standing on skis in the snow covered mountain
[271]

two people standing on skis in the snow covered mountain XE+10CIDEr:
[272]

two people standing country skiing on a mountain mountain with
[273]

two people standing country skiing in the snow covered mountain
[274]

two people standing country skiing in the snow mountain slope
[275]

two people standing country skiing in a mountain slope
[276]

two people standing country skiing on a mountain slope
[277]

two people cross country skiing on a snow mountain mountain
[278]

two people standing on skis in the snow mountain mountain
[280]

two people standing country skiing on a mountain mountain
[281]

two people standing country skiing in the snow covered mountain R-DPP(m=5):
[282]

two people cross country skiing in the snow
[283]

two people cross country skiing on a snowy mountain
[285]

two skiers are standing on skis in the snow
[286]

two people are riding skis on a snowy mountain
[287]

two people standing on skis in the snow
[288]

two people skiing skis on top of a snow covered mountain
[289]

two people on skis standing in the snow
[290]

two people are standing in the snow on skis
[291]

a couple of people riding skis down a snow covered ski slope R-DPP(m=7):
[292]

two people are cross country skiing in the snow
[293]

two people in the back of a ski slope
[294]

two people are riding skis on a snowy slope
[295]

two people on skis in the snow
[296]

a couple of people riding skis down a snow covered slope
[297]

two skiers on skis in the snow
[298]

two people holding skis on a snowy slope
[299]

two skiers are on skis on top of a snow covered slope
[300]

two people that are standing on skis in the snow
[2015]

Cider: Consensus-based image description evaluation. In CVPR. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural im- age caption generator. In CVPR. Liwei Wang, Alexander Schwing, and Svetlana Lazeb- nik. 2017. Diverse and accurate i...

2015 arXiv
[2017]

Towards diverse and natural image descrip- tions via a conditional gan. In ICCV. M. Denkowski and A. Lavie. 2014. Meteor universal: Language speciﬁc translation evaluation for any tar- get language. In EACL Workshop on Statistical Ma- chine Translation. Aditya Deshpande, Jyoti...

2014 arXiv
[2018]

CoRR, abs/1811.10652

Show, control and tell: A framework for gen- erating controllable and grounded captions. CoRR, abs/1811.10652. Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

a man sits in a diner photograph- ing his meal

[2] [2]

photographer taking a picture of a meal in a small restaurant

[3] [3]

a man taking a photo of food on a table

[4] [4]

a man takes a picture of his food in a restaurant

[5] [5]

a man taking a picture of his meal at a diner table Att2in-XE:

[6] [6]

a man in the waiting taking while woman

[7] [7]

a man sits in a chair holding a white and a pair of wine sitting around

[8] [8]

a person looking at himself and sitting on a table