{"id":"1ae853f8-d739-4a93-aa56-f9ee20c4c981","arxiv_id":"2607.23235","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reference-free caption quality is scored by the downstream vision-language accuracy of a caption-conditioned reconstructed image, via a new CTTD benchmark.","lead":"This paper proposes judging image captions without human references: regenerate the image from the caption, then score how well a vision-language model can answer questions about the regenerated image. A new compact benchmark, CTTD, is introduced to make this evaluation cheap and scalable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) is not monotone in the Eq. (4) semantic-equivalence criterion: a lossy caption can produce an easier reconstruction and score higher, so the central operational claim is unvalidated.","rationale":"The reader's weakest_assumption already identifies Eq. (5) as not algebraically equivalent to Eq. (4) and as assuming monotonicity of judge accuracy on reconstructions. My stress test sharpens this into a concrete inversion risk: because Eq. (5) drops the original-image term, any caption that makes the reconstruction easier for the judge can be scored above a semantically richer caption. This is not an external disagreement with consensus; it is an internal gap between the stated criterion and the deployed score. I credit the paper for explicitly acknowledging both the non-equivalence and the simplification bias, and for labeling its trends as internal consistency rather than validation. However, the central claim as stated ('operationalized by Eq. (5)') is not established until this inversion rate is measured. The proposed test would settle it: if degraded captions rarely outscore faithful ones and Eq. (4)/Eq. (5) rankings correlate strongly, the concern is mostly theoretical; if not, the metric requires revision (e.g., report Eq. (4) distances or use original-image-normalized scores). Because the paper is framed as a diagnostic with an explicit call for human validation, the appropriate verdict remains conditional rather than accept or reject, matching the reader's CONDITIONAL assessment.","tokens_in":19717,"tokens_out":6065,"duration_ms":56190,"concrete_test":"On a sample of CTTD images, take each model's caption C_full and construct a degraded C_degraded by deleting attribute, OCR-relevant, and fine-detail phrases. Reconstruct both with Qwen-Image and judge with Qwen3-VL-8B. Compute per-image Eq. (5) scores and Eq. (4) distances to the original-image judge performance. Report (i) the fraction of images where Eq. (5)(C_degraded) > Eq. (5)(C_full), and (ii) the Spearman correlation between Eq. (5) and Eq. (4) across all captions. If the fraction is non-negligible or the correlation is weak/negative, Eq. (5) is not a valid operationalization of the semantic-equivalence criterion and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Eq. (5) operationalizes Principle 2, but Eq. (5) ranks captions by judge accuracy on the reconstruction alone, whereas Eq. (4) ranks by absolute deviation from original-image judge accuracy. For fixed image I, Eq. (4) is |A_i - J(hat I)| with A_i=J(I,D_i), while Eq. (5) is J(hat I); the two rankings coincide only if J(hat I) <= A_i for every caption, i.e., only if a reconstruction is never easier for the judge than the original. The paper concedes 'a simplified reconstruction may be easier for the judge' and that Eq. (5) 'is not algebraically equivalent' to Eq. (4). If a caption drops hard-to-render content (OCR text, small objects, unusual layout), the reconstruction can be cleaner and can yield higher judge accuracy than both the original image and the reconstruction from a more faithful caption. Then Eq. (5) ranks the lossy caption above the faithful one, inverting the semantic-preservation ordering that Eq. (4) is supposed to capture. No experiment measures how often this flip occurs; the reported tables compare model-level averages, where such inversions and generator/judge confounds are averaged away.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reference-free, reconstruction-based framework for image-caption evaluation. It replaces exact pixel reconstruction with a semantic-equivalence criterion: a caption is good if its caption-conditioned reconstruction is semantically interchangeable with the original image, as judged by downstream vision-language task performance (Eq. (4)). Because this is expensive, the score actually used is Eq. (5), the average judge accuracy on the reconstructed image, and a compact surrogate, the Captioning Turing Test Dataset (CTTD), is introduced. The paper reports model rankings for 13 captioners across VQA, STEM, OCR, and grounding tasks, plus CTTD, and includes ablations over generators, judgers, and caption length. The authors explicitly state throughout that these are internal consistency results and that direct correlation with human caption judgments remains the central unmeasured validation.","tokens_in":20049,"tokens_out":5731,"duration_ms":60834,"significance":"If the operational score Eq. (5) were shown to track semantic preservation, the framework would be a useful reference-free, task-conditioned diagnostic for captioning, complementing human preference metrics and reference-based scores. The paper has notable strengths: it releases the CTTD benchmark and code, evaluates across multiple established vision-language datasets, includes generator/judge/length ablations, and is unusually candid about component dependence and the lack of human validation. However, the significance is conditional: the central operational claim is not yet validated, and the paper itself concedes that Eq. (5) is not algebraically equivalent to Eq. (4), that simplified reconstructions may be easier for the judge, and that direct human correlation is still required. These concessions are not merely cosmetic; they affect the interpretation of every reported ranking.","major_comments":[{"comment":"The central operational claim is that Eq. (5) realizes Principle 2, but it does not. For fixed I and D_i, Eq. (4) thresholds the mean absolute deviation |J(I,D_i)-J(\\hat I,D_i)|, while Eq. (5) ranks by mean J(\\hat I,D_i). These rankings coincide only if J(\\hat I) is never above J(I) for every caption; the paper concedes 'a simplified reconstruction may be easier for the judge.' A caption that drops OCR text, small objects, or unusual layout can produce a cleaner reconstruction and score above both the original image and a more faithful caption, inverting the intended ordering. No experiment quantifies how often this flip occurs. Tables 1 and 3 use Eq. (5), so the load-bearing monotonicity is unvalidated. Please report per-image or per-caption inversion rates against an Eq. (4)-style original-relative score, with error bars.","section":"§3.2, Eqs. (4)–(5)"},{"comment":"The paper repeatedly states that direct correlation with human judgments of caption quality 'remains unmeasured and is therefore the central validation still required.' I agree, and this absence is load-bearing for the title's claim of 'caption evaluation.' The Q1/A1 patterns—traditional captioners below LVLMs, within-family scaling—are internal consistency checks, not evidence that the score tracks semantic fidelity as perceived by humans. Without at least system-level and ideally caption-level correlation with human semantic-preservation judgments, the framework is a proposed diagnostic rather than a validated evaluation metric. This is the single most important missing experiment.","section":"§5 and Appendix A.3 ('Discussion about human correlation')"},{"comment":"The CTTD surrogate claim rests on 'reproduces broad trends,' but no statistical agreement measure is given; only model-level averages are visually compared. Moreover, CTTD question generation and curation use the same LVLM family as the judge (Qwen2.5-VL-72B) and an automated relevance-diversity filter. The manuscript itself notes this filtering 'may favor questions that the curation components find easier,' but no analysis quantifies the selection bias. Report rank correlation (Spearman or Kendall) between CTTD and the full task suite at the image or caption level, and include a human audit of question validity and answer correctness.","section":"§4.3, Q2 and Table 3"},{"comment":"The ablation tables are used to conclude that CTTD is 'robust and not easily affected by component.' With only two generators, two judgers, and no variance estimates, this conclusion is not supported. Table 7 changes the judge and shifts absolute scores by roughly ten points while preserving rank order; systematic interactions are plausible. Report per-condition correlation coefficients, interaction analyses, and uncertainty intervals, or qualify the robustness claim accordingly.","section":"§4.2/§A.3, Tables 5–7"}],"minor_comments":[{"comment":"Typos and inconsistent labels: 'Perfermance' (Fig. 3), 'Sence' (Fig. 2), 'judger' vs. 'judge', 'Upperbound', and 'CTDD (ours)' in Fig. 10. Please unify terminology and fix typos.","section":"Throughout"},{"comment":"Eq. (6) uses GMM thresholds tau_low and tau_high but their values are not specified; the relevance-diversity parameter alpha is set to 0.5 with no sensitivity analysis. Please provide the threshold values and a small sensitivity study.","section":"§3.3 / §4.2"},{"comment":"Reference [21] duplicates [11] (both are Radford et al., CLIP). The self-citation [10] (CCCaption) is not load-bearing to the framework; if kept, it should be explicitly related in the related-work discussion.","section":"References"},{"comment":"The claim that CTTD shows 'stronger consistency' than CaptionQA/FLEUR is based on visual rank comparison. Report rank-correlation coefficients with the VLT reference ranking and confidence intervals.","section":"Table 2 / Fig. 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and well executed, but the central validation is missing and the operational score can invert the stated semantic-equivalence criterion. I would not accept without either adding human correlation experiments and quantifying Eq. (5) vs. Eq. (4) inversions, or reframing the contribution as a benchmark/diagnostic rather than a validated evaluation metric. The self-citation [10] is incidental and not a concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the genuinely new piece is the protocol: score a caption by generating a reconstruction and running a fixed multi-task vision-language judge on it, with CTTD as a cheaper version of the full task suite. Second, the paper never anchors its practical score to its own semantic-equivalence criterion, and it openly says so. Eq. (5) ranks captions by the reconstruction's judge accuracy; Eq. (4) defines equivalence by the absolute gap to original-image judge accuracy. A caption that drops OCR text or a small object can produce a cleaner, easier reconstruction and score higher under Eq. (5) than a faithful caption. That inversion is exactly the failure mode the stress-test flagged, and the authors concede it in Section 3.2 and the conclusion. They do not measure how often it happens.\n\nWhat the paper does well: it is unusually candid. It explicitly labels Eq. (5) as not algebraically equivalent to Eq. (4), identifies generator, judge, and question-selection confounds, and says human correlation is \"the central validation still required.\" That is real honesty. The internal experiments are also clean: within-family scaling trends, CTTD reproduces the full suite's broad ordering, and ablations across generators, judges, and caption lengths do not overturn the relative rankings. The CTTD pipeline is transparent, the free parameters are stated, code is promised, and no parameters are tuned to produce the model ordering. The single self-citation is not load-bearing.\n\nSoft spots, in order of size. The Eq. (4)/(5) mismatch is load-bearing because the whole framework is justified by semantic equivalence, but the reported scores come from a different quantity. The paper needs at least an inversion-rate analysis on a sample where Eq. (4) and Eq. (5) are both computed. Second, no error bars or sensitivity to alpha, tau_low/tau_high, or sigma are reported; all conclusions are point estimates on a single pipeline. Third, no correlation with human caption judgments is measured. The authors' own discussion that \"human evaluation often fails to distinguish fluency and semantics\" is an argument, not evidence; the construct validity of the score is unestablished.\n\nWho this is for: caption-evaluation people who want a reference-free, task-conditioned diagnostic. It should go to peer review, because the idea is testable and the paper is transparent. But I would treat the current version as a proposal plus a benchmark, not a validated metric. A serious referee should ask for the inversion-rate analysis and a human-correlation study before the score is used for model selection.","headline":"A transparent proposal whose practical score is not tied to its own equivalence criterion and lacks human validation.","tokens_in":20508,"tokens_out":3286,"would_cite":false,"duration_ms":63287,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that a caption's quality be measured by whether the image reconstructed from it can answer the same downstream vision-language questions as the original.","keywords":["image captioning evaluation","reference-free metric","semantic equivalence","text-to-image reconstruction","vision-language tasks","downstream question answering","captioning turing test dataset","caption quality"],"falsifier":"Have humans rank a set of captions for the same image by semantic adequacy and measure the rank correlation with CTTD scores using the default generator and judge; near-zero or negative correlation would refute the central claim. A second check: hold captions fixed, vary the text-to-image generator, and see whether model rankings invert; if they do, the score is measuring the generator at least as much as the caption.","tokens_in":19625,"feed_emoji":"🖼️","tokens_out":4884,"duration_ms":49052,"temperature":0.7,"pith_summary":"The paper proposes that a caption's quality be judged by whether a text-to-image model, building an image from the caption alone, lets a vision-language judge answer the same downstream questions as the original image. The authors formalize this as semantic equivalence: a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original, operationalized by the average downstream-task accuracy of the reconstruction. This yields a reference-free, task-conditioned score that complements human-written reference metrics. To make the idea practical, they introduce the Captioning Turing Test Dataset (CTTD), a compact curated question-answer benchmark that reproduces the broad trends of the full task suite at about ten times lower compute. The paper is explicit that the practical score is not algebraically equivalent to the semantic-equivalence criterion, that a simplified reconstruction can be easier for a judge, and that direct correlation with human caption judgments remains the central validation still required.","feed_headline":"Judge captions by rebuilding the image and asking it questions","feed_subtitle":"New reference-free score ranks captions by how well a text-to-image reconstruction preserves the original's semantics.","key_machinery":"The load-bearing object is the semantic equivalence class [I] = {I' : I' ~ I}, with the equivalence relation approximated by Eq. (4): a reconstruction and the original are semantically equivalent when a fixed judge model achieves comparable performance across a suite of vision-language datasets. The practical scoring identity is Eq. (5): S_D(I, C) = (1/n) sum_i J(f_img(C), D_i), the judge's accuracy when conditioned on the caption-generated reconstruction. CTTD is a curated set of about 7,000 images with roughly 15 categorized multiple-choice question-answer pairs per image, built by clustering LVLM-proposed questions into 15 categories and selecting by relevance-diversity; it serves as a lo","core_discovery":"The central claim is Principle 2: for a fixed image, a caption is better than another if the image reconstructed from it lies closer to the semantic equivalence class of the original. Because semantic equivalence is not directly observable, the paper approximates it with a suite of downstream vision-language tasks: if replacing the original image with the reconstruction lets a fixed judge achieve comparable performance, the two images count as semantically equivalent. The practical score, Eq. (5), is the judge's average accuracy on the reconstructed image, with original-image performance as an internal upper reference. Across thirteen captioning models, the authors find that traditional capt","pith_inferences":["The reconstruction score could serve as a training signal for captioning models: optimizing captions to maximize downstream judge accuracy would directly target semantic preservation without human references, though the paper does not propose this.","The same semantic-equivalence test could extend to video captioning, image editing, or instruction evaluation, wherever a text-conditioned reconstruction can be judged by downstream tasks.","Different generator-judge pairs effectively operationalize different notions of semantic content, so reporting scores across a matrix of generators and judges would be a natural stress-test of the metric's stability.","Because Eq. (5) can favor simplified reconstructions that are easier for the judge, an editorial inference is that scores should be normalized against an image-level baseline (such as original-image or random-guess accuracy) before comparing captions across heterogeneous images."],"forward_implications":["Caption evaluation can proceed without human-written references: the score is computed from the caption, a frozen text-to-image generator, a judge model, and a question bank.","The score is task-conditioned: a caption may score high on VQA and low on OCR or grounding, revealing which semantic dimensions survive caption compression.","CTTD is a scalable, lower-cost proxy that reproduces the broad model rankings of the full task suite at about ten times less compute.","OCR and grounding gaps indicate that fine-grained textual detail and spatial layout are the hardest information to preserve through caption compression.","Scores must be interpreted under fixed pipeline components; the paper notes that changing the generator, judge, or question set can change the ranking."],"fun_headline_variants":["Rebuild the image to score captions without references","New score: captions ranked by how well they rebuild the image","Reference-free caption eval via semantic image reconstruction","Ask the rebuilt image: a better way to judge captions","Caption quality by semantic equivalence to the original image"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The score assumes that a judge's accuracy on a caption-conditioned reconstruction is a valid and roughly monotone proxy for how faithfully the caption preserves image semantics; if a simpler reconstruction inflates accuracy, or if generator and judge errors dominate, the ranking no longer tracks caption quality.","fun_headline_variants_meta":{"raw":{"variants":["Rebuild the image to score captions without references","New score: captions ranked by how well they rebuild the image","Reference-free caption eval via semantic image reconstruction","Ask the rebuilt image: a better way to judge captions","Caption quality by semantic equivalence to the original image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000108,"raw_usage":{"total_tokens":863,"prompt_tokens":705,"completion_tokens":158,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":80}},"tokens_in":449,"tokens_out":158,"duration_ms":2087,"temperature":1.0,"reasoning_tokens":80,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:59:18.319916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have humans rank a set of captions for the same image by semantic adequacy and measure the rank correlation with CTTD scores using the default generator and judge; near-zero or negative correlation would refute the central claim. A second check: hold captions fixed, vary the text-to-image generator, and see whether model rankings invert; if they do, the score is measuring the generator at least as much as the caption.","supporting_citations":[],"review_version":1}