Towards Automatic Evaluation for Image Transcreation

Claire He; Graham Neubig; Simran Khanuja; Vivek Iyer

REVIEW 3 major objections 4 minor 12 references

Towards Automatic Evaluation for Image Transcreation

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automatic evaluation of image transcreation is now feasible, with segment-level correlations to human ratings of 0.55–0.87 across seven countries, provided the metric is matched to the dimension being judged.

desk verdict First automatic evaluation suite for image transcreation, with shared code and honest limitations, but the 0.87 visual-similarity correlation is likely sign-flipped and should be fixed before the results are used. read the letter →

arxiv 2412.13717 v3 pith:ZCFK4TP5 submitted 2024-12-18 cs.CL cs.CV

Simran Khanuja , Vivek Iyer , Claire He , Graham Neubig This is my paper · ORCID

classification cs.CLcs.CV

keywords imagetranscreationautomaticevaluationculturalrelevancesemanticequivalencevisualsimilarityvision-languagemodelsmeta-evaluationmachinetranslationmetrics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Image transcreation — adapting an image so it resonates with a different culture — has been stalled as a machine learning problem because evaluation required expensive human annotation. This paper proposes the first suite of automatic metrics for the task, organized around three dimensions borrowed from translation studies: cultural relevance, semantic equivalence, and visual similarity. The metrics come in three flavors — object-based overlap of culture-specific items, embedding-based cosine similarities from SigLIP, and scores from vision-language models such as GPT-4o and Gemini. Meta-evaluation against human ratings for 3,543 images across seven countries reports segment-level correlations from 0.55 to 0.87, with the best metric varying by dimension. If these numbers hold, image transcreation systems can be trained, benchmarked, and validated at scale without per-image human review.

What carries the argument

The load-bearing mechanism is a three-part metric suite, each part tied to one of the identified transcreation dimensions. Object-based CSI-Overlap detects culture-specific items (CSIs) in the source with Gemini-1.5-Pro, asks a language model for valid target-culture replacements, and scores the fraction of CSIs correctly replaced. Embedding-based metrics use SigLIP to compute three cosine similarities: the change in alignment with a cultural reference phrase ($\Delta S_{c\text{-}r}$), the alignment of the transcreated image with the transcreation intent ($S_{s\text{-}e}$), and the source-to-target image similarity ($S_{v\text{-}s}$). VLM-based metrics prompt GPT-4o, Gemini-1.5-Pro, or Molmo for 1–5 ratings with step-by-step reasoning. All scores are compared to human ratings through Kendall's tau at segment level, which is the evaluation harness that produces the reported correlations.

What would settle it

Apply the metrics to a new image transcreation test set with several independent human ratings per image; if the best metric on each dimension drops below roughly 0.5 Kendall's tau on average, the paper's claim of strong metric–human agreement would be undercut.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that automatic evaluation of image transcreation can be made to work, provided the metric is matched to the dimension being judged. Proprietary vision-language models give the strongest agreement with human ratings on cultural relevance and semantic equivalence, while SigLIP embedding similarity — a dual-encoder approach — outperforms every VLM on visual similarity, reaching a Kendall's tau of 0.87. The paper also shows that a simple object-based metric that checks whether culture-specific items are replaced with valid local substitutes correlates weakly or negatively on some dimensions, and that abstract categories such as religion and education are harder for all metrics than concrete ones like food or birds. The authors' recommended practice is a hybrid evaluation strategy: use VLM scores where reasoning is required and embedding similarity where fine visual fidelity matters.

Load-bearing premise

The human ratings used as ground truth come from one dataset where a single annotator rated each of about 3,500 images, so noisy or biased human scores would weaken every reported correlation.

Editorial extensions

If this is right

Transcreation systems can be developed with automatic, inexpensive feedback instead of continuous human annotation, at least for the three measured dimensions.
Benchmarking becomes practical: a hybrid evaluation—VLM scores for cultural relevance and semantic equivalence, SigLIP embeddings for visual similarity—can rank competing pipelines without human judges.
The 0.87 result for embedding-based visual similarity suggests that dual-encoder training captures fine-grained visual change better than generative multimodal reasoning.
The metric suite can serve as a reward model or data filter in training loops for image transcreation models, enabling optimization toward human-aligned adaptation.
Because cultural relevance is the hardest dimension across all metrics, the paper gives clear headroom for new methods aimed at measuring cultural fit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The three chosen axes are assumed sufficient to capture transcreation quality; adding dimensions like naturalness or offensiveness, which already exist in the underlying human dataset, could yield a more complete evaluation profile.
With only one annotator per image, the reported correlations are likely upper bounds on true metric-human agreement; re-annotating a subset with multiple raters would provide a stability estimate.
The cultural-relevance metrics rely on country names ('This image belongs to country X'); richer cultural descriptors tied to specific festivals, foods, or demographics could raise the currently low cultural-relevance correlations.
The same three-axis framework could transfer to video, audio, or mixed-media localization, where cultural adaptation matters as much as it does in still images.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know before you read this. It's the first automatic evaluation suite for image transcreation, and they've released code and prompts, so the work is easy to build on. But the headline 0.87 for SigLIP visual similarity is contradicted by the paper's own ground-truth scale, and you should not trust that number until the authors explain the sign handling.

What's genuinely new: three metric families—object-based, embedding-based, and VLM-based—grounded in translation studies, and a meta-evaluation across 7 countries against human ratings from a prior dataset. No free parameters are fitted to the human judgments, and the metrics are tested on independently collected ratings, which is good practice. The hybrid recommendation (proprietary VLMs for cultural relevance and semantic equivalence, SigLIP for visual similarity) is sensible if the underlying correlations are right.

Now the problems, in size order. First, the polarity. Section 4.2 defines visual similarity as cosine(V_src, V_tgt), and says a high value means the image stayed close to the source. The human question used as ground truth is 'Is there any visual change in the generated image compared to the original image?', where 1 = no change and 5 = high changes. That's a change-positive scale. The VLM prompt in Appendix A uses the same scale. So a positive correlation between the similarity metric and the change rating is contradictory unless one of them was reverse-coded. The paper nowhere states that. This is load-bearing: the 0.87 entry is the basis for recommending embeddings for visual similarity. The stress-test note about this is correct. The authors need to state explicitly whether they correlated Sv-s with the raw change score or with a similarity-transformed score, and if a transform was used, justify it. The same ambiguity affects the Gemini and GPT-4o visual-similarity numbers.

Second, the empirical base is thin. One dataset, one annotator per image, three systems, no confidence intervals or significance tests. The authors own this in the Limitations section, and I respect that, but the correlations should be reported as point estimates, not as settled benchmark numbers. Excluding segments with non-parsable outputs or identical scores is fine, but the paper should quantify how many were dropped.

Third, the object-based metric is the weakest, and the paper says so; that's not a flaw, but the category-level analysis is only a link to a GitHub page, so the claims about concrete vs. abstract categories are hard to verify without digging into the repo.

Who is this for? Anyone building or evaluating image transcreation systems, or working on cultural localization and multimodal evaluation. It deserves a serious referee, but the polarity question has to be resolved before publication. I'd also ask for uncertainty bounds or a sensitivity analysis. If that's fixed, this is a solid first contribution.

Recommendation: send to peer review, with a clear request that the authors fix the visual-similarity sign handling and add at least a paragraph on why the correlations are stable.

Referee Report

3 major / 4 minor

Summary. The paper proposes a suite of automatic evaluation metrics for image transcreation, a task that adapts an image from a source culture to a target culture. Drawing on translation studies, the authors identify three evaluation dimensions—cultural relevance, semantic equivalence, and visual similarity—and implement three metric families: an object-based CSI-Overlap metric, embedding-based cosine-similarity metrics using SigLIP, and VLM-based prompted scores using Gemini-1.5-Pro, GPT-4o, and Molmo. The metrics are meta-evaluated against human ratings from the Khanuja et al. (2024) dataset across seven countries, reporting segment-level Kendall-Tau correlations. The findings indicate that proprietary VLMs best capture cultural relevance and semantic equivalence, while SigLIP embeddings excel at visual similarity, leading the authors to recommend a hybrid evaluation strategy.

Significance. If the reported results are reliable, this paper would provide the first automatic evaluation framework for image transcreation, enabling training and benchmarking beyond costly human evaluation. The theoretical grounding in translation studies, the explicit identification of three evaluation dimensions, and the release of code are strengths. However, the empirical evidence is limited to a single dataset with one annotator per image and only three systems, and the paper reports no confidence intervals or significance tests. More critically, the sign of the visual similarity correlation appears inconsistent with the stated definitions of the metric and the human rating scale, which threatens the central recommendation about embedding-based metrics. The paper makes a useful conceptual contribution, but the validity of its headline quantitative claims requires substantial revision.

major comments (3)

[§4.2(c), Table 3, Appendix A] The visual similarity metric Sv-s is defined as cosine(Vsrc, Vtgt), with higher values indicating that the transcreated image has stayed close to the source. The human rating used as ground truth (Table 2) is the answer to "Is there any visual change in the generated image compared to the original image?", and the VLM prompt in Appendix A anchors 1 = "No visual change", 5 = "High visual changes". Under these definitions, higher human scores denote more change, so Sv-s should correlate negatively with the human ratings. Table 3 reports +0.87 for SigLIP, +0.52 for Gemini, and +0.17 for GPT-4o, without any mention of reverse-coding the human scores or the metric. This sign inconsistency is load-bearing because the recommendation to use embedding-based metrics for visual similarity and the conclusion that these "excel" at this dimension depend on the magnitude and sign of this correlation. The paper must either provide the reversed human scoring, explain a sign flip in the correlation computation, or reinterpret the human item as a similarity scale.
[§5.4, Table 3] The correlations in Table 3 are presented as average segment-level values across countries, but no confidence intervals, significance tests, or per-country variance are reported. With only three systems and a single annotator per image (as acknowledged in §7), the estimates may be unstable; for example, the cultural relevance correlation for Molmo is -0.16 while Gemini is 0.81, a spread that is not discussed statistically. The paper should provide bootstrap confidence intervals or significance tests for the correlations, and should report per-country results to substantiate the claim that the metrics "agree strongly" with human ratings. This is necessary because the comparative claims about metric families rest on these point estimates.
[§5.3, Appendix A] The VLM-based cultural relevance metric is not fully specified: the prompt returns separate scores for the source and target images, but the paper does not state how these two numbers are combined into a single dimension score (e.g., difference, ratio, or target-only). Similarly, the visual similarity VLM prompt asks for a "visual changes" score (1 = no change, 5 = high changes), and it is unclear whether this score is used directly as a "similarity" metric or reversed. Without these aggregation details, the reported correlations for VLM-based metrics cannot be reproduced or interpreted. Please provide the exact scoring function for each dimension and the full prompts for all models (GPT-4o and Molmo prompts are not listed in Appendix A).

minor comments (4)

[Abstract and §1] The phrase "average segment-level correlations ranging from 0.55-0.87" is ambiguous; specify which metric, which dimension, and whether these are averages across countries or across systems.
[§5.3] The paper skips "segments with non-parsable model outputs, or those scored identically by all systems" without reporting how many segments were excluded. Please clarify the exact criterion and quantify the excluded data, as this may affect the correlation estimates.
[§4.2(b) and §5.3] The semantic equivalence embedding metric is described with Tint as "This is a food item" in §4.2(b), but §5.3 says "This image belongs to category X"; the notation and description should be aligned.
[Table 3 and §5.5] The statement that "VLM-based metrics are highly effective in capturing cultural relevance and semantic equivalence" is too broad, since Molmo's cultural relevance correlation is -0.16; qualify the claim by model or specify that it refers to proprietary VLMs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed metrics are evaluated against independently collected human ratings from prior work, with no fitted parameters and no load-bearing self-citation chain.

full rationale

The paper's derivation chain is not circular. The three metric families (object-based, embedding-based, VLM-based) are defined from translation-studies dimensions (Sections 2-4), and their quality is assessed by correlating metric outputs with human ratings from Khanuja et al. (2024). No metric parameter is fitted to those ratings; each metric is computed from images and text prompts alone (Section 5.3), and the correlations in Table 3 are external evaluations, not constructions. The VLM prompts paraphrase the human evaluation questions (Appendix A), but this is construct alignment—the VLM is an independent judge, and the human ratings are independent ground truth. The reliance on a single dataset from the same research group, and the paper's own caveat that 'we only have one evaluation dataset available (Khanuja et al., 2024), which only provides scores for three systems, with only one annotator rating per target image' (Section 7), is a generalization limitation, not circularity. The apparent sign mismatch between the visual-similarity metric (cosine similarity, high = close to source) and the human 'visual change' item (high = more change) is a potential correctness/validity concern, but it is not a reduction of the result to its inputs, so it does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters and no new entities. Its central claim rests on domain assumptions about the sufficiency of the three evaluation dimensions, the validity of single-annotator human ratings as ground truth, the reliability of model-generated object and replacement lists, and the use of English prompts for cross-cultural judgment.

assumptions (4)

domain assumption The three dimensions, cultural relevance, semantic equivalence, and visual similarity, are a sufficient decomposition of transcreation quality.
Adopted from translation studies in Sections 2 and 4. The paper does not prove these axes cover all relevant aspects, and Section 7 acknowledges that the operationalization via country names and category descriptions loses finer-grained nuance.
domain assumption The human ratings from Khanuja et al. (2024) are valid ground truth for the three evaluation dimensions.
Used as the benchmark in Section 5.1. The prior dataset has one annotator per image and only three systems, both acknowledged as limitations in Section 7.
domain assumption English prompts can elicit culturally valid assessments for non-English-speaking target cultures.
All VLM prompts are in English, as stated in Section 7. Cultural relevance may be expressed differently in target languages, and this is untested.
domain assumption Gemini-1.5-Pro's object detections and proposed replacement lists are reliable enough for the object-based metric.
The CSI-Overlap metric depends on model-generated object lists, culture-specific item identification, and valid replacement sets (Section 4.1 and Appendix A). No human verification of these intermediate lists is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Automatic Evaluation for Image Transcreation." pith.science (2026). https://pith.science/paper/ZCFK4TP5

@misc{pith2026241213717,
  author       = {Pith},
  title        = {Pith review of: Towards Automatic Evaluation for Image Transcreation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCFK4TP5}},
  note         = {Machine review of arXiv:2412.13717}
}

read the original abstract

Beyond conventional paradigms of translating speech and text, recently, there has been interest in automated transcreation of images to facilitate localization of visual content across different cultures. Attempts to define this as a formal Machine Learning (ML) problem have been impeded by the lack of automatic evaluation mechanisms, with previous work relying solely on human evaluation. In this paper, we seek to close this gap by proposing a suite of automatic evaluation metrics inspired by machine translation (MT) metrics, categorized into: a) Object-based, b) Embedding-based, and c) VLM-based. Drawing on theories from translation studies and real-world transcreation practices, we identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity, and design our metrics to evaluate systems along these axes. Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity. Meta-evaluation across 7 countries shows our metrics agree strongly with human ratings, with average segment-level correlations ranging from 0.55-0.87. Finally, through a discussion of the merits and demerits of each metric, we offer a robust framework for automated image transcreation evaluation, grounded in both theoretical foundations and practical application. Our code can be found here: https://github.com/simran-khanuja/automatic-eval-img-transcreation.

Figures

Figures reproduced from arXiv: 2412.13717 by the authors.

**Figure 1.** Image Transcreation in the real world: The Simpsons (1989) was remade into Al-Shamshoon (2005) for a Middle-Eastern audience. transcreation, seeks to adapt (or localize) images originating from a source culture for a given target culture in an attempt to enhance familiarity and relatability for the target audience (Khanuja et al., 2024). We provide a popular real-world example of image transcreation in [PITH_FULL_I… view at source ↗

**Figure 2.** The Coca-Cola homepage depicts different [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Both Slack ads convey how the product can be used to facilitate hybrid work, but the names and avatars of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Examples of transcreation to adhere to cultural norms. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Transcreation reflecting cultural differences – in the U.S. adaptation of Sailor Moon, Artemis is portrayed as cunning and mischievous (“catty"), while in Japan, cats are seen as part of the cute kawaii culture. 3.3 Children’s Media Intent: to aid learning and cogniti…

**Figure 6.** Figure 6: The newscaster in the Pixar movie "Zootopia" [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 6 canonical work pages

[2]

arXiv preprint arXiv:2409.17146

Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146. Mar Díaz-Millón and María Dolores Olvera-Lobo

arXiv
[4]

The New York Times

How manga was translated for america. The New York Times . Accessed: 2024-07-23. Agrim Gupta, Piotr Dollar, and Ross Girshick

work page 2024
[5]

arXiv preprint arXiv:2404.01247

An image speaks a thousand words, but can everyone listen? on translating images for cultural relevance. arXiv preprint arXiv:2404.01247. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Feder- mann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al

arXiv
[6]

Tom Kocmi and Christian Federmann

Preliminary wmt24 ranking of general mt systems and llms.arXiv preprint arXiv:2407.19884. Tom Kocmi and Christian Federmann

arXiv
[7]

arXiv preprint arXiv:2303.16634

G-eval: Nlg evaluation using gpt-4 with better human align- ment. arXiv preprint arXiv:2303.16634. Chi-kiu Lo

arXiv
[9]

https://openai.com/ index/hello-gpt-4o/

Hello gpt-4o. https://openai.com/ index/hello-gpt-4o/. Accessed: 2024-10-14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu

work page 2024
[11]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Katharina Reiss, Hans J. Vermeer, Christiane Nord, and Marina Dudenhöfer

arXiv
[12]

https://www.translateplus.com/blog/ transcreation-examples-that-hit-the-mark/

Transcreation examples that hit the mark. https://www.translateplus.com/blog/ transcreation-examples-that-hit-the-mark/ . [Online; accessed 18-July-2024]. Unicef et al

work page 2024

Show all 12 references

[2009]

Mutatis Mutandis: Revista Latinoamericana de Traducción, 2:209–243

El funcionalismo en la en- señanza de traducción. Mutatis Mutandis: Revista Latinoamericana de Traducción, 2:209–243. Vilde Olaussen. Why Marketing Translation Matters, and How to Get it Right. https://redokun.com/ blog/marketing-translation. [Online; accessed 18-July-2024]. OpenAI

2024
[2020]

In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online

COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin...

2020
[2023]

arXiv preprint arXiv:2302.04166

Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. Gabriel Gianordoli and Robert Ito

arXiv
[2024]

In Pro- ceedings of the 21st International Conference on Spo- ken Language Translation (IWSLT 2024), pages 1–

Find- ings of the iwslt 2024 evaluation campaign. In Pro- ceedings of the 21st International Conference on Spo- ken Language Translation (IWSLT 2024), pages 1–

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

arXiv preprint arXiv:2409.17146

Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146. Mar Díaz-Millón and María Dolores Olvera-Lobo

arXiv

[2] [4]

The New York Times

How manga was translated for america. The New York Times . Accessed: 2024-07-23. Agrim Gupta, Piotr Dollar, and Ross Girshick

work page 2024

[3] [5]

arXiv preprint arXiv:2404.01247

An image speaks a thousand words, but can everyone listen? on translating images for cultural relevance. arXiv preprint arXiv:2404.01247. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Feder- mann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al

arXiv

[4] [6]

Tom Kocmi and Christian Federmann

Preliminary wmt24 ranking of general mt systems and llms.arXiv preprint arXiv:2407.19884. Tom Kocmi and Christian Federmann

arXiv

[5] [7]

arXiv preprint arXiv:2303.16634

G-eval: Nlg evaluation using gpt-4 with better human align- ment. arXiv preprint arXiv:2303.16634. Chi-kiu Lo

arXiv

[6] [9]

https://openai.com/ index/hello-gpt-4o/

Hello gpt-4o. https://openai.com/ index/hello-gpt-4o/. Accessed: 2024-10-14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu

work page 2024

[7] [11]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Katharina Reiss, Hans J. Vermeer, Christiane Nord, and Marina Dudenhöfer

arXiv

[8] [12]

https://www.translateplus.com/blog/ transcreation-examples-that-hit-the-mark/

Transcreation examples that hit the mark. https://www.translateplus.com/blog/ transcreation-examples-that-hit-the-mark/ . [Online; accessed 18-July-2024]. Unicef et al

work page 2024