REVIEW 4 major objections 5 minor 29 references
Pun Intended: Multi-Agent Translation of Wordplay with Contrastive Learning and Phonetic-Semantic Embeddings
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper argues that translating puns works best by recreating the joke in the target language rather than translating literally, and that phonetic-semantic retrieval plus iterative multi-agent evaluation deliver that better than direct…
desk verdict A solid shared-task system paper with an externally validated ranking and a useful negative result about BLEU, undermined mainly by a circular component-evaluation procedure rather than by the one-point first/second gap the stress-test flags. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-part generation pipeline. First, a phonetic-semantic retrieval stage: French phonetic embeddings are trained from IPA pronunciations represented as articulatory-feature bigrams, then concatenated with semantic embeddings; retrieval keeps candidate words whose semantic vector is close to one intended meaning and whose phonetic vector is close to the other, using thresholds $\cos(\mathbf{w}_{\mathrm{sem}}, \mathbf{S}) > 0.75$ and $\cos(\mathbf{w}_{\mathrm{phon}}, \mathbf{P}) > 0.75$. Second, an iterative multi-agent evaluation loop: four LLM judges score each candidate on equivalence, quality, emotion, and authenticity, return textual feedback, and the generation loop refines until the average score reaches 2.0 or five iterations pass. The combination is the core mechanism: explicit retrieval injects phonetically plausible target-language material, then iterative evaluation pushes generation toward functional equivalence rather than literal overlap.
What would settle it
Have several independent native French-speaking raters, blind to which system produced each translation, score a larger sample from the same shared task; if the multi-agent and guided systems do not beat the baseline under that protocol, or if human rankings align with BLEU and BERTScore, the paper's central claim would be refuted.
Extended reading notes
Core claim
The central discovery is that functional equivalence beats lexical correspondence for pun translation: systems that are explicitly pushed to recreate the humor—through phonetically and semantically guided candidate retrieval, or through iterative feedback from multiple specialized LLM judges—produce translations that expert raters judge successful far more often than a baseline that merely generates with a discriminator filter. In the evaluation described in the paper, the multi-agent system was judged to have produced successful wordplay in 37 of 42 sampled translations and the guided system in 36, versus 20 for the baseline, and the systems ranked first and second among 51 entries in the shared task under human evaluation and a pun-location metric. The paper argues this inversion—low lexical-overlap scores but high human scores—shows that BLEU and BERTScore reward the wrong objective for wordplay.
Load-bearing premise
The rankings rest on the authors' own reference annotations of what counts as the pun and its meanings, and on a single native French speaker judging only 42 translations; if that reference standard is biased or that rater is not representative, the claimed ordering of the three systems is not well supported.
Editorial extensions
If this is right
- Wordplay translation systems should be evaluated, and optimized, by whether the translation recreates the joke, not by BLEU or BERTScore, because those metrics reward lexical overlap that successful puns often abandon.
- Retrieval of target-language candidates using combined phonetic-semantic embeddings can steer an LLM toward the second meaning of a pun while keeping the sound close, so building such embeddings for new languages is a direct route to expanding this approach.
- Iterative multi-agent evaluation with specialized LLM judges can improve creative translation without supervised training, suggesting that run-time evaluation is currently a stronger lever than more elaborate generation prompting.
- Since the multi-agent and guided systems scored nearly identically, the extra engineering of explicit linguistic reasoning may be replaceable by simpler iterative evaluation, a hypothesis the paper leaves open.
- The success under human judgment despite low automatic scores implies shared-task leaderboards for humor translation should weight functional-equivalence metrics, or risk ranking systems inversely to their actual quality.
Reading between the lines
- One testable extension is to build phonetic-semantic embeddings for other target languages and measure expert-human success; the paper's method is language-agnostic but only demonstrated for French.
- If functional equivalence is the right objective, similar recreation-based pipelines could apply to idioms, culturally specific humor, or poetry, where a literal translation is also the wrong target.
- The contrastive discriminator in the baseline may become unnecessary if iterative LLM evaluation is enough; an ablation that removes the discriminator while keeping multi-agent refinement would isolate its contribution.
- Because the manual evaluation used a single rater on 42 items, a natural follow-up is a multi-rater, larger-sample human evaluation; if rankings stay stable, the claim that lexical metrics invert human quality would be much stronger.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents three LLM-based pipelines for translating English puns into French: a discriminator-guided baseline, a guided chain-of-thought system that retrieves French candidates with phonetic-semantic embeddings, and a multi-agent system that iteratively evaluates and refines candidate translations. The authors report that the multi-agent and guided systems ranked first and second on the official CLEF JOKER 2025 Task 2 evaluation under the shared-task pun-location and manual-evaluation metrics, despite near-bottom BLEU and BERTScore ranks, and they argue that this supports prioritizing functional equivalence over lexical correspondence for wordplay translation. The paper also reports component evaluations for pun identification, synonym-list translation, and a contrastive discriminator.
Significance. If the headline results hold, the paper is a useful empirical contribution: it demonstrates in a shared-task setting that retrieval-grounded and evaluator-refined LLM pipelines can produce French puns judged successful by a human expert, and it adds evidence that BLEU and BERTScore are poorly suited to creative translation. The public release of code, prompts, and augmented data, and the reliance on an external organizer-provided ranking for the end-to-end claim, are genuine strengths. However, the manual evaluation underpinning the first/second ranking is a single-rater assessment of 42 items, and the paper itself concedes the top-two difference is statistically insignificant; the component evaluations are also tied to author-produced reference annotations that were revised after exposure to LLM predictions. These issues limit the strength of the comparative and explanatory claims until they are addressed with uncertainty quantification and independent annotation.
major comments (4)
- [Abstract; §5.2] The abstract states that the multi-agent and guided systems "ranked first and second, respectively, in the CLEF JOKER 2025 Task 2 competition under expert human evaluation" without qualification, but §5.2 concedes that the two systems' manual-evaluation scores are "within one point of each other, a statistically insignificant difference on such a small sample." Since Table 3 shows the gap is 37/42 versus 36/42 from a single rater, the first/second placement is not evidence that multi-agent evaluation outperforms guided reasoning. Please revise the abstract and conclusions to present the two systems as statistically indistinguishable in manual evaluation, and report a confidence interval or bootstrap result for the one-point difference.
- [§3.1; Table 1] The reference annotations for pun location, type, and intended meanings were "produced collaboratively by the authors" and then revised after "compar[ing] them with LLM predictions, manually reviewing any disagreements." Using these post-hoc revised annotations as the gold standard for Table 1 can inflate agreement, since the reference was adjusted in light of the very systems being scored. The paper should quantify how many annotations changed during the review, report inter-annotator agreement on an independent sample, or use annotations created without exposure to model outputs; without this, the component-level claims in Table 1 are not a clean evaluation.
- [§4.2; Table 3] The manual evaluation that supports the main ranking consists of 42 examples judged by a single native French speaker, and the paper reports no inter-rater reliability, no per-item scores, and no significance test among the three systems. Because the headline claim depends on these counts, please report the sampling procedure for the 42 examples, state whether all three systems were judged on the same items, and provide a paired significance test (e.g., McNemar or bootstrap confidence intervals) for 37/42 versus 36/42 versus 20/42; a second rater on a subset would also help establish that the baseline gap is robust rather than a single-rater artifact.
- [§5.1] The paper claims that its "results provide empirical support" for functional equivalence over lexical correspondence, but the three systems differ in architecture, prompting, and retrieval, so the translation objective is not isolated. The higher manual scores for the advanced systems could stem from iterative refinement, better retrieval, or more detailed prompts rather than from the functional-equivalence objective per se. Support the claim with an ablation that varies only the objective (e.g., a literal-translation prompt with the same multi-agent loop), or soften the causal attribution to a hypothesis consistent with the shared-task outcome.
minor comments (5)
- [§3.3, Eq. (1)] In Equation (1), the notation F(P_a1, P_a2) is not defined; please state explicitly that it denotes the set of articulatory-feature bigrams for a phoneme pair, and define the Jaccard similarity over those sets.
- [§3.3, Eq. (2)] The thresholds in Equation (2) (top-2 candidates, cosine greater than 0.75) are justified only as "determined empirically"; please report the range of values explored and the sensitivity of the guided system to these two hyperparameters.
- [§4.2] The "shared-task pun location metric" is not defined in the paper; please explain how the 1,682 evaluated translations relate to the 376 English puns and why the top-ranked system achieves only 9.27% on this metric, since a reader cannot otherwise interpret the rank.
- [Table 3] Please state in the text whether the manual evaluation was performed by the CLEF JOKER organizers or by the authors, since Table 3 is labeled as "official evaluation results" but the text describes the rater's background without indicating who employed her.
- [References] References [2] and [3] appear to be duplicate citations of the same Attardo and Raskin (1991) paper with slightly different page ranges; please merge them or clarify the distinction.
Circularity Check
Minor circularity in the pun-identification reference standard; the central ranking is measured against an external shared-task benchmark and is not circular.
-
self definitional
[Section 3.1 (Data and Resources) and Section 4.1 (Table 1)]
"We also used the CLEF JOKER 2023 Pun Location and Interpretation dataset [10], supplementing it with manual annotations of the pun word, pun type, intended meanings, and supporting context for each English example. The annotations were produced collaboratively by the authors. We then compared them with LLM predictions, manually reviewing any disagreements before finalizing the annotations. These annotations served as the reference standard for pun identification."
The reference standard used to compute Table 1 accuracy was finalized only after comparison with the LLM predictions that are then scored against it. Any disagreements resolved in favor of the LLM predictions are thus baked into the gold standard, so the reported pun-identification accuracy partly measures the models' own outputs rather than an independent human truth. This makes the component validation in Table 1 circular by construction, though it does not affect the externally evaluated end-to-end ranking.
full rationale
The paper's central claim, that the multi-agent and guided systems ranked first and second in CLEF JOKER 2025 Task 2, is measured against an external shared-task benchmark using a manual evaluation by a native French speaker and the official pun-location metric. That ranking is not derived from the systems' own outputs or from a self-citation chain, so it is not circular. The paper itself notes in §5.2 that the top two systems 'achieved virtually identical human evaluation scores within one point of each other, a statistically insignificant difference on such a small sample,' which is a reliability limitation rather than circularity. The only genuine circular step is the pun-identification reference standard in §3.1: the authors wrote the annotations, compared them with LLM predictions, reviewed disagreements, and finalized the annotations that then served as the gold standard for Table 1. This contaminates the component accuracy numbers, but those numbers are not load-bearing for the externally determined ranking. No self-citations are used as load-bearing evidence, and no uniqueness theorem or imported ansatz is invoked. The discriminator's synthetic negative examples are used for training, while its reported accuracy is measured on a manually labeled 450-example set, so that evaluation is independent. Overall, the central derivation is self-contained against the external benchmark, and the circularity found is minor and localized.
Assumptions & free parameters
free parameters (4)
- cosine similarity threshold =
0.75
- top-k retrieved candidates =
2
- max refinement iterations =
5
- few-shot discriminator example counts =
25 positive and 25 negative
assumptions (3)
- ad hoc to paper Reference annotations of pun word, pun type, and intended meanings produced by the authors are correct, even after being revised in light of LLM predictions.
- domain assumption Cosine similarity in the concatenated phonetic-semantic embedding space is a valid proxy for finding French words that are semantically related to one meaning and phonetically related to the other.
- domain assumption A single native French speaker's judgment of whether meaning and wordplay are preserved is a reliable measure of translation quality.
Cite this review
Pith. "Pith review of Pun Intended: Multi-Agent Translation of Wordplay with Contrastive Learning and Phonetic-Semantic Embeddings." pith.science (2026). https://pith.science/paper/3UXX7SKB
@misc{pith2026260804311,
author = {Pith},
title = {Pith review of: Pun Intended: Multi-Agent Translation of Wordplay with Contrastive Learning and Phonetic-Semantic Embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/3UXX7SKB}},
note = {Machine review of arXiv:2608.04311}
}
read the original abstract
Translating wordplay across languages has long challenged both professional translators and machine translation systems. We investigate three approaches to translating puns from English to French by combining large language models with linguistic constraints for wordplay generation. Our baseline uses a large language model with feedback from a discriminator prompted with positive and negative French examples. Our guided reasoning pipeline uses combined phonetic-semantic embeddings to retrieve lexical candidates for wordplay generation. Finally, our multi-agent framework iteratively evaluates and regenerates candidate translations using specialized feedback. Moving beyond literal translation, our objective is to preserve the linguistic creativity, ambiguity, and humor of the source-text wordplay rather than simply reproduce its vocabulary. The multi-agent and guided chain-of-thought systems ranked first and second, respectively, in the CLEF JOKER 2025 Task 2 competition under expert human evaluation, despite only modest improvements in BLEU and BERTScore. These findings suggest that both explicit phonetic-semantic guidance and iterative multi-agent evaluation can improve LLM-based wordplay translation relative to direct discriminator-guided generation, particularly when balancing semantic fidelity, phonetic similarity, and natural target-language expression
Figures
Reference graph
Works this paper leans on
-
[1]
In: The Routledge Handbook of Language and Humor, pp
Aarons, D.: Puns and tacit linguistic knowledge. In: The Routledge Handbook of Language and Humor, pp. 80–94. Routledge (2017)
work page 2017
-
[2]
Humor: International Journal of Humor Research4(3-4), 293–347 (1991)
Attardo, S., Raskin, V.: Script theory revis(it)ed: Joke similarity and joke repre- sentation model. Humor: International Journal of Humor Research4(3-4), 293–347 (1991)
work page 1991
-
[3]
Humor4(3–4), 293–348 (1991).https://doi.org/10.1515/humr
Attardo, S., Raskin, V.: Script theory revis(it)ed: Joke similarity and joke represen- tation model. Humor4(3–4), 293–348 (1991).https://doi.org/10.1515/humr. 1991.4.3-4.293
-
[4]
Baziotis, C., Mathur, P., Hasler, E.: Automatic evaluation and analysis of id- ioms in neural machine translation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pp. 3682–
-
[5]
Delabastita, D.: Focus on the pun. Target. International Journal of Transla- tion Studies6(2), 223–243 (1994).https://doi.org/https://doi.org/10.1075/ target.6.2.07del
work page 1994
-
[6]
The Translator2(2), 127–139 (1996).https://doi
Delabastita, D.: Introduction. The Translator2(2), 127–139 (1996).https://doi. org/10.1080/13556509.1996.10798970
-
[7]
The Translator2(2), 127–139 (1996)
Delabastita, D.: Introduction to the special issue on wordplay and translation. The Translator2(2), 127–139 (1996)
work page 1996
-
[8]
Ermakova, L., Campos, R., Bosser, A.G., Miller, T.: Overview of the CLEF 2025 JOKER lab: Humour in machine. In: Carrillo-de Albornoz, J., Gonzalo, J., Plaza, L., García Seco de Herrera, A., Mothe, J., Piroi, F., Rosso, P., Spina, D., Faggioli, G., Ferro, N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. pp. 315–337. Lecture ...
Show all 29 references
-
[9]
In: Faggioli, G., Ferro, N., Rosso, P., Spina, D
Ermakova, L., Campos, R., Bosser, A.G., Miller, T.: Overview of the CLEF 2025 JOKER task 2: Wordplay translation from English into French. In: Faggioli, G., Ferro, N., Rosso, P., Spina, D. (eds.) Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2025). CEU...
2025
-
[10]
In: Arampatzis, A., Kanoulas, E., Tsikrika, T., Vrochidis, S., Giachanou, A., Li, D., Aliannejadi, M., Vlachos, M., Faggioli, G., Ferro, N
Ermakova, L., Miller, T., Bosser, A.G., Palma Preciado, V.M., Sidorov, G., Ja- towt, A.: Overview of JOKER–CLEF-2023 track on automatic wordplay anal- ysis. In: Arampatzis, A., Kanoulas, E., Tsikrika, T., Vrochidis, S., Giachanou, A., Li, D., Aliannejadi, M., Vlachos, M., Fagg...
2023 doi
-
[11]
Grave, E., Bojanowski, P., Gupta, P., Joulin, A., Mikolov, T.: Learning word vec- torsfor157languages.In:ProceedingsoftheInternationalConferenceonLanguage Resources and Evaluation (LREC 2018) (2018)
2018
- [12]
-
[13]
He, H., Peng, N., Liang, P.: Pun generation with surprise. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa- pers). pp. 173–181. Association for Com...
2019 doi
-
[14]
In: Handbook of Language and Humor
Hempelmann, C.F., Miller, T.: Puns. In: Handbook of Language and Humor. De Gruyter (2022)
2022
-
[15]
Perspectives: Studies in Translatology 19(1), 59–70 (2011)
Low, P.A.: Translating jokes and puns. Perspectives: Studies in Translatology 19(1), 59–70 (2011)
2011
-
[16]
In: Proceedings of the Human-Informed Translation and Interpreting Technology Workshop (HiT-IT 2019)
Miller, T.: The punster’s amanuensis: The proper place of humans and ma- chines in the translation of wordplay. In: Proceedings of the Human-Informed Translation and Interpreting Technology Workshop (HiT-IT 2019). pp. 57–65. In- coma Ltd., Shoumen, Bulgaria, Varna, Bulgaria (s...
2019
-
[17]
Mittal, A., Tian, Y., Peng, N.: AmbiPun: Generating humorous puns with ambigu- ouscontext.In:Proceedingsofthe2022ConferenceoftheNorthAmericanChapter of the Association for Computational Linguistics: Human Language Technologies. pp. 1053–1062. Association for Computational Ling...
2022 doi
-
[18]
In: Matsumoto, Y., Prasad, R
Mortensen, D.R., Littell, P., Bharadwaj, A., Goyal, K., Dyer, C., Levin, L.: Pan- Phon: A resource for mapping IPA segments to articulatory feature vectors. In: Matsumoto, Y., Prasad, R. (eds.) Proceedings of COLING 2016, the 26th Interna- tional Conference on Computational Li...
2016
-
[19]
Shanahan, M., McDonell, K., Reynolds, L.: Role-play with large language models (2023)
2023
-
[20]
Sharma, R., Dhawan, K., Pailla, B.: Phonetic word embeddings (2021)
2021
-
[21]
arXiv preprint arXiv:2210.13513 (2022)
Sun, J., Narayan-Chen, A., Oraby, S., Cervone, A., Chung, T., Huang, J., Liu, Y., Peng, N.: Expunations: Augmenting puns with keywords and explanations. arXiv preprint arXiv:2210.13513 (2022)
2022 arXiv
-
[22]
Perspec- tives: Studies in Translatology5(1), 92–102 (1997)
Veisbergs, A.: The contextual use of idioms, wordplay and translation. Perspec- tives: Studies in Translatology5(1), 92–102 (1997)
1997
-
[23]
Wang, H., Zhao, Y., Li, D., Wang, X., Liu, G., Lan, X., Wang, H.: Innovative think- ing, infinite humor: Humor research of large language models through structured thought leaps (2024)
2024
-
[24]
Wang, S., Zhang, G., Wu, H., Loakman, T., Huang, W., Lin, C.: MMTE: Corpus and metrics for evaluating machine translation quality of metaphorical language (2024)
2024
-
[25]
Xu, Z., Yuan, S., Chen, L., Yang, D.: A Good Pun Is Its Own Reword: Can large language models understand puns? In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 11766–11782. Association for Computational Linguistics, Miami, Florida...
2024 doi
-
[26]
Journal of Pragmatics35(9), 1295– 1331 (2003)
Yus, F.: Humor and the search for relevance. Journal of Pragmatics35(9), 1295– 1331 (2003)
2003
-
[27]
In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lan- guage Resources and Evaluation (LREC-COLING 2024)
Zeng, J., Yang, L., Kang, J., Diao, Y., Yang, Z., Lin, H.: Barking up the Right Tree, a GAN-based pun generation model through semantic pruning. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lan- guage Resources and Evaluation (LREC-C...
2024 doi
-
[28]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Zhong, S., Huang, Z., Gao, S., Wen, W., Lin, L., Zitnik, M., Zhou, P.: Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). ...
2024
-
[3700]
https://doi.org/10.18653/v1/2023.eacl-main.267
Association for Computational Linguistics, Dubrovnik, Croatia (may 2023). https://doi.org/10.18653/v1/2023.eacl-main.267
2023 doi
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.