Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Intrinsic semantic scores do not predict—and can invert—how well a sentence embedding performs as a machine-translation evaluator.

desk verdict A genuinely useful negative result about Czech sentence embeddings, but the headline inversion is partially confounded with embedding size and the statistical support is thin. read the letter →

arxiv 2506.20203 v1 pith:ZEAFJGLU submitted 2025-06-25 cs.CL

Petra Barančíková , Ondřej Bojar This is my paper
classification cs.CL
keywords sentenceembeddingsintrinsicevaluationextrinsicmachinetranslationqualityestimationCzechlanguageCOMETsemantictextualsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether performance on intrinsic semantic benchmarks can predict how useful a sentence-embedding model will be for machine translation evaluation after fine-tuning. Using English-to-Czech data, the authors compare Czech-specific and multilingual embeddings on Costra, a Czech sentence-transformation benchmark, and on semantic textual similarity tests, then fine-tune the same embeddings inside COMET-style estimators for translation evaluation and reference-free quality estimation. They report a clear inversion: SimCSE, the best model in the intrinsic tests, ranks among the worst evaluation metrics, while XLM-R and FERNET, which perform poorly intrinsically, become the best. The paper concludes that intrinsic metrics of this kind cannot reliably forecast a model's usefulness for MT evaluation and argues for intrinsic benchmarks that capture the semantics translation scoring actually needs.

What carries the argument

The central mechanism is the COMET-style dual-encoder estimator: source, reference, and hypothesis are each encoded by the candidate embedding model, pooled with a sparsemax-based mixed-layer pooling (a weighted combination of encoder layers), passed through two hidden layers (3072 and 2048 units depending on the task, then 1024), and scored against human judgments after fine-tuning with a frozen-then-unfrozen encoder and layer-wise decay. This identical training recipe makes the intrinsic-to-extrinsic comparison fair and is what turns each embedding space into a machine translation evaluation or quality estimation metric. On the intrinsic side, the load-bearing objects are Costra, a Czech sentence-transformation benchmark scoring six categories, and an STS evaluation tool that searches over pooling strategies and similarity metrics to assign each encoder its best score.

What would settle it

Fine-tune the same embedding models at matched dimensionality, for example by projecting XLM-R and FERNET down to 256 dimensions or enlarging SimCSE and RetroMAE to 768, and re-run the WMT21 and WMT22 system-level and segment-level correlations with human scores; if the intrinsic-to-extrinsic inversion disappears once sizes are equal, the paper's central disconnect claim is falsified, and if it persists, the size confound is ruled out.

Watch

Extended reading notes

Core claim

The paper's central finding is that, for English-to-Czech machine translation evaluation, the ranking of sentence embeddings flips after fine-tuning: intrinsic semantic quality and fine-tuned evaluation quality are not merely uncorrelated, but can be inversely related. On Costra and semantic textual similarity benchmarks, SimCSE is the strongest encoder and XLM-R among the weakest, yet after the same COMET-style fine-tuning, XLM-R and FERNET produce the best machine translation evaluation and quality estimation metrics while SimCSE and RetroMAE are among the worst. The authors interpret this as evidence that these intrinsic tests measure a different, more general notion of semantic similarity than the fine-grained, translation-error-sensitive distinction an evaluation estimator must learn, and that an apparently over-smoothed embedding space can be reshaped by fine-tuning into a discriminative task-specific representation. They further note that embedding dimensionality tracks the extrinsic outcome, with the smallest models being the worst estimators, and that Czech-specific and multilingual models perform comparably.

Load-bearing premise

The load-bearing premise is that the extrinsic performance differences reflect what the embeddings represent rather than simply how many dimensions they have; if the 256-dimensional models fail mainly because they are small, the inversion is a size effect, not evidence against intrinsic semantics.

Editorial extensions

If this is right

  • Intrinsic benchmarks such as Costra and STS should not be used to select an encoder for fine-tuned machine translation evaluation, since the best intrinsic model can be one of the worst downstream metrics.
  • Fine-tuning can convert an apparently over-smoothed embedding space, whose raw vectors are nearly constant across random Czech sentences, into a highly discriminative translation-quality metric.
  • Embedding dimensionality, not language specificity, is the factor that tracks fine-tuned evaluation success: the smallest models underperform whether they are Czech-specific or multilingual.
  • Segment-level machine translation evaluation and quality estimation scores are strongly correlated and stable across WMT21 and WMT22, so reference-free evaluation can serve as a reliable proxy for reference-based evaluation on Czech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's own size confound is to fine-tune the models at matched dimensionality, for example by projecting XLM-R and FERNET down to 256 dimensions or enlarging SimCSE to 768; if the inversion disappears, the conclusion should be restated as a capacity effect rather than a semantic-relevance effect.
  • The negative 2022 correlation between the Costra-minus score and system-level quality estimation hints that a smooth or less discriminative prior may be easier for fine-tuning to reshape; interpolating between random and pretrained initialization would probe this directly.
  • If the pattern extends beyond Czech, intrinsic benchmark design should shift toward task-specific, error-sensitive triplet orderings, the paper's 'operationalizable semantics', instead of generic semantic similarity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper compares Czech-specific and multilingual sentence embedding models through intrinsic and extrinsic evaluation. Intrinsic evaluation uses the Costra dataset and three STS benchmarks; extrinsic evaluation fine-tunes the embeddings as COMET-style estimators for machine translation evaluation (MTE with references) and quality estimation (QE without references), using WMT17-19 for training, WMT20 for validation, and WMT21/22 for testing. The central finding is an inversion: SimCSE dominates the intrinsic benchmarks but ranks among the worst fine-tuned COMET estimators, while XLM-R and FERNET, which are poor intrinsically, become the best MTE/QE metrics. The paper concludes that intrinsic semantic evaluation cannot reliably predict downstream MT evaluation performance.

Significance. If the reported inversion is robust, the paper is a useful, falsifiable empirical contribution to the sentence-embedding evaluation literature. Its protocol is clean: the train/validation/test split is held out, 18 COMET estimators are trained with a fixed recipe, and both system-level and segment-level correlations are reported alongside traditional string-matching metrics. The paper also gives an explicit mechanism for the intrinsic failure of XLM-R (over-smoothed embedding space with near-perfect cosine similarity even for shuffled sentence pairs). The main weakness is that the headline inversion is partially collinear with embedding dimensionality, and the paper does not provide size-controlled evidence for the claim that semantic relevance is unhelpful for MT evaluation. The absence of confidence intervals or significance tests on the small model/system-level correlation sets further weakens the central claim in its current form.

major comments (3)
  1. [Section 6, Table 4] The central inversion is confounded with embedding size. The two 256-d models, SimCSE and RetroMAE, are the worst trained estimators in most extrinsic columns (e.g., CE_MTE(SimCSE)=0.74 and CE_MTE(RetroMAE)=0.82 on WMT22 system-level MTE, and CE_QE(SimCSE)=-0.92 on WMT22 QE), whereas the best extrinsic models are 768-1024-d. The paper acknowledges this in Section 6 but does not test it. A size-controlled analysis is needed, for example within-size comparisons among 768-d models, an upscaled version of SimCSE or RetroMAE, or partial correlations controlling for dimensionality. Without such an analysis, the data support only the weaker claim that larger embeddings fine-tune better for COMET, not that semantic relevance is unhelpful for MT evaluation.
  2. [Figure 1, Table 4] Correlations are computed over small sets of points and are reported without confidence intervals or significance tests. Figure 1 is based on 8-9 embedding models, and Table 4's system-level correlations are computed over a small number of WMT systems in each year. The negative correlations that carry the headline (e.g., Costra- vs. system QE 2022 = -0.52) may be unstable. Bootstrap confidence intervals or permutation tests should be provided before concluding that intrinsic and extrinsic rankings are reliably disconnected.
  3. [Section 4.1, Costra-] The intrinsic score used in the paper's main comparisons is a post hoc modification of Costra: after observing that pretrained models underperform random vectors on Basic and Modality, the authors exclude these categories and define Costra- as the mean of the remaining four categories. The paper is transparent about this decision, but it is data-dependent and affects the intrinsic-vs-extrinsic correlations in Figure 1 and the inversion narrative. Please report sensitivity of these correlations to using the full Costra score instead of Costra-, and discuss how the conclusions would change if Basic and Modality are included.
minor comments (5)
  1. [Section 5.1] There is a typo: 'WTM21' should be 'WMT21'.
  2. [Section 4.2] The STS evaluation tool is described as selecting the highest average score across similarity metrics and pooling strategies; please report which combination was actually selected for each embedding model, since this affects the comparability of the resulting numbers.
  3. [Section 6] The sentence 'The size of the embeddings seem not to matter in the intrinsic tasks--the top 3 best performing embeddings (SimCSE, LaBSE and mE5) are small, base and large, respectively' mixes model-size categories with embedding dimensionalities; please clarify whether 'base' refers to model capacity or hidden size.
  4. [Table 1] The table reports random vectors with dimension 256 as a separate baseline from random BERT (768-d). It would help to state explicitly why both baselines are included and whether the random-vector result is dimension-controlled.
  5. [Section 5.3] The notation CEMTE(X) and CEQE(X) is used to refer to fine-tuned estimators, but the text says 'for a given embeddings X'; please clean up the phrasing so that X is defined as an embedding model rather than a set of vectors.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the intrinsic–extrinsic inversion is an empirical held-out result, not a consequence of how the inputs were defined.

full rationale

The paper's central claim — that intrinsic semantic metrics cannot reliably predict MT evaluation performance — rests on an extrinsic evaluation that is genuinely independent of the intrinsic measurements. COMET estimators are fine-tuned on WMT17–19, validated on WMT20, and tested on held-out WMT21 and WMT22 test sets, so the headline rankings are not fitted to the conclusion. The self-citations (Costra, Barančíková and Bojar 2020; the Czech STS evaluation tool, Bednář et al. 2024) are used as pre-existing benchmark resources, not as authoritative support for the disconnect claim; the conclusion does not reduce to those citations. The post-hoc introduction of the Costra– score is a data-dependent redefinition of an intrinsic metric, but it does not force the extrinsic ranking, and the embedding-size confound acknowledged in Section 6 is a validity concern, not a circularity: no prediction is definitionally equivalent to an input, no fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from the authors' prior work. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's contribution is empirical; its main assumptions are domain-level (reliability of WMT human judgments, validity of Costra annotations, appropriateness of the COMET task). The only hand-chosen element introduced in this paper is the Costra scissors subset, defined after results were observed. COMET hyperparameters are inherited from prior work and applied uniformly, so they are not fitted to the central claim.

free parameters (3)
  • Costra scissors subset selection = excludes Basic and Modality categories (4 of 6)
    Defined in Section 4.1 after observing random vectors beat all models in the excluded categories; changes the intrinsic ranking.
  • COMET hidden layer sizes = 3072/2048 and 1024
    Hand-chosen from Rei et al. (2020), applied uniformly; not tuned per model.
  • COMET training hyperparameters = lr 1.5e-5/1e-6, decay 0.95, freeze 0.3 epochs, 5 epochs, dropout 0.1
    Default settings from Rei et al. (2020); applied identically across models, so fair, but fixed choices may interact with model capacity.
assumptions (5)
  • domain assumption WMT Direct Assessment human scores, z-normalized, are a reliable ground truth for translation quality.
    Sections 5.1-5.4 use DA scores as labels for fine-tuning and as the reference for all reported correlations.
  • domain assumption The Costra dataset annotations correctly capture the intended semantic properties (paraphrase, modality, time, style, generalization, opposite).
    Section 4.1 relies on Costra triplets; the surprising result that random vectors win two categories suggests these categories may not behave as intended.
  • domain assumption Selecting the highest STS score across pooling strategies and similarity metrics per model is a fair intrinsic measure.
    Section 4.2 uses the Bednar et al. (2024) tool, which chooses the maximum over configurations for each model.
  • domain assumption The COMET dual-encoder fine-tuning task is a valid extrinsic probe of sentence embedding quality for MT evaluation.
    Sections 5.3-5.4 assume MTE/QE performance reflects the usefulness of the embedding space.
  • ad hoc to paper The observed inversion is attributed to the embedding models' representational properties rather than to model size, although size is acknowledged as a confound.
    Section 6 lists size as an alternative explanation but does not test it; the paper's preferred hypotheses (over-smoothed vs. task-specific spaces) rest on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation." pith.science (2026). https://pith.science/paper/ZEAFJGLU

@misc{pith2026250620203,
  author       = {Pith},
  title        = {Pith review of: Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZEAFJGLU}},
  note         = {Machine review of arXiv:2506.20203}
}
read the original abstract

In this paper, we compare Czech-specific and multilingual sentence embedding models through intrinsic and extrinsic evaluation paradigms. For intrinsic evaluation, we employ Costra, a complex sentence transformation dataset, and several Semantic Textual Similarity (STS) benchmarks to assess the ability of the embeddings to capture linguistic phenomena such as semantic similarity, temporal aspects, and stylistic variations. In the extrinsic evaluation, we fine-tune each embedding model using COMET-based metrics for machine translation evaluation. Our experiments reveal an interesting disconnect: models that excel in intrinsic semantic similarity tests do not consistently yield superior performance on downstream translation evaluation tasks. Conversely, models with seemingly over-smoothed embedding spaces can, through fine-tuning, achieve excellent results. These findings highlight the complex relationship between semantic property probes and downstream task, emphasizing the need for more research into 'operationalizable semantics' in sentence embeddings, or more in-depth downstream tasks datasets (here translation evaluation)

Figures

Figures reproduced from arXiv: 2506.20203 by the authors.

Figure 1
Figure 1. Correlation heatmaps for different method of embedding evaluations. The heatmaps are intentionally kept [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ond r ej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espa \ n a-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, ...

    work page 2021

  4. [4]

    Amir Bakarov. 2018. http://dblp.uni-trier.de/db/journals/corr/corr1801.html#abs-1801-09536 A S urvey of W ord E mbeddings E valuation M ethods. CoRR, abs/1801.09536

    arXiv 2018

  5. [5]

    Petra Baran c \' kov \'a and Ond r ej Bojar. 2020. https://aclanthology.org/2020.lrec-1.434/ COSTRA 1.0: A D ataset of C omplex S entence T ransformations . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3535--3541, Marseille, France. European Language Resources Association

    work page 2020

  6. [6]

    Lo \"i c Barrault, Magdalena Biesialska, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljube s i \'c , Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 20...

    work page 2020

  7. [7]

    Lo \"i c Barrault, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias M \"u ller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. https://doi.org/10.18653/v1/W19-5301 Findings of the 2019 C onference on M achine T ranslation ( WMT...

    doi:10.18653/v1/w19-5301 2019

  8. [8]

    Ji r \' Bedn \'a r , Jakub N \'a plava, Petra Baran c \' kov \'a , and Ond r ej Lisick \`y . 2024. S ome L ike I t S mall: C zech S emantic E mbedding M odels for I ndustry A pplications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22734--22742

    work page 2024

Show all 41 references
  1. [9]

    Ond r ej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno - Yepes, Philipp Koehn, and Julia Kreutzer, editors. 2017. https://aclanthology.org/volumes/W17-47/ Proceedings of the Second Conference on Machine...

    2017

  2. [10]

    Ond r ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. https://doi.org/10.18653/v1/W18-6401 Findings of the 2018 C onference on M achine T ranslation ( WMT 18) . In Proceedings of the Third Conferen...

    2018 doi

  3. [11]

    Ond r ej C \'i fka and Ond r ej Bojar. 2018. https://doi.org/10.18653/v1/P18-1126 Are BLEU and M eaning R epresentation in O pposition? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1362--1371, Melbour...

    2018 doi

  4. [12]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1911.02116 Unsupervised C ross-lingual R epresentation L earning at S cale...

    2019 arXiv

  5. [13]

    Michael Denkowski and Alon Lavie. 2010. https://aclanthology.org/W10-1751/ METEOR - NEXT and the METEOR P araphrase T ables: I mproved E valuation S upport for F ive T arget L anguages . In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and M etrics...

    2010

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : P re-training of D eep B idirectional T ransformers for L anguage U nderstanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the As...

    2019 doi

  7. [15]

    Steffen Eger, Andreas R \"u ckl \'e , and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/W19-4308 Pitfalls in the E valuation of S entence E mbeddings . In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 55--60, Florence, Italy. Ass...

    2019 doi

  8. [16]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.62 Language-agnostic BERT S entence E mbedding . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2022 doi

  9. [17]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2/ Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics...

    2022

  10. [18]

    Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. https://aclanthology.org/W13-2305/ C ontinuous M easurement S cales in H uman E valuation of M achine T ranslation . In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Disc...

    2013

  11. [19]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Nov \'a k, Ma...

    2022

  12. [20]

    Jan Lehe c ka and Jan S vec. 2021. https://doi.org/10.1007/978-3-030-89579-2_3 Comparison of C zech T ransformers on T ext C lassification T asks . In Statistical Language and Speech Processing, pages 27--37, Cham. Springer International Publishing

    2021 doi

  13. [21]

    Jindrich Libovick \' y and Pranava Madhyastha. 2019. https://arxiv.org/abs/1908.11125 Probing R epresentations L earned by M ultimodal R ecurrent and T ransformer M odels . CoRR, abs/1908.11125

    2019 arXiv

  14. [22]

    Ibraheem Muhammad Moosa, Rui Zhang, and Wenpeng Yin. 2024. https://openreview.net/forum?id=Rry1SeSOQL MT - R anker: R eference-free machine translation evaluation by inter-system ranking . In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a M ethod for A utomatic E valuation of M achine T ranslation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2002

  16. [24]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  17. [25]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A C all for C larity in R eporting BLEU S cores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  18. [26]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022 a . https://aclanthology.org/2022.wmt-1.52/ COMET -22: Unbabel- IST 2022 S ubmission for the M etrics S hared T ask ...

    2022

  19. [27]

    Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andr \'e F. T. Martins, and Alon Lavie. 2021. https://aclanthology.org/2021.wmt-1.111/ Are R eferences R eally N eeded? U nbabel- IST 2021 S ubmission for the M etrics S...

    2021

  20. [28]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A N eural F ramework for MT E valuation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--270...

    2020 doi

  21. [29]

    Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G

    Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andr \'e F. T. Martins. 2022 b . https://aclanthology.org/2022.wmt-1.60/ C omet K iwi: IST -...

    2022

  22. [30]

    Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. https://aclanthology.org/C16-1009/ Task- O riented I ntrinsic E valuation of S emantic T extual S imilarity . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers ,...

    2016

  23. [31]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT: Learning Robust Metrics for Text Generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Asso...

    2020 doi

  24. [32]

    Jakub Sido, Ond r ej Pra z \'a k, Pavel P r ib \'a n , Jan Pa s ek, Michal Sej \'a k, and Miloslav Konop \'i k. 2021. https://aclanthology.org/2021.ranlp-1.149/ Czert -- C zech BERT -like M odel for L anguage R epresentation . In Proceedings of the International Conference on ...

    2021

  25. [33]

    Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. https://aclanthology.org/2006.amta-papers.25/ A Study of Translation Edit Rate with Targeted Human Annotation . In Proceedings of the 7th Conference of the Association for Machine Translation...

    2006

  26. [34]

    Milan Straka, Jakub N \'a plava, Jana Strakov \'a , and David Samuel. 2021. RobeCzech: Czech RoBERTa, a Monolingual Contextualized Language Representation Model . In Text, Speech, and Dialogue, pages 197--209, Cham. Springer International Publishing

    2021

  27. [35]

    Jay Kuo, and Haizhou Li

    Bin Wang, C.-C. Jay Kuo, and Haizhou Li. 2022. https://doi.org/10.18653/v1/2022.acl-long.419 Just Rank: Rethinking Evaluation with Word and Sentence Similarities . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2022 doi

  28. [36]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 Text Embeddings: A Technical Report . arXiv preprint arXiv:2402.05672

    2024 arXiv

  29. [37]

    Liu Xiaoming, Lin Hongyu, Han Xianpei, and Sun Le. 2023. https://aclanthology.org/2023.ccl-1.69/ S ent B ench: C omprehensive E valuation of S elf- S upervised S entence R epresentation with B enchmark C onstruction . In Proceedings of the 22nd Chinese National Conference on C...

    2023

  30. [38]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with BERT . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April...

    2020

  31. [39]

    Vitalii Zhelezniak, Aleksandar Savkov, April Shen, and Nils Hammerla. 2019. https://doi.org/10.18653/v1/N19-1100 Correlation Coefficients and Semantic Textual Similarity . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational...

    2019 doi

  32. [40]

    , " * write output.state after.block = add.period write

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.