Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

Alexandre Alcoforado; Anna Helena Reali Costa; Lucas Hideki Takeuchi Okamura

REVIEW 3 major objections 4 minor 22 references

Portuguese sentence-encoder performance is strongly task-dependent, so multilingual rankings cannot substitute for Portuguese-specific evaluation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:41 UTC pith:CS4TPT2Q

load-bearing objection Useful Portuguese MTEB slice with honest limitations; the central rank-transfer claim needs variance estimates before it can be stated as strongly as it is. the 3 major comments →

arxiv 2607.04071 v2 pith:CS4TPT2Q submitted 2026-07-05 cs.CL cs.AIcs.LG

Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

Lucas Hideki Takeuchi Okamura , Alexandre Alcoforado , Anna Helena Reali Costa This is my paper

classification cs.CL cs.AIcs.LG

keywords text embeddingsPortuguesesentence encodersbenchmarkMTEB-PTMatryoshka Representation Learningretrievalsemantic textual similarity

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MTEB-PT, a Portuguese evaluation slice of a massive multilingual embedding benchmark, covering 14 datasets across STS, classification, retrieval, and reranking. It evaluates 17 open- and closed-source embedding models under a single protocol and shows that Portuguese rankings diverge from multilingual averages, especially on retrieval and reranking, where long-context and search-oriented models pull ahead. A second finding is that Portuguese contrastive fine-tuning with Matryoshka Representation Learning lifts STS scores sharply — 86.1 vs 63.1 for one backbone — and also transfers to retrieval, while remaining competitive under dimensional truncation. The authors conclude that multilingual benchmarks provide useful priors but cannot replace language-specific, task-diverse evaluation.

Core claim

The central discovery is that Portuguese embedding quality is task-dependent to a degree that multilingual aggregate rankings miss. On MTEB-PT, the same models reorder substantially across task families: STS is led by Portuguese-adapted and sentence-pair-trained encoders; classification shows a small spread; retrieval and reranking favor closed-source systems and multilingual encoders with long-context support. Comparing MMTEB and MTEB-PT ranks shows the largest reordering on search-oriented tasks, with, for example, text-embedding-3-large dropping and multilingual-e5-base rising in reranking. The paper also demonstrates that fine-tuning three backbones on Portuguese contrastive data produce

What carries the argument

MTEB-PT itself: a curated Portuguese-language subset of existing MMTEB datasets spanning four task families (3 STS, 5 classification, 3 retrieval, 3 reranking), evaluated under the official MTEB implementations with per-model input formatting. The load-bearing device is the task-family decomposition of scores (macro-averaged within each family) plus the rank-change comparison against MMTEB, which isolates whether multilingual standing transfers. The fine-tuning experiments use CoSENT and Multiple Negatives Ranking Loss wrapped with Matryoshka Representation Learning (MRL), letting a single model serve at multiple embedding dimensions.

Load-bearing premise

The load-bearing premise is that the measured Portuguese subset scores are reliable enough to support the rank-order claims, but several subsets are small (down to 200 queries), all scores are single-run point estimates with no error bars or seeds, and closed-source retrieval numbers come from a 20% subsample of WebFaqRetrieval, so the rank reorderings in Table 4 could be partly measurement noise.

What would settle it

Re-run the 17 models on MTEB-PT with multiple random seeds, full-size subsets, and bootstrap confidence intervals on each per-task score; if the rank changes between MMTEB and MTEB-PT in Table 4 fall within overlapping intervals (i.e., are not statistically significant), the claim that multilingual rankings do not reliably transfer to Portuguese would fail. A complementary check is evaluating closed-source models on the full WebFaqRetrieval set, since their current retrieval averages rest on a 20% subsample.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Portuguese model selection should be task-specific: a model's rank on multilingual averages or on STS alone does not predict retrieval or reranking quality.
Long-context capacity and search-oriented training are concrete advantages on Portuguese retrieval and reranking, so context length deserves reporting alongside parameter count.
Portuguese contrastive fine-tuning remains worthwhile even for strong multilingual encoders, with the largest gains on symmetric semantic tasks such as STS.
MRL-trained models stay competitive after dimensional truncation, enabling compact Portuguese embeddings without training a separate model per dimension.
Multilingual benchmark ranks are a useful prior but not a substitute for language-specific evaluation, implying that similar per-language benchmark slices are needed for other under-resourced languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the rank shifts in Table 4 survive a multi-seed, full-size replication with confidence intervals, the same task-family mismatch likely appears in other low-resource languages, making aggregate multilingual leaderboards systematically misleading.
The closed-source retrieval scores are flagged as preliminary (20% of WebFaqRetrieval, possible contamination), so the reported open- versus closed-source gaps on retrieval may not hold on full data.
Because MTEB-PT omits clustering and bitext mining, the true task-dependence could be even stronger than shown; adding those families could change the picture for models that group well but rank poorly on STS.
The MRL fine-tuning recipe (contrastive pairs plus Matryoshka loss on a strong MLM backbone, then task-family evaluation) is a transferable template for building competitive language-specific sentence encoders elsewhere.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful Portuguese MTEB slice with honest limitations; the central rank-transfer claim needs variance estimates before it can be stated as strongly as it is.

read the letter

Worth reading this one. The authors built a clean Portuguese slice of MMTEB—14 datasets across STS, classification, retrieval, and reranking—evaluated 17 models under a unified protocol, and released three Portuguese fine-tuned MRL baselines with truncation curves. That is a real service for anyone choosing sentence encoders for Portuguese. The benchmark construction is sensible, the task choices are well documented, and the limitations section is unusually candid: they flag the 20% WebFaqRetrieval subsample for closed-source models, the missing clustering family, and the contamination risk.

The main empirical claim—that Portuguese performance is task-dependent and MMTEB rank does not transfer uniformly across task families—is plausible and the paper gives it real support in Tables 3 and 4. But the strength of the claim runs slightly ahead of the data. All scores are single-run point estimates: no seeds, no error bars, no bootstrap. Several subsets are small (MultiLongDoc retrieval and reranking have only 200 queries each; TweetSentiment has 870 examples), and nDCG@10 on 200 queries can easily carry standard errors in the low single digits. Table 4 shows rank swings of 3–6 positions among models separated by small margins. So the headline that multilingual rankings “do not reliably predict” Portuguese performance is under-supported at that level of detail. It should be softened to “we see substantial reordering, but confidence intervals are needed to rule out noise.” The paper’s own Limitations section is honest about the WebFaqRetrieval issue but never addresses the broader variance problem.

Also, the abstract and Section 6.1 say the fine-tuned baselines “also improve retrieval.” That is true for the two Portuguese MLM backbones, but not for e5-large-matryoshka, which drops to 70.7 from 73.4 on retrieval in Table 3. The sentence needs qualification.

Minor: the code and model release is promised without a URL or commit hash, and the rank-transfer analysis uses only the 8 models with all four MMTEB scores. None of this breaks the benchmark. The benchmark itself is solid, and the paper’s candid hedging keeps it honest. A bootstrap or repeated-seed evaluation would settle the main concern; as is, the central conclusion is directionally right but stronger than the evidence.

I would send this to a serious referee. The benchmark is a useful artifact, the protocol is reproducible in spirit, and the task-dependence finding is worth careful scrutiny. With a rephrased retrieval claim and variance estimates, it becomes a solid contribution.

Referee Report

3 major / 4 minor

Summary. MTEB-PT is a Portuguese evaluation suite built from 14 existing MMTEB datasets spanning STS, classification, retrieval, and reranking. The paper evaluates 17 open- and closed-source models under a unified MTEB-based protocol, and additionally fine-tunes three backbones with Portuguese contrastive supervision and Matryoshka Representation Learning (MRL). The central claims are that Portuguese embedding performance is strongly task-dependent, that multilingual (MMTEB) rankings do not reliably transfer to Portuguese across task families, that long-context/search-oriented models are advantageous on retrieval and reranking, and that Portuguese-specific contrastive fine-tuning yields large gains on STS while also improving retrieval and remaining competitive under dimensional truncation. The benchmark, fine-tuned models, and code are released.

Significance. The paper makes a useful and timely contribution: Portuguese is underrepresented in embedding evaluation, and MTEB-PT provides a compact, MMTEB-aligned benchmark with a unified protocol. The release of the benchmark, models, and code is a concrete asset for the community. The headline findings are falsifiable and practically relevant for model selection. However, the strength of the central rank-transfer claim currently exceeds what the evidence supports: the paper relies on single-run point estimates on small subsets, with no uncertainty quantification. If the observed rank reorderings survive bootstrap or repeated-seed analysis, the paper's conclusions would be well-supported; until then, the claims need to be framed more cautiously.

major comments (3)

[§6.2, Table 4] The strongest claim — that multilingual rankings do not reliably predict Portuguese performance — rests on rank changes computed from single-run point estimates. Table 1 shows that MultiLongDocRetrieval and MultiLongDocReranking have only 200 queries each, and Table 4 reports 3–6 position rank swings in reranking while Table 3 separates several models by less than 2 points. No standard errors, bootstrap intervals, seeds, or indices are reported anywhere. Without an uncertainty estimate, the observed reordering could be partly or wholly due to measurement noise, and the conclusion 'MMTEB rank is informative but not sufficient' would need to be weakened to 'the current data do not allow us to confirm transfer.' Please add variance estimates (e.g., bootstrap over queries or repeated runs with different seeds) or substantially temper the claim.
[§4.2, Table 3, §7] Closed-source retrieval averages are not fully comparable to open-model scores: WebFaqRetrieval was evaluated on only 20% of the dataset for closed-source models, with no indication of which subsample was used or how it was drawn. The footnote in Table 3 and §7 acknowledge this, but the headline comparison still places closed-source models at 86.9–91.1 in retrieval as if these were full-corpus scores. If the 20% subsample is not representative, the conclusion that closed-source models dominate Portuguese retrieval is not supported. Please either run the full evaluation, report the partial-subset results separately, or provide a variance estimate for the subsample.
[§5, §6.1] The fine-tuned models are trained on the official training splits of STSBenchmark, ASSIN/ASSIN2, SICK-BR, and NLI-style Portuguese data, while MTEB-PT evaluation uses test splits of the same or closely related datasets. The large STS gains (e.g., bertimbau-large-matryoshka 86.1 vs 63.1 backbone) therefore measure in-domain supervised adaptation, not language-specific fine-tuning in general. This is stated in §5, but the abstract and §6.1 frame it as a general finding about Portuguese adaptation. Please make the in-domain nature explicit in the abstract and conclusions, or add an evaluation on held-out Portuguese tasks not represented in the training data.

minor comments (4)

[Figure 2] The legend for all panels includes 'paraphrase-multilingual-mpnet-base-v2', but this model does not appear in Table 2 or Table 3. It should either be added to the model metadata and main results or removed from the figure.
[§4.2, reranking aggregation] Macro-averaging MAP and nDCG@10 is justified only by the assertion that both are bounded and capture ranking quality. Since Table 3's reranking column is used in the central argument, please either justify the aggregation more rigorously or report the individual reranking metrics separately.
[Table 4 caption] The caption refers to green/red arrows (▲/▼), which may be indistinguishable in grayscale printing. Consider using text symbols such as 'up', 'down', 'same' or distinct glyphs.
[§4.2, Table 1] For WebFaqRetrieval, the evaluation size for closed-source models differs from the nominal 10,000 queries, but Table 1 does not indicate this. Please add a footnote or separate column noting that closed-source evaluation used an approximately 2,000-query subsample.

Circularity Check

0 steps flagged

No significant circularity: the benchmark is an external measurement suite and fine-tuning is evaluated on held-out splits.

full rationale

The paper's derivation chain is not circular. MTEB-PT is a curated selection of existing MMTEB datasets; no dataset is defined by the paper's models or fitted parameters. The evaluation protocol uses official MTEB/MMTEB implementations and standard metrics (Section 4.2). The fine-tuned models are trained only on official training splits and evaluated on held-out test splits, with the paper explicitly stating this separation (Section 5). The main claims—task-dependent rankings, rank shifts relative to MMTEB, and long-context advantages on retrieval/reranking—are empirical measurements from running models on the benchmark, not derivations from the benchmark's definition. The only self-referential element is that fine-tuned models are trained on the same task families (STS, and retrieval pairs from MLDR) on which they are later evaluated; however, the paper does not present this as a prediction from first principles, but as an experimental finding, and it explicitly interprets the STS gains as 'consistent with the predominantly symmetric supervision'—which is an observed alignment, not a circular reduction. The Limitations section flags the 20% WebFaqRetrieval subsample for closed-source models, which is a measurement reliability concern, not a circularity concern. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted quantity is renamed as a prediction. The rank-transfer conclusion is based on measured scores; its statistical robustness is a separate correctness issue outside circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

Central benchmark claims rest on the MMTEB ecosystem, not on new derivations, so free parameters are few and mostly protocol choices: the MRL training recipe, the 20% WebFaqRetrieval subsample, and the macro-averaging scheme. Axioms are dominated by domain assumptions about official task implementations and the reliability/representativeness of the chosen Portuguese subsets. No invented entities: no new forces, dimensions, or mediators; the three fine-tuned models are engineering artifacts with claimed public release.

free parameters (3)

MRL fine-tuning hyperparameters = lr 5e-5, weight decay 0.2, warmup 0.1, 20 epochs, batch 512
Hand-chosen recipe applied identically to the three fine-tuned baselines (§5). The 'fine-tuning works' claim depends on these choices; they are not fitted to MTEB-PT test scores, but they are free choices of the authors.
WebFaqRetrieval subsample for closed-source models = 20% of the dataset
Closed-source retrieval averages are computed on 20% of WebFaqRetrieval due to budget (§4.2, §7). This limits comparability of the closed-source retrieval claim; the subsample is not described by seed or indices.
Task-family macro-averaging scheme = equal weight per dataset; reranking macro-average mixes MAP and nDCG@10
Aggregation choice in §4.2: macro-averaging across datasets, and mixing MAP and nDCG@10 within reranking. This affects all reported family averages and is justified only by boundedness, not equivalence of metrics.

axioms (5)

domain assumption Official MMTEB/MTEB task implementations and default metrics validly measure embedding quality in Portuguese
The entire benchmark rests on the 14 official implementations (§4.2). Any quirks or bugs in those implementations propagate to all results and rankings.
domain assumption The Portuguese-labeled MMTEB subsets are large and reliable enough for stable ranking
Several subsets are small (TweetSentiment 870 examples, MultiLongDoc 200 queries) and all scores are single-run with no error bars; the rank-change claims in Table 4 depend on this assumption (§6.2).
domain assumption Held-out separation: only official training splits were used for fine-tuning and evaluation test splits are disjoint
The 'fine-tuning improves STS' claim depends on no train/test leakage across STSBenchmark/ASSIN2/SICK-BR splits (§5). If any dataset's official split is violated or contaminated, the gains are inflated.
domain assumption MMTEB published scores for the 8 compared models were computed under equivalent protocols
The rank-transfer analysis (§6.2) compares MTEB-PT ranks with MMTEB ranks taken from the MMTEB paper [2] without re-running those models, silently assuming protocol comparability.
domain assumption First-d truncation is a fair baseline for non-MRL models
The truncation protocol (§4.3) assumes the leading d dimensions carry the most information for non-MRL models. Non-MRL models were not trained for truncation, so this choice structurally favors MRL models in the dimensional-efficiency analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 10974 in / 17076 out tokens · 176186 ms · 2026-08-02T08:41:14.064353+00:00 · methodology

0 comments

read the original abstract

Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world. As a result, embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear. We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking. We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol. Our results show that Portuguese performance is strongly task-dependent: multilingual rankings do not reliably predict Portuguese-specific performance across task families, no single model dominates all settings, and models with stronger long-context capacity are particularly advantageous on longer-input tasks such as retrieval and reranking. The benchmark also shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely. To examine this effect, we fine-tune three representative backbone models with Portuguese contrastive supervision and Matryoshka Representation Learning (MRL). These benchmark-informed baselines yield their strongest gains on STS, consistent with the predominantly symmetric supervision used during training, while also improving retrieval and remaining competitive under dimensional truncation. We release the MTEB-PT benchmark, the fine-tuned models, and the training and evaluation code.

Figures

Figures reproduced from arXiv: 2607.04071 by Alexandre Alcoforado, Anna Helena Reali Costa, Lucas Hideki Takeuchi Okamura.

**Figure 2.** Figure 2: Task-wise performance under embedding truncation on MTEB-PT. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

22 extracted references · 5 linked inside Pith

[1]

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation (2024)

2024
[2]

arXiv preprint arXiv:2502.13595 (2025)

Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., Gala, J., Siblini, W., Krzemiński, D., Winata, G.I., et al.: Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595 (2025). https://doi. org/10.48550/arXiv.2502.13595, https://arxiv.org/abs/2502.13595

doi:10.48550/arxiv.2502.13595 2025
[3]

Information11(10), 484 (2020)

Fialho, P., Coheur, L., Quaresma, P.: Benchmarking natural language inference and semantic textual similarity for portuguese. Information11(10), 484 (2020)

2020
[4]

In: Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal

Fonseca, E., Santos, L., Criscuolo, M., Aluisio, S.: Assin: Avaliacao de similaridade semantica e inferencia textual. In: Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal. pp. 13–15 (2016)

2016
[5]

In: EPIA Conference on Artificial Intelligence

Gomes, L., Branco, A., Silva, J., Rodrigues, J., Santos, R.: Open sentence embed- dings for portuguese with the serafim pt* encoders family. In: EPIA Conference on Artificial Intelligence. pp. 267–279. Springer (2024)

2024
[6]

arXiv preprint arXiv:2205.13147 (2022) MTEB-PT, a Benchmark for Portuguese Sentence Encoders 15

Kusupati, A., et al.: Matryoshka representation learning. arXiv preprint arXiv:2205.13147 (2022) MTEB-PT, a Benchmark for Portuguese Sentence Encoders 15

Pith/arXiv arXiv 2022
[7]

Preprint (Jun 2022), https://osf

Laurer, M., Atteveldt, W.v., Casas, A.S., Welbers, K.: Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint (Jun 2022), https://osf. io/74b8k, publisher: Open Science Framework

2022
[8]

IEEE Access (2024), https://ieeexplore.ieee.org/document/10531646

Li, P., Li, H.: Cosent: Consistent sentence embedding via similarity ranking. IEEE Access (2024), https://ieeexplore.ieee.org/document/10531646

arXiv 2024
[9]

(2021), https: //github.com/PhilipMay/stsb-multi-mt

May, P.: Machine translated multilingual sts benchmark dataset. (2021), https: //github.com/PhilipMay/stsb-multi-mt

2021
[10]

In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R

Melo, R., Santos, P.A., Dias, J.: A semantic search system for the supremo tribunal de justiça. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds.) Progress in Artificial Intelligence. pp. 142–154. Springer Nature Switzerland, Cham (2023)

2023
[11]

In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: Mteb: Massive text embed- ding benchmark. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pp. 2014–2037. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/10.18653/ v1/2023.eacl-main.148, https://aclanthol...

2014
[12]

local sen- tence embeddings for brazilian portuguese: Revisiting monolingual models in the age of foundation models

Peixoto, M., Silva, G., Figueredo, G., Silva, P., Luz, E.J.: Global vs. local sen- tence embeddings for brazilian portuguese: Revisiting monolingual models in the age of foundation models. In: Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026)-Vol. 1. pp. 529–539 (2026)

2026
[13]

Master’s thesis, Universidade Federal do Rio Grande do Sul (2025)

Pinto, G.V.: Portuguese Sentence Embedding Models Using Matryoshka Repre- sentation Learning. Master’s thesis, Universidade Federal do Rio Grande do Sul (2025)

2025
[14]

In: Proceedings of the ASSIN 2 Shared Task (2020), https://ceur-ws

Real, L., Fonseca, E., Gonçalo Oliveira, H.: The assin 2 shared task: A quick overview. In: Proceedings of the ASSIN 2 Shared Task (2020), https://ceur-ws. org/Vol-2583/

2020
[15]

In: International Conference on Computational Processing of the Portuguese Language

Real, L., Rodrigues, A., Vieira e Silva, A., Albiero, B., Thalenberg, B., Guide, B., Silva, C., de Oliveira Lima, G., Câmara, I.C., Stanojević, M., et al.: Sick-br: a portuguese corpus for inference. In: International Conference on Computational Processing of the Portuguese Language. pp. 303–312. Springer (2018)

2018
[16]

Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP). pp. 3982–3992 (2019)

2019
[17]

In: Proceedings of BRACIS (2020)

Souza, F., Nogueira, R., Lotufo, R.: BERTimbau: Pretrained BERT models for Brazilian Portuguese. In: Proceedings of BRACIS (2020)

2020
[18]

Stekel, T.R.C.: Mteb-br: A text embedding benchmark for brazilian portuguese (2026), https://arxiv.org/abs/2607.04581

Pith/arXiv arXiv 2026
[19]

arXiv preprint arXiv:2104.08663 (2021)

Thakur, N., Reimers, N., Daxenberger, J., Srivastava, A., Gurevych, I.: Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 (2021)

Pith/arXiv arXiv 2021
[20]

arXiv preprint arXiv:2402.05672 (2024)

Wang, L., et al.: Multilingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024)

Pith/arXiv arXiv 2024
[21]

arXiv preprint arXiv:2412.13663 (2024)

Warner, B., et al.: Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663 (2024)

Pith/arXiv arXiv 2024
[22]

In: Encontro Nacional de Inteligência Artificial e Computacional (ENIAC)

Wu, W.B.T.L., Garcia, L.P.F.: Modbertbr: A modernbert-based model for brazil- ian portuguese. In: Encontro Nacional de Inteligência Artificial e Computacional (ENIAC). pp. 2044–2055. SBC (2025)

2044

[1] [1]

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation (2024)

2024

[2] [2]

arXiv preprint arXiv:2502.13595 (2025)

Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., Gala, J., Siblini, W., Krzemiński, D., Winata, G.I., et al.: Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595 (2025). https://doi. org/10.48550/arXiv.2502.13595, https://arxiv.org/abs/2502.13595

doi:10.48550/arxiv.2502.13595 2025

[3] [3]

Information11(10), 484 (2020)

Fialho, P., Coheur, L., Quaresma, P.: Benchmarking natural language inference and semantic textual similarity for portuguese. Information11(10), 484 (2020)

2020

[4] [4]

In: Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal

Fonseca, E., Santos, L., Criscuolo, M., Aluisio, S.: Assin: Avaliacao de similaridade semantica e inferencia textual. In: Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal. pp. 13–15 (2016)

2016

[5] [5]

In: EPIA Conference on Artificial Intelligence

Gomes, L., Branco, A., Silva, J., Rodrigues, J., Santos, R.: Open sentence embed- dings for portuguese with the serafim pt* encoders family. In: EPIA Conference on Artificial Intelligence. pp. 267–279. Springer (2024)

2024

[6] [6]

arXiv preprint arXiv:2205.13147 (2022) MTEB-PT, a Benchmark for Portuguese Sentence Encoders 15

Kusupati, A., et al.: Matryoshka representation learning. arXiv preprint arXiv:2205.13147 (2022) MTEB-PT, a Benchmark for Portuguese Sentence Encoders 15

Pith/arXiv arXiv 2022

[7] [7]

Preprint (Jun 2022), https://osf

Laurer, M., Atteveldt, W.v., Casas, A.S., Welbers, K.: Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint (Jun 2022), https://osf. io/74b8k, publisher: Open Science Framework

2022

[8] [8]

IEEE Access (2024), https://ieeexplore.ieee.org/document/10531646

Li, P., Li, H.: Cosent: Consistent sentence embedding via similarity ranking. IEEE Access (2024), https://ieeexplore.ieee.org/document/10531646

arXiv 2024

[9] [9]

(2021), https: //github.com/PhilipMay/stsb-multi-mt

May, P.: Machine translated multilingual sts benchmark dataset. (2021), https: //github.com/PhilipMay/stsb-multi-mt

2021

[10] [10]

In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R

Melo, R., Santos, P.A., Dias, J.: A semantic search system for the supremo tribunal de justiça. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds.) Progress in Artificial Intelligence. pp. 142–154. Springer Nature Switzerland, Cham (2023)

2023

[11] [11]

In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: Mteb: Massive text embed- ding benchmark. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pp. 2014–2037. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/10.18653/ v1/2023.eacl-main.148, https://aclanthol...

2014

[12] [12]

local sen- tence embeddings for brazilian portuguese: Revisiting monolingual models in the age of foundation models

Peixoto, M., Silva, G., Figueredo, G., Silva, P., Luz, E.J.: Global vs. local sen- tence embeddings for brazilian portuguese: Revisiting monolingual models in the age of foundation models. In: Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026)-Vol. 1. pp. 529–539 (2026)

2026

[13] [13]

Master’s thesis, Universidade Federal do Rio Grande do Sul (2025)

Pinto, G.V.: Portuguese Sentence Embedding Models Using Matryoshka Repre- sentation Learning. Master’s thesis, Universidade Federal do Rio Grande do Sul (2025)

2025

[14] [14]

In: Proceedings of the ASSIN 2 Shared Task (2020), https://ceur-ws

Real, L., Fonseca, E., Gonçalo Oliveira, H.: The assin 2 shared task: A quick overview. In: Proceedings of the ASSIN 2 Shared Task (2020), https://ceur-ws. org/Vol-2583/

2020

[15] [15]

In: International Conference on Computational Processing of the Portuguese Language

Real, L., Rodrigues, A., Vieira e Silva, A., Albiero, B., Thalenberg, B., Guide, B., Silva, C., de Oliveira Lima, G., Câmara, I.C., Stanojević, M., et al.: Sick-br: a portuguese corpus for inference. In: International Conference on Computational Processing of the Portuguese Language. pp. 303–312. Springer (2018)

2018

[16] [16]

Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP). pp. 3982–3992 (2019)

2019

[17] [17]

In: Proceedings of BRACIS (2020)

Souza, F., Nogueira, R., Lotufo, R.: BERTimbau: Pretrained BERT models for Brazilian Portuguese. In: Proceedings of BRACIS (2020)

2020

[18] [18]

Stekel, T.R.C.: Mteb-br: A text embedding benchmark for brazilian portuguese (2026), https://arxiv.org/abs/2607.04581

Pith/arXiv arXiv 2026

[19] [19]

arXiv preprint arXiv:2104.08663 (2021)

Thakur, N., Reimers, N., Daxenberger, J., Srivastava, A., Gurevych, I.: Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 (2021)

Pith/arXiv arXiv 2021

[20] [20]

arXiv preprint arXiv:2402.05672 (2024)

Wang, L., et al.: Multilingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024)

Pith/arXiv arXiv 2024

[21] [21]

arXiv preprint arXiv:2412.13663 (2024)

Warner, B., et al.: Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663 (2024)

Pith/arXiv arXiv 2024

[22] [22]

In: Encontro Nacional de Inteligência Artificial e Computacional (ENIAC)

Wu, W.B.T.L., Garcia, L.P.F.: Modbertbr: A modernbert-based model for brazil- ian portuguese. In: Encontro Nacional de Inteligência Artificial e Computacional (ENIAC). pp. 2044–2055. SBC (2025)

2044