The reviewed record of science sign in
Pith

arxiv: 2305.06721 · v2 · pith:MENTTJRE · submitted 2023-05-11 · cs.CL

Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*

Jo\~ao Rodrigues , Lu\'is Gomes , Jo\~ao Silva , Ant\'onio Branco , Rodrigo Santos , Henrique Lopes Cardoso , Tom\'as Os\'orio This is my paper

Reviewed by Pithpith:MENTTJREopen to challenge →

classification cs.CL
keywords portuguesealbertinapt-brlanguagept-ptsetsdataencoding
0
0 comments X
read the original abstract

To advance the neural encoding of Portuguese (PT), and a fortiori the technological preparation of this language for the digital age, we developed a Transformer-based foundation model that sets a new state of the art in this respect for two of its variants, namely European Portuguese from Portugal (PT-PT) and American Portuguese from Brazil (PT-BR). To develop this encoder, which we named Albertina PT-*, a strong model was used as a starting point, DeBERTa, and its pre-training was done over data sets of Portuguese, namely over data sets we gathered for PT-PT and PT-BR, and over the brWaC corpus for PT-BR. The performance of Albertina and competing models was assessed by evaluating them on prominent downstream language processing tasks adapted for Portuguese. Both Albertina PT-PT and PT-BR versions are distributed free of charge and under the most permissive license possible and can be run on consumer-grade hardware, thus seeking to contribute to the advancement of research and innovation in language technology for Portuguese.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

    cs.CL 2026-04 unverdicted novelty 7.0

    NorBERTo, a ModernBERT encoder trained on the largest open Portuguese corpus of 331B tokens, reports top encoder results on several PLUE and ASSIN 2 tasks.

  2. LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

    cs.CL 2026-04 conditional novelty 7.0

    Fine-tuned BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 on LegalBench-BR, outperforming commercial LLMs by 22-28 points and eliminating their systematic bias toward civil law on Brazilian legal classification.