Pith. sign in

REVIEW 4 major objections 5 minor 54 references

A new benchmark built on Brazilian proverbs claims to measure how well LLMs understand figurative regional Portuguese, and finds a wide gap: large commercial models score above 0.95 while small open models stay near random.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:53 UTC pith:TKKEJ27K

load-bearing objection A solid, transparent Brazilian proverb benchmark with a clear scaling result; the main soft spots are missing error bars and the post-hoc distractor revision, but it deserves serious refereeing. the 4 major comments →

arxiv 2509.08960 v1 pith:TKKEJ27K submitted 2025-09-10 cs.CL

BRoverbs -- Measuring how much LLMs understand Portuguese proverbs

Thales Sales Almeida , Giovana Kerche Bon\'as , Jo\~ao Guilherme Alves Santos This is my paper
classification cs.CL
keywords Brazilian Portuguese proverbsLLM evaluationfigurative languagecultural understandingmultiple-choice benchmarkstory-to-proverb and proverb-to-storymodel scalepretraining data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces BRoverbs, a dataset of 193 Brazilian proverbs paired with 579 short narratives, and uses two multiple-choice tasks—Proverb to Story and Story to Proverb—to test language models' grasp of culturally embedded figurative meaning. The authors report a sharp performance divide: GPT-4o, Claude 3.5, and Sabiá-3 average above 0.95, Qwen 2.5 14B reaches 0.93, Llama-2-7B sits near random at 0.23, and models under 3B parameters mostly hover around the 20% chance level. The paper argues that translated benchmarks miss regional nuance, and that proverbs are a valid probe of a model's ability to connect an abstract moral to a concrete situation. It concludes that cultural comprehension depends less on raw Portuguese token volume than on a capacity or training threshold, since additional Portuguese pretraining did not lift small models above chance.

Core claim

BRoverbs is proposed as a culturally grounded evaluation tool for Portuguese-language LLMs, built from 193 manually vetted Brazilian proverbs, each with three GPT-4-generated, human-verified short narratives. The two tasks—selecting the story that fits a given proverb (PtS) and the proverb that fits a given story (StP)—yield scores that separate models by scale and training composition: commercial models exceed 0.95 on average, Qwen 2.5 models show steady gains with size and reach 0.93 at 14B, while Llama-1/2 and several small open models perform at chance. Notably, continued pretraining on Portuguese data did not help Curió-1.1B leave the random baseline, and Sabiá-7B improved only modestly

What carries the argument

The load-bearing mechanism is the paired-task design: each proverb is coupled with three human-validated short narratives that instantiate its meaning without quoting it, and the model must choose the correct match among five alternatives in both directions. Manual review of distractors removes near-synonymous proverbs from the answer pool, and lexical-diversity checks (Jaccard distance) confirm the three stories differ in wording. This setup forces the model to map between an abstract cultural formula and a concrete narrative situation, which the authors treat as a test of figurative, regional understanding rather than surface pattern matching.

Load-bearing premise

The benchmark's validity rests on the assumption that each GPT-4-generated and human-verified story unambiguously illustrates exactly one proverb; if a story can plausibly match several proverbs or carries surface cues from its generator, the tasks measure pattern matching instead of proverb comprehension.

What would settle it

Construct a control set of stories generated by a different model or by humans, or paraphrase the existing stories to change characters and settings while preserving the moral. If model performance drops sharply on these controls, the original scores are at least partly driven by surface lexical overlap or generator style; if performance holds, the benchmark likely captures the intended cultural-semantic mapping.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the benchmark measures what it claims, culturally grounded datasets expose gaps in LLM regional knowledge that translated evaluations would miss.
  • The near-random scores of small Portuguese-specialized models imply that adding Portuguese tokens alone does not confer cultural comprehension; model scale or training quality matters.
  • The strong performance of Qwen 2.5 models suggests that large-scale multilingual pretraining can transfer cultural understanding even when Portuguese documents are a small share of data.
  • A capacity threshold may exist below which proverb comprehension fails to emerge; the authors suggest this may resemble emergent abilities observed at scale.
  • The benchmark's current saturation at commercial level points to the next step: harder distractors or generation tasks rather than five-way matching.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the stories were generated by GPT-4 and manually cleaned, the benchmark may partially reward models that mimic GPT-4's narrative style; a cross-generator control would tell whether the scores measure meaning or stylistic familiarity.
  • The authors' threshold interpretation predicts that a small model with intensive, proverb-specific fine-tuning should jump above chance; that is testable without scaling up.
  • High commercial scores despite low Portuguese representation in training data hint that figurative understanding may transfer across languages via commonsense reasoning, which would generalize beyond Portuguese.
  • A harder variant that requires generating the proverb from a story, or distinguishing stories that share the same moral but different surface content, would directly test whether models grasp the moral rather than lexical echoes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. BRoverbs is a new benchmark for evaluating LLM comprehension of Brazilian proverbs. The authors collect 193 proverbs, generate 579 short stories with GPT-4, manually verify them, and construct two five-option multiple-choice tasks: Proverb-to-Story (PtS) and Story-to-Proverb (StP). They evaluate six commercial API models and ten open-weight models, reporting that large commercial models achieve near-perfect accuracy, Qwen models perform strongly among open models, and several small models perform near the random baseline. They also track Curió-1.1B intermediate checkpoints. The dataset is publicly available on Hugging Face.

Significance. If the validity of the items is established, BRoverbs is a useful contribution to Portuguese-language evaluation: it is a native rather than translated resource, the generation and curation pipeline is transparent, the approach targets culturally embedded figurative language, and the inclusion of both task directions plus continued-pretraining checkpoints is informative. The authors are also appropriately candid that GPT-4 was used for story generation and that GPT-4o does not dramatically outperform comparable-cost models. The main value depends on the benchmark measuring what it claims to measure, which is not currently demonstrated sufficiently.

major comments (4)
  1. [Section 3.2] The distractor-revision stage is a direct threat to the validity of the headline results. The authors re-examined all questions that at least one of the four commercial models (GPT-4o, GPT-4o-mini, Sabiazinho-3, Sabiá-3) answered incorrectly, which was about 12% of all questions, and modified roughly 35% of those. Because the same four models then appear in Table 1 with near-perfect scores, the benchmark has been partly shaped by those models' errors. This is not necessarily disqualifying, but the paper should report how many items were modified, characterize the nature of the modifications, compare scores before versus after revision, and ideally validate the final version with human annotators. Without this, the near-perfect commercial-model scores may be inflated by the revision process.
  2. [Section 3.1 / Section 3.2] The core validity assumption is that each story is a faithful and unambiguous illustration of exactly one proverb. The authors themselves found three proverbs whose meaning was disputed and later identified unintended overlaps among distractors, demonstrating that ambiguity exists. Yet no residual ambiguity estimate is provided, and there is no human baseline or inter-annotator agreement on the final dataset. If human readers would also select a different but semantically valid option, then model errors may reflect item ambiguity rather than lack of proverb understanding. I request a human evaluation on a representative sample (or all items) with agreement statistics, plus an analysis of items where humans disagree. This is load-bearing for interpreting any accuracy number on BRoverbs.
  3. [Section 4.1 / Table 1] All model comparisons rest on point accuracies with no number of evaluation instances, no confidence intervals, and no significance tests. For example, if PtS has on the order of 193 questions, the difference between 0.96 and 0.94 is within sampling noise; similarly, the claim that StP is consistently easier than PtS and the observation that Sabiá-7B improves only on StP are not quantified. The authors should report the number of evaluation instances per model and task, standard errors or bootstrap confidence intervals, and ideally multiple seeds for the randomly selected one-shot exemplar. This is necessary to support the scaling and training-data claims in Sections 4.1 and 4.2.
  4. [Section 4.2 / Figure 6] The claim that continued Portuguese pretraining of Curió-1.1B produces 'no significant gains' is based on point estimates at four or five checkpoints with no uncertainty quantification. Differences of roughly 0.02 around the random baseline are not distinguishable from noise. Please provide confidence intervals or a simple statistical test, and clarify whether the PtS 'slight improvement' and StP decline are within noise. This matters because the conclusion that Portuguese data does not help is used to support the capacity-threshold interpretation at the end of Section 4.2.
minor comments (5)
  1. [Section 2.1] 'sentimental analysis' should be 'sentiment analysis'.
  2. [Section 2.1] 'an small extractive QA dataset' should be 'a small extractive QA dataset'.
  3. [Section 3.2] The text says 'none of the three randomly chosen proverbs unintentionally matched...' but each question has four distractors; this should be 'four'.
  4. [Figure 6] The caption says 'Broverbs tasks'; should be 'BRoverbs'.
  5. [Section 4.1] The spelling 'Sábia 7B' is inconsistent with 'Sabiá-7B' elsewhere.

Circularity Check

0 steps flagged

No significant circularity: BRoverbs reports empirical benchmark scores, not a fitted derivation; self-citations and post-hoc item revision are present but not load-bearing.

full rationale

The paper's central claim is an empirical measurement—LLM accuracy on a newly constructed proverb–story matching benchmark—rather than a derivation from fitted parameters or equations. The gold labels are defined by human-verified mappings between proverbs and GPT-4-generated stories (Section 3.1), not by the evaluated models. The authors explicitly address the potential GPT-4-generation/GPT-4o-evaluation confound by noting that 'GPT-4o was used to help in story elaboration, it does not show a significant performance gap when compared to the other commercial models of similar cost,' and Claude 3.5 Sonnet achieves essentially the same score. Self-citations to Sabiá, TiEBe, and Curió are used to describe models and related benchmarks, not to justify the benchmark's validity, so they are not load-bearing. The main limitation is a validity risk, not circularity: Section 3.2 states that after running four commercial models, the authors 're-examined all questions that were answered incorrectly by at least one of the models—this accounted for about 12% of the total questions. From these questions, around 35% were modified.' This post-hoc revision of test items based on model errors could inflate reported scores, but it does not make the accuracy numbers equivalent to the input by construction; the modifications were aimed at removing genuine ambiguities, and the reported result is still an independent behavioral measurement. Overall, the paper is not circular in the formal sense; the observed score of 2 reflects minor, non-load-bearing self-citation plus this benchmark-curation risk, not a reduction of the result to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The benchmark's central claims rest on assumptions about story fidelity, distractor uniqueness, and the task's validity as a measure of cultural understanding. The paper is transparent about manual verification but does not provide evidence that the stories are unambiguous instantiations of the proverbs.

axioms (3)
  • domain assumption Accuracy on the BRoverbs tasks is a proxy for LLM understanding of Brazilian proverbs and cultural knowledge.
    The benchmark's validity rests on the assumption that matching a proverb to a narrative tests non-literal, cultural comprehension rather than memorization or shallow text heuristics.
  • domain assumption The stories generated by GPT-4 and manually verified accurately and unambiguously instantiate the intended proverbs.
    Stated in Section 3.1; any story that could be matched to multiple proverbs or fails to instantiate the proverb would make the benchmark measure something else.
  • domain assumption Manual review of distractors ensures exactly one correct answer per question.
    Section 3.2 describes manual review of about 12% of questions and modification of about 35% of those; this assumes the review fully resolves all ambiguous alternatives.

pith-pipeline@v1.3.0-alltime-deepseek · 14187 in / 9374 out tokens · 99028 ms · 2026-08-04T19:53:22.548569+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of BRoverbs -- Measuring how much LLMs understand Portuguese proverbs." pith.science (2026). https://pith.science/paper/TKKEJ27K

@misc{pith2026250908960,
  author       = {Pith},
  title        = {Pith review of: BRoverbs -- Measuring how much LLMs understand Portuguese proverbs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKKEJ27K}},
  note         = {Machine review of arXiv:2509.08960}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) exhibit significant performance variations depending on the linguistic and cultural context in which they are applied. This disparity signals the necessity of mature evaluation frameworks that can assess their capabilities in specific regional settings. In the case of Portuguese, existing evaluations remain limited, often relying on translated datasets that may not fully capture linguistic nuances or cultural references. Meanwhile, native Portuguese-language datasets predominantly focus on structured national exams or sentiment analysis of social media interactions, leaving gaps in evaluating broader linguistic understanding. To address this limitation, we introduce BRoverbs, a dataset specifically designed to assess LLM performance through Brazilian proverbs. Proverbs serve as a rich linguistic resource, encapsulating cultural wisdom, figurative expressions, and complex syntactic structures that challenge the model comprehension of regional expressions. BRoverbs aims to provide a new evaluation tool for Portuguese-language LLMs, contributing to advancing regionally informed benchmarking. The benchmark is available at https://huggingface.co/datasets/Tropic-AI/BRoverbs.

Figures

Figures reproduced from arXiv: 2509.08960 by Giovana Kerche Bon\'as, Jo\~ao Guilherme Alves Santos, Thales Sales Almeida.

Figure 1
Figure 1. Figure 1: Methodological flow of BRoverbs, illustrating the steps of data collection, story generation, creation of questions and answers, and model evaluation. 3.1 Data Collection and Story Genera￾tion Our process began by collecting Brazilian proverbs from online sources. We conducted three web searches 1 for Brazilian proverbs and gathered the top five results for each query, resulting in 15 different sources. Pr… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of Jaccard distances among the three short stories for each proverb. 3.2 Generation of Question-Answer Pairs After compiling the proverbs and their corresponding narratives, we designed two tasks to assess whether lan- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example from the BRoverbs dataset: The first section shows a PtS (Proverb to Story) task in Brazilian Portuguese, and the second presents the literal English translation of the same task. expected score, and the performance improvement from LLama1 to LLama2 is negligible, a possible reason for this is that, while Llama2 trains in the double of tokens, the proportion of Portuguese tokens in it’s train is st… view at source ↗
Figure 5
Figure 5. Figure 5: Example from the BRoverbs dataset: The first section shows a StP (story to proverb) task in Brazilian Portuguese, and the second presents the literal English translation of the same task. 0 20 40 60 80 100 120 140 160 Billion Tokens 0.17 0.18 0.19 0.20 0.21 0.22 0.23 Accuracy StP PtS Average [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy of Curió in Broverbs tasks through training 4.2 Does More Portuguese Data Im￾prove Cultural Understanding? Since intermediate checkpoints for Curió-1.1B are avail￾able, we can track their performance throughout training. This setup is particularly relevant because Curió starts from an English-only model and transitions the training to Portuguese data, allowing us to observe the impact of the conti… view at source ↗
Figure 7
Figure 7. Figure 7: Prompt completo utilizado para geração das histórias (redimensionado) [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 34 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    S., Laitz, T., Junior, R

    Abonizio, H., Almeida, T. S., Laitz, T., Junior, R. M., Bon \'a s, G. K., Nogueira, R., and Pires, R. (2024). Sabi 'a-3 technical report. arXiv preprint arXiv:2410.12049

    Pith/arXiv arXiv 2024

  3. [3]

    I., and Ekanata, Y

    Alfina, I., Mulia, R., Fanany, M. I., and Ekanata, Y. (2017). Hate speech detection in the indonesian language: A dataset and preliminary study. In 2017 international conference on advanced computer science and information systems (ICACSIS) , pages 233--238. IEEE

    2017

  4. [4]

    S., Abonizio, H., Nogueira, R., and Pires, R

    Almeida, T. S., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabi 'a-2: A new generation of portuguese large language models. arXiv preprint arXiv:2403.09887

    Pith/arXiv arXiv 2024

  5. [5]

    S., Bon \'a s, G

    Almeida, T. S., Bon \'a s, G. K., Santos, J. G. A., Abonizio, H., and Nogueira, R. (2025a). Tiebe: A benchmark for assessing the current knowledge of large language models. arXiv preprint arXiv:2501.07482

    Pith/arXiv arXiv

  6. [6]

    S., Laitz, T., Bon \'a s, G

    Almeida, T. S., Laitz, T., Bon \'a s, G. K., and Nogueira, R. (2023). Bluex: A benchmark based on brazilian leading universities entrance exams. In Brazilian Conference on Intelligent Systems , pages 337--347. Springer

    2023

  7. [7]

    S., Nogueira, R., and Pedrini, H

    Almeida, T. S., Nogueira, R., and Pedrini, H. (2025b). Building high-quality datasets for portuguese llms: From common crawl snapshots to industrial-grade corpora. To Appear

  8. [8]

    Introducing claude 3.5 haiku

    Anthropic (2024a). Introducing claude 3.5 haiku. https://www.anthropic.com/claude/haiku

  9. [9]

    Introducing claude 3.5 sonnet

    Anthropic (2024b). Introducing claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

  10. [10]

    A., Tonja, A

    Azime, I. A., Tonja, A. L., Belay, T. D., Chanie, Y., Balcha, B. F., Abadi, N. H., Ademtew, H. B., Nerea, M. A., Yadeta, D. D., Geremew, D. D., et al . (2024). Proverbeval: Exploring llm evaluation challenges for low-resource language understanding. arXiv preprint arXiv:2411.05049

    Pith/arXiv arXiv 2024

  11. [11]

    P., P \'e rez, N., Salles, A., Docio, S

    Baucells, I., Aula-Blasco, J., de Dios-Flores, I., Su \'a rez, S. P., P \'e rez, N., Salles, A., Docio, S. S., Falc \ a o, J., Saiz, J. J., Sep \'u lveda-Torres, R., et al . (2025). Iberobench: A benchmark for llm evaluation in iberian languages. In Proceedings of the 31st International Conference on Computational Linguistics , pages 10491--10519

    2025

  12. [12]

    Brum, H. B. and Nunes, M. d. G. V. (2017). Building a sentiment corpus of tweets in brazilian portuguese. arXiv preprint arXiv:1712.08917

    Pith/arXiv arXiv 2017

  13. [13]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al . (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    Pith/arXiv arXiv 2021

  14. [14]

    K., Sen, A., Falk, S., and Fatimah, S

    Corr \^e a, N. K., Sen, A., Falk, S., and Fatimah, S. (2024). Tucano: Advancing neural text generation for portuguese. arXiv preprint arXiv:2411.07854

    Pith/arXiv arXiv 2024

  15. [15]

    H., and Rademaker, A

    Delfino, P., Cuconato, B., Haeusler, E. H., and Rademaker, A. (2017). Passing the brazilian oab exam: data preparation and some experiments. In Legal knowledge and information systems , pages 89--94. IOS Press

    2017

  16. [16]

    Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. (2019). Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161

    Pith/arXiv arXiv 2019

  17. [17]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al . (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  18. [18]

    R., Wanner, L., Nunes, S., et al

    Fortuna, P., da Silva, J. R., Wanner, L., Nunes, S., et al . (2019). A hierarchically-labeled portuguese hate speech dataset. In Proceedings of the third workshop on abusive language online , pages 94--104

    2019

  19. [19]

    Giagkou, M., Lynn, T., Dunne, J., Piperidis, S., and Rehm, G. (2023). European language technology in 2022/2023. In European Language Equality: A Strategic Agenda for Digital Language Equality , pages 75--94. Springer

    2023

  20. [20]

    S., Samin, K., Li, Y.-F., Kang, Y.-B., Rahman, M

    Hasan, T., Bhattacharjee, A., Islam, M. S., Samin, K., Li, Y.-F., Kang, Y.-B., Rahman, M. S., and Shahriyar, R. (2021). Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. arXiv preprint arXiv:2106.13822

    Pith/arXiv arXiv 2021

  21. [21]

    D., Sugawara, S., and Aizawa, A

    Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. (2020). Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060

    Pith/arXiv arXiv 2020

  22. [22]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al . (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  23. [23]

    Jiang, Z., Anastasopoulos, A., Araki, J., Ding, H., and Neubig, G. (2020). X-factr: Multilingual factual knowledge retrieval from pretrained language models. arXiv preprint arXiv:2010.06189

    Pith/arXiv arXiv 2020

  24. [24]

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al . (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453--466

    2019

  25. [25]

    Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017). Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683

    Pith/arXiv arXiv 2017

  26. [26]

    Larcher, C., Piau, M., Finardi, P., Gengo, P., Esposito, P., and Carid \'a , V. (2023). Cabrita: closing the gap for foreign languages. arXiv preprint arXiv:2308.11878

    Pith/arXiv arXiv 2023

  27. [27]

    Li, Y., Wang, S., Ding, H., and Chen, H. (2023). Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance , pages 374--382

    2023

  28. [28]

    Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al . (2024). Mapping the increasing use of llms in scientific papers. arXiv preprint arXiv:2404.01268

    Pith/arXiv arXiv 2024

  29. [29]

    Longpre, S., Singh, N., Cherep, M., Tiwary, K., Materzynska, J., Brannon, W., Mahari, R., Dey, M., Hamdy, M., Saxena, N., et al . (2024). Bridging the data provenance gap across text, speech and video. arXiv preprint arXiv:2412.17847

    Pith/arXiv arXiv 2024

  30. [30]

    Mi, M., Villavicencio, A., and Moosavi, N. S. (2024). Rolling the dice on idiomaticity: How llms fail to grasp context. arXiv preprint arXiv:2410.16069

    Pith/arXiv arXiv 2024

  31. [31]

    Moayeri, M., Tabassi, E., and Feizi, S. (2024). Worldbench: Quantifying geographic disparities in llm factual recall. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , pages 1211--1228

    2024

  32. [32]

    A., et al

    Myung, J., Lee, N., Zhou, Y., Jin, J., Putri, R., Antypas, D., Borkakoty, H., Kim, E., Perez-Almendros, C., Ayele, A. A., et al . (2024). Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems , 37:78104--78146

    2024

  33. [33]

    U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A

    Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435

    Pith/arXiv arXiv 2023

  34. [34]

    Overwijk, A., Xiong, C., and Callan, J. (2022). Clueweb22: 10 billion web documents with rich information. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval , pages 3360--3362

    2022

  35. [35]

    M., and Bowman, S

    Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. (2021). Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193

    Pith/arXiv arXiv 2021

  36. [36]

    S., and Nogueira, R

    Pires, R., Abonizio, H., Almeida, T. S., and Nogueira, R. (2023). Sabi \'a : Portuguese large language models. pages 226--240

    2023

  37. [37]

    Potts, C., Wu, Z., Geiger, A., and Kiela, D. (2020). Dynasent: A dynamic benchmark for sentiment analysis. arXiv preprint arXiv:2012.15349

    Pith/arXiv arXiv 2020

  38. [38]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

    Pith/arXiv arXiv 2016

  39. [39]

    L., Tawde, A., Choudhary, S

    Rane, N. L., Tawde, A., Choudhary, S. P., and Rane, J. (2023). Contribution and performance of chatgpt and other large language models (llm) for scientific and research advancements: a double-edged sword. International Research Journal of Modernization in Engineering Technology and Science , 5(10):875--899

    2023

  40. [40]

    Rudinger, R., Naradowsky, J., Leonard, B., and Van Durme, B. (2018). Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301

    Pith/arXiv arXiv 2018

  41. [41]

    F., Araujo, A

    Sayama, H. F., Araujo, A. V., and Fernandes, E. R. (2019). Faquad: Reading comprehension dataset in the domain of brazilian higher education. In 2019 8th Brazilian conference on intelligent systems (BRACIS) , pages 443--448. IEEE

    2019

  42. [42]

    Silveira, I. C. and Mau \'a , D. D. (2017). University entrance exam as a guiding test for artificial intelligence. In 2017 Brazilian Conference on Intelligent Systems (BRACIS) , pages 426--431. IEEE

    2017

  43. [43]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al . (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615

    Pith/arXiv arXiv 2022

  44. [44]

    Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. (2018). Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355

    Pith/arXiv arXiv 2018

  45. [45]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al . (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    Pith/arXiv arXiv

  46. [46]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al . (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    Pith/arXiv arXiv

  47. [47]

    A., Carvalho, I., de G \'o es, F

    Vargas, F. A., Carvalho, I., de G \'o es, F. R., Benevenuto, F., and Pardo, T. A. S. (2021). Hatebr: A large expert annotated corpus of brazilian instagram comments for offensive language and hate speech detection. arXiv preprint arXiv:2103.14972

    Pith/arXiv arXiv 2021

  48. [48]

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems , 32

    2019

  49. [49]

    Watts, I., Gumma, V., Yadavalli, A., Seshadri, V., Swaminathan, M., and Sitaram, S. (2024). Pariksha: A large-scale investigation of human-llm evaluator agreement on multilingual and multi-cultural data. arXiv preprint arXiv:2406.15053

    Pith/arXiv arXiv 2024

  50. [50]

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al . (2022). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    Pith/arXiv arXiv 2022

  51. [51]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al . (2024). Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  52. [52]

    Yu, W., Jiang, Z., Dong, Y., and Feng, J. (2020). Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326

    Pith/arXiv arXiv 2020

  53. [53]

    Zhang, P., Zeng, G., Wang, T., and Lu, W. (2024). Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385

    Pith/arXiv arXiv 2024

  54. [54]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al . (2023). A survey of large language models. arXiv preprint arXiv:2303.18223 , 1(2)

    Pith/arXiv arXiv 2023