Pith. sign in

REVIEW 3 major objections 5 minor 24 references

HalleluBERT, a Hebrew RoBERTa encoder trained from scratch on 49.1 GB of web and Wikipedia text, outperforms every compared monolingual and multilingual model on native NER and sentiment benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HalleluBERT, a Hebrew-only RoBERTa encoder family trained from scratch at scale, reports the highest unweighted mean scores on BMC, NEMO, and SMCD benchmarks, but without statistical significance testing.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A genuinely useful large Hebrew RoBERTa resource, but the SOTA claim needs error bars before it should be taken as established. the 3 major comments →

arxiv 2510.21372 v2 pith:RGNJLSAJ submitted 2025-10-24 cs.CL

HalleluBERT: Let Every Token That Has Meaning Bear Its Weight

Raphael Schmitt This is my paper
classification cs.CL
keywords Hebrew NLPRoBERTapretrainingnamed entity recognitionsentiment classificationbyte-level BPEmonolingual language modelbenchmark evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a Hebrew-specific RoBERTa encoder trained at scale, in both base and large variants, closes the gap left by earlier Hebrew models and multilingual encoders. On the native BMC, NEMO, and SMCD benchmarks, the large model reaches an average of 88.95 across the three tasks, the base model 87.83, and both outperform existing monolingual Hebrew models and multilingual baselines. If correct, this supports the broader claim that fully converged monolingual pretraining with a language-specific tokenizer pays off for a mid-resource language like Hebrew. The work matters because the field lacks a large-scale, reproducible Hebrew encoder: both variants and the tokenizer are released for public use.

Core claim

The paper's central claim is that HalleluBERT-large and HalleluBERT-base achieve the highest unweighted mean score across the BMC, NEMO, and SMCD benchmarks among all compared models, with the large variant at 88.95 versus 87.49 for the strongest compared multilingual baseline. The authors attribute this to training a RoBERTa-style encoder from scratch on a 49.1 GB deduplicated Hebrew web and Wikipedia corpus, a Hebrew-specific byte-level BPE vocabulary, high-batch pretraining for about 61 epochs, and scaling to a large variant. They further observe that scaling yields modest but consistent gains on NEMO and sentiment, while the smaller BMC benchmark does not reward the large model.

What carries the argument

The load-bearing mechanism is the pretraining recipe: a RoBERTa-style masked-language-model head trained from scratch in full precision for 100k update steps with a global batch size of 8k on about 49.1 GB of deduplicated Hebrew text, using a 52k-subword byte-level BPE vocabulary learned from 20 GB of that text. The argument is that this combination—monolingual data at scale, a Hebrew-specific tokenizer, high batch size, and long training—produces representations that transfer better to downstream Hebrew NER and sentiment than less-trained Hebrew models or multilingual encoders.

Load-bearing premise

The paper's ranking rests on the assumption that selecting the best of ten fine-tuning runs by validation score, with no significance tests or standard deviations, yields fair and stable comparisons across all models; if run-to-run variation exceeds the roughly 1.5-point margins, the state-of-the-art claim collapses.

What would settle it

Re-running the three benchmarks with every model on the same hyperparameter grid but reporting the full distribution over random seeds (mean, standard deviation, and per-seed best) would settle the matter: if the best-of-ten HalleluBERT-large average falls within one standard deviation of the strongest multilingual baseline, the claimed superiority is not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fully converged monolingual pretraining can push Hebrew encoder performance past multilingual models despite much larger multilingual training data.
  • The base variant (126M parameters) is competitive, while the large variant (357M) yields the best average, so scaling helps on two of the three tasks.
  • A prior Hebrew RoBERTa model may not have been trained long enough or optimally: under the same evaluation protocol, the new base model outperforms it on NER and the overall average.
  • Releasing both variants and the Hebrew-specific tokenizer under an MIT license lets Hebrew NLP applications start from better representations.
  • The results suggest that a conservative hyperparameter schedule can still produce strong gains when combined with large-batch, long training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported 1.5-point average margin over the strongest multilingual baseline may not be stable: the paper selects the best of ten fine-tuning runs per model without significance tests, so run-to-run noise could shrink the gap if score distributions were compared.
  • The causal role of the Hebrew-specific tokenizer is inferred rather than demonstrated, since the paper does not quantify its separate impact on compression or downstream performance.
  • The gains are confined to three tasks; extending the same protocol to question answering, natural language inference, and long-context understanding would show whether the advantage generalizes beyond NER and sentiment.
  • If the margin survives a distribution-based comparison, it would add to the cross-linguistic pattern that monolingual encoders trained to convergence at moderate scale can beat much larger multilingual encoders on native benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces HalleluBERT, a family of Hebrew RoBERTa-style encoders (base and large) pre-trained from scratch on 49.1 GB of deduplicated Hebrew web text and Wikipedia, with a Hebrew-specific byte-level BPE vocabulary. The authors evaluate the models on three native Hebrew benchmarks — BMC and NEMO for named entity recognition, and SMCD for sentiment classification — using the HeRo benchmark suite and an NNI/Huggingface grid-search pipeline. They report that HalleluBERT-large achieves the highest unweighted mean score (88.95) across the three benchmarks, followed by HalleluBERT-base (87.83), surpassing monolingual baselines (HeBERT, AlephBERT, HeRo) and multilingual baselines (mmBERT, XLM-RoBERTa). The models and tokenizer are released under the MIT license.

Significance. If the empirical claims are reliable, this paper makes a useful contribution: it provides the first large-scale Hebrew RoBERTa encoder in both base and large variants, trained on a substantial corpus and evaluated on native benchmarks. The resource release under MIT, the transparent reporting of hyperparameters and training configurations, and the honest limitations section are strengths. The paper also adds evidence for the value of monolingual, scale-oriented pretraining for Hebrew. However, the central claim — state-of-the-art performance over existing baselines — rests on small differences between models under a best-of-10 selection protocol with no variance reporting or significance testing, so the magnitude and stability of the claimed improvements are not yet established.

major comments (3)
  1. [§4, Table 1; Appendix C] The headline results are reported as a single number per model: the best score out of 10 runs selected by validation performance, with no standard deviation, confidence interval, or significance test. The decisive margins are small — HalleluBERT-large vs. XLM-R-large is 1.46 points overall (88.95 vs. 87.49), and HalleluBERT-base is only 0.34 points ahead of XLM-R-large (87.83 vs. 87.49). On SMCD the large-model gap is 1.17 points. Given the small sizes of the test sets (SMCD has roughly 1,700 test comments; BMC and NEMO are small) and the validation set being only 8% of the training portion for SMCD, these differences can easily arise from run-to-run noise. Best-of-10 selection amplifies this: selecting the run with the best validation score produces an optimistic, high-variance estimate of test performance, and the bias may differ across models if the restricted hyperparameter grid (bat
  2. [§5, Table 1] The paper states that the authors were unable to reproduce HeRo's reported state-of-the-art results, and in their evaluation HeRo (85.43 overall) falls behind AlephBERT-Gimmel (86.99) and well behind HalleluBERT-base (87.83). This is a load-bearing issue for the SOTA claim: if the adopted evaluation protocol does not reproduce a published baseline's reported performance, that baseline may not be evaluated at its best, and the comparison may be skewed in favor of the new model. The manuscript should clarify whether the HeRo numbers in Table 1 come from the authors' own runs or from the original paper; if from own runs, the authors should show that the hyperparameter grid and evaluation protocol recover the published results when the published hyperparameters or settings are used, or discuss why the discrepancy does not affect the fairness of the comparison.
  3. [Appendix C; §3.4] The downstream evaluation relies on a restricted grid search (batch sizes {16,32} and learning rates {5e-6, 7e-6, 1e-5, 2e-5, 5e-5}) and best-of-10 selection. The paper does not provide evidence that the optimal hyperparameters for each baseline lie within this grid. Given that the authors themselves report failing to reproduce HeRo's published numbers, the grid's neutrality across all models is doubtful. At minimum, the authors should report the distribution of validation and test scores across the 10 runs and show that the selected hyperparameters are not on the grid boundary for any model, or that the conclusions are robust to alternative hyperparameter choices.
minor comments (5)
  1. [Appendix D, title] Typographical error: 'Squence Length' should be 'Sequence Length'.
  2. [§3.3] 'Due to the fairseq implementation single- or mixed-precision training was not employed' is grammatically awkward and could be rephrased for clarity.
  3. [§3.4.1] The description of the validation split says '10% of the training portion' but the resulting distribution is stated as 'approximately 72% train, 8% validation, and 20% test.' If the official test is 20%, 10% of the 80% training portion would be 8% of the total, so the numbers are consistent, but the wording could be clearer.
  4. [Figure 1] The figure caption is unclear: 'Top based on a validation at the checkpoints. Bottom based on the validation of each optimization cycle during the training.' Please clarify what is plotted on each panel.
  5. [§5] The sentence beginning 'This contrasts with Scheible et al. (2024), where conservative pre-training learning rates and TPU-based pretraining...' is difficult to follow and seems to describe the authors' own setup rather than a contrast. Please rewrite for clarity.

Circularity Check

0 steps flagged

No circularity: HalleluBERT's claims are direct measurements on external benchmarks, not derived from fitted parameters or self-citation.

full rationale

The paper contains no derivation chain to reduce. The central claim (HalleluBERT outperforms baselines on BMC, NEMO, SMCD) is an empirical result obtained by fine-tuning released models on held-out test sets. None of the reported scores are defined in terms of a fitted parameter or an equation that includes the target outcome; the best-of-10 selection by validation performance and restricted hyperparameter grid are evaluation-protocol choices, not circular reductions. Self-citations (GottBERT, GeistBERT, He et al. 2025) supply a training recipe and grid-search pipeline, but the benchmark outcomes are not defined by those references, and the tokenizer's contribution is explicitly left unquantified rather than being presented as a prediction. The inability to reproduce HeRo's reported scores and the lack of significance testing are correctness/statistical concerns, not circularity. The paper is self-contained against external benchmarks and baselines, so the score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The empirical comparison depends on training choices and evaluation assumptions rather than mathematical axioms. The most consequential are: the corpus is clean enough, the tokenization choice is adequate (explicitly unquantified), the three benchmarks and unweighted mean are adequate proxies for Hebrew NLP quality, and the best-of-10 validation selection without significance testing is a fair estimator. These are domain and procedural assumptions, not derived results.

free parameters (5)
  • Pretraining peak learning rates = base 0.0004, large 0.00015
    Hand-chosen following RoBERTa/GottBERT conventions; the paper notes the large model used a conservative LR and no extensive tuning was done.
  • Pretraining batch size and update steps = 8192, 100k
    Standard RoBERTa schedule chosen by hand; the paper states the dataset permitted roughly 61 epochs.
  • Tokenizer training sample size = 20 GB
    Ad hoc; the paper explicitly says the tokenizer's impact on compression or downstream performance was not quantified.
  • Downstream task hyperparameters per model/task = batch sizes 16/32, learning rates 5e-6 to 5e-5, selected by validation (Table 4)
    Grid-selected per model; the reported best-of-10 scores depend on these choices, and the fairness of the grid across models is a key assumption.
  • Maximum sequence lengths per dataset = 64 (BMC), 192 (NEMO/SMCD)
    Chosen by hand from 95th percentile sequence lengths plus a safety margin; affects how much context is seen during fine-tuning.
axioms (4)
  • domain assumption HeDC4 plus Wikipedia provides a sufficiently clean and representative corpus for Hebrew pretraining; no additional filtering or cross-source deduplication was applied.
    Section 3.1 and the Limitations section admit residual noise and biases may remain; if the corpus is not representative, the learned representations and benchmark results could shift.
  • domain assumption BMC, NEMO, and SMCD are adequate benchmarks for measuring Hebrew NLP model quality, and the unweighted mean across them is a meaningful combined metric.
    Section 3.4 and Table 1; the paper excludes QA and long-context tasks, so the SOTA claim is limited to these three tasks.
  • domain assumption Byte-level BPE without pretokenization is appropriate for Hebrew morphology, and the tokenizer's contribution does not need to be quantified.
    Section 3.2 explicitly states the tokenizer impact was not separately measured; if the Hebrew tokenizer is suboptimal, the comparisons could be affected.
  • ad hoc to paper The grid-search and best-of-10 validation selection protocol yields fair, stable estimates of model quality.
    Appendix C and Table 1; no standard deviations or significance tests are reported, so the ranking could be sensitive to this protocol.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HalleluBERT: Let Every Token That Has Meaning Bear Its Weight." pith.science (2026). https://pith.science/paper/RGNJLSAJ

@misc{pith2026251021372,
  author       = {Pith},
  title        = {Pith review of: HalleluBERT: Let Every Token That Has Meaning Bear Its Weight},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RGNJLSAJ}},
  note         = {Machine review of arXiv:2510.21372}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT, a RoBERTa-based encoder family trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia using a Hebrew-specific byte-level BPE vocabulary. On native Hebrew benchmarks for named entity recognition (BMC, NEMO) and sentiment classification (SMCD), HalleluBERT outperforms monolingual and multilingual baselines, and yields the highest unweighted mean score across the three benchmarks. We release model weights and tokenizer under the MIT license to support reproducible Hebrew NLP research.

Figures

Figures reproduced from arXiv: 2510.21372 by Raphael Schmitt.

Figure 1
Figure 1. Figure 1: Perplexity of the HalleluBERT models. Top [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 1 canonical work pages

  1. [1]

    Julien Abadji , Pedro Ortiz Suarez , Laurent Romary , and Beno \^ t Sagot . 2022. https://arxiv.org/abs/2201.06642 Towards a Cleaner Document-Oriented Multilingual Crawled Corpus . arXiv e-prints, arXiv:2201.06642

    Pith/arXiv arXiv 2022

  2. [2]

    Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. https://aclanthology.org/C18-1190/ Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from M odern H ebrew . In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242--2252, Santa Fe, New Mexico, USA. Associa...

    2018

  3. [3]

    Dan Bareket and Reut Tsarfaty. 2021. https://doi.org/10.1162/tacl_a_00404 Neural Modeling for Named Entities and Morphology (NEMO2) . Transactions of the Association for Computational Linguistics, 9:909--928

    work page doi:10.1162/tacl_a_00404 2021

  4. [4]

    Naama Ben-Mordecai. 2005. https://www.cs.bgu.ac.il/ elhadad/nlpproj/naama/ Hebrew named entity recognition . Master's thesis, Department of Computer Science, Ben-Gurion University

    2005

  5. [5]

    Branden Chan. 2020. https://towardsdatascience.com/xlm-roberta-the-multilingual-alternative-for-non-english-nlp-cf0b889ccbbf XLM - RoBERTa : The multilingual alternative for non-english NLP . Library Catalog: towardsdatascience.com

    2020

  6. [6]

    Branden Chan, Stefan Schweter, and Timo M \"o ller. 2020. https://doi.org/10.18653/v1/2020.coling-main.598 G erman ' s next language model . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788--6796, Barcelona, Spain (Online). International Committee on Computational Linguistics

    doi:10.18653/v1/2020.coling-main.598 2020

  7. [7]

    Avihay Chriqui and Inbal Yahav. 2022. https://doi.org/10.1287/ijds.2022.0016 Hebert and hebemo: A hebrew bert model and a tool for polarity analysis and emotion recognition . INFORMS Journal on Data Science, 1(1):81–95

    arXiv 2022

  8. [8]

    Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. http://arxiv.org/abs/2001.06286 RobBERT : a Dutch RoBERTa -based Language Model . arXiv:2001.06286 [cs]. ArXiv: 2001.06286

    Pith/arXiv arXiv 2020

  9. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies , Volume 1 ( Lon...

    doi:10.18653/v1/n19-1423 2019

  10. [10]

    Henry He, Johann Frei, and Raphael Scheible-Schmitt. 2025. https://doi.org/10.21203/rs.3.rs-7332811/v1 The Word and the Way : Strategies for Domain - Specific BERT Pre - Training in German Medical NLP . ISSN: 2693-5015

    doi:10.21203/rs.3.rs-7332811/v1 2025

  11. [11]

    Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. 2023. https://doi.org/10.48550/arXiv.2304.01433 TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddin...

    Pith/arXiv arXiv doi:10.48550/arxiv.2304.01433 2023

  12. [12]

    Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. https://www.aclweb.org/anthology/P07-2045 Moses: Open Source Toolkit for Statistical Machine Translation . In Proceedings of the...

    2007

  13. [13]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 RoBERTa : A Robustly Optimized BERT Pretraining Approach . arXiv:1907.11692 [cs]. ArXiv: 1907.11692

    Pith/arXiv arXiv 2019

  14. [14]

    Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. https://arxiv.org/abs/2509.06888 mmbert: A modern multilingual encoder with annealed language learning . Preprint, arXiv:2509.06888

    Pith/arXiv arXiv 2025

  15. [15]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, \' E ric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. https://www.aclweb.org/anthology/2020.acl-main.645 CamemBERT : a Tasty French Language Model . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7203--7219,...

    2020

  16. [16]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. http://arxiv.org/abs/1904.01038 fairseq: A Fast , Extensible Toolkit for Sequence Modeling . arXiv:1904.01038 [cs]. ArXiv: 1904.01038

    Pith/arXiv arXiv 2019

  17. [17]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9

    2019

  18. [18]

    Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1183 G ott BERT : a pure G erman language model . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21237--21250, Miami, ...

    doi:10.18653/v1/2024.emnlp-main.1183 2024

  19. [19]

    Raphael Scheible-Schmitt and Johann Frei. 2025. https://arxiv.org/abs/2506.11903 Geistbert: Breathing life into german nlp . Preprint, arXiv:2506.11903

    Pith/arXiv arXiv 2025

  20. [20]

    Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. https://arxiv.org/abs/2104.04052 Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with . Preprint, arXiv:2104.04052

    Pith/arXiv arXiv 2021

  21. [21]

    Vitaly Shalumov and Harel Haskey. 2023. https://arxiv.org/abs/2304.11077 Hero: Roberta and longformer hebrew language models . Preprint, arXiv:2304.11077

    Pith/arXiv arXiv 2023

  22. [22]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. https://www.aclweb.org/anthology/2020.emnlp-demos.6 Transformers...

    2020

  23. [23]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  24. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.