{"id":"41bf357d-54e7-43bf-94b0-0dceef4153fe","arxiv_id":"2510.21372","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HalleluBERT, a Hebrew-only RoBERTa encoder family trained from scratch at scale, reports the highest unweighted mean scores on BMC, NEMO, and SMCD benchmarks, but without statistical significance testing.","lead":"This paper trains and releases HalleluBERT, a family of Hebrew-only RoBERTa text encoders (base and large) pretrained on about 49 GB of Hebrew web text and Wikipedia. It reports top average scores on three Hebrew NER and sentiment benchmarks, positioning HalleluBERT as a new open resource for Hebrew NLP—though the reported wins are small and come without error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significance testing or variance reporting under best-of-10 selection makes the 0.34–1.46 point SOTA margins indistinguishable from noise; the central claim is conditional on a check that the ranking is stable across seeds.","rationale":"The reader's weakest assumption — that the reported ranking is not an artifact of the evaluation protocol — is exactly the load-bearing point. The paper's central claim is an empirical superiority claim over strong baselines, and the evidence for that claim is a single best-of-10 number per model with no error bars. The margins are small, especially for HalleluBERT-base (0.34 over XLM-R-large), and best-of-10 selection on a tiny validation set can inflate scores unevenly. The restricted hyperparameter grid, applied uniformly, may also disadvantage some baselines; the authors' admitted failure to reproduce HeRo's published results underscores that the grid is not equally favorable to all models. This concern is not about external consensus or style; it is about whether the reported differences are statistically distinguishable from noise. The proposed concrete test — additional seeds, CIs, and mean-based ranking — directly settles whether the SOTA claim holds. Since the reader already issued a conditional verdict, my stress-test does not change that verdict; it strengthens the condition under which the claim would be acceptable.","tokens_in":9123,"tokens_out":3840,"duration_ms":107955,"concrete_test":"Re-fine-tune HalleluBERT-large, HalleluBERT-base, and XLM-R-large with 10 additional seeds using the same protocol; compute the per-task and overall mean ± std and a paired bootstrap 95% CI on the difference between HalleluBERT and XLM-R-large. Also recompute the ranking using the mean over the 10 runs rather than the best validation run. If the CI includes 0 or the mean-based ranking changes, the central SOTA claim should be downgraded or conditioned on the selection protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's headline comparison rests on a single scalar per model chosen as the best of 10 runs by validation performance, with no standard deviation, confidence interval, or significance test. The decisive margins are small: HalleluBERT-large vs XLM-R-large is 1.46 overall (88.95 vs 87.49), and HalleluBERT-base is only 0.34 ahead of XLM-R-large (87.83 vs 87.49); on SMCD the large-model gap is 1.17. Given SMCD has ~1,700 test instances and NEMO/BMC are small, these differences can easily arise from seed/run noise. Best-of-10 selection amplifies this: with a validation set of only 8% of training data, the selected run's validation score is a noisy estimator, so the associated test score is optimistically biased, and the bias can differ across models if their optimal hyperparameters lie outside the grid (Appendix C: batch {16,32}, five LRs). The authors' own failure to reproduce HeRo's published scores (§5) shows the grid is not neutral for all baselines. Without variance or paired significance testing, the SOTA claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HalleluBERT, a family of Hebrew RoBERTa-style encoders (base and large) pre-trained from scratch on 49.1 GB of deduplicated Hebrew web text and Wikipedia, with a Hebrew-specific byte-level BPE vocabulary. The authors evaluate the models on three native Hebrew benchmarks — BMC and NEMO for named entity recognition, and SMCD for sentiment classification — using the HeRo benchmark suite and an NNI/Huggingface grid-search pipeline. They report that HalleluBERT-large achieves the highest unweighted mean score (88.95) across the three benchmarks, followed by HalleluBERT-base (87.83), surpassing monolingual baselines (HeBERT, AlephBERT, HeRo) and multilingual baselines (mmBERT, XLM-RoBERTa). The models and tokenizer are released under the MIT license.","tokens_in":9458,"tokens_out":2817,"duration_ms":42540,"significance":"If the empirical claims are reliable, this paper makes a useful contribution: it provides the first large-scale Hebrew RoBERTa encoder in both base and large variants, trained on a substantial corpus and evaluated on native benchmarks. The resource release under MIT, the transparent reporting of hyperparameters and training configurations, and the honest limitations section are strengths. The paper also adds evidence for the value of monolingual, scale-oriented pretraining for Hebrew. However, the central claim — state-of-the-art performance over existing baselines — rests on small differences between models under a best-of-10 selection protocol with no variance reporting or significance testing, so the magnitude and stability of the claimed improvements are not yet established.","major_comments":[{"comment":"The headline results are reported as a single number per model: the best score out of 10 runs selected by validation performance, with no standard deviation, confidence interval, or significance test. The decisive margins are small — HalleluBERT-large vs. XLM-R-large is 1.46 points overall (88.95 vs. 87.49), and HalleluBERT-base is only 0.34 points ahead of XLM-R-large (87.83 vs. 87.49). On SMCD the large-model gap is 1.17 points. Given the small sizes of the test sets (SMCD has roughly 1,700 test comments; BMC and NEMO are small) and the validation set being only 8% of the training portion for SMCD, these differences can easily arise from run-to-run noise. Best-of-10 selection amplifies this: selecting the run with the best validation score produces an optimistic, high-variance estimate of test performance, and the bias may differ across models if the restricted hyperparameter grid (bat","section":"§4, Table 1; Appendix C"},{"comment":"The paper states that the authors were unable to reproduce HeRo's reported state-of-the-art results, and in their evaluation HeRo (85.43 overall) falls behind AlephBERT-Gimmel (86.99) and well behind HalleluBERT-base (87.83). This is a load-bearing issue for the SOTA claim: if the adopted evaluation protocol does not reproduce a published baseline's reported performance, that baseline may not be evaluated at its best, and the comparison may be skewed in favor of the new model. The manuscript should clarify whether the HeRo numbers in Table 1 come from the authors' own runs or from the original paper; if from own runs, the authors should show that the hyperparameter grid and evaluation protocol recover the published results when the published hyperparameters or settings are used, or discuss why the discrepancy does not affect the fairness of the comparison.","section":"§5, Table 1"},{"comment":"The downstream evaluation relies on a restricted grid search (batch sizes {16,32} and learning rates {5e-6, 7e-6, 1e-5, 2e-5, 5e-5}) and best-of-10 selection. The paper does not provide evidence that the optimal hyperparameters for each baseline lie within this grid. Given that the authors themselves report failing to reproduce HeRo's published numbers, the grid's neutrality across all models is doubtful. At minimum, the authors should report the distribution of validation and test scores across the 10 runs and show that the selected hyperparameters are not on the grid boundary for any model, or that the conclusions are robust to alternative hyperparameter choices.","section":"Appendix C; §3.4"}],"minor_comments":[{"comment":"Typographical error: 'Squence Length' should be 'Sequence Length'.","section":"Appendix D, title"},{"comment":"'Due to the fairseq implementation single- or mixed-precision training was not employed' is grammatically awkward and could be rephrased for clarity.","section":"§3.3"},{"comment":"The description of the validation split says '10% of the training portion' but the resulting distribution is stated as 'approximately 72% train, 8% validation, and 20% test.' If the official test is 20%, 10% of the 80% training portion would be 8% of the total, so the numbers are consistent, but the wording could be clearer.","section":"§3.4.1"},{"comment":"The figure caption is unclear: 'Top based on a validation at the checkpoints. Bottom based on the validation of each optimization cycle during the training.' Please clarify what is plotted on each panel.","section":"Figure 1"},{"comment":"The sentence beginning 'This contrasts with Scheible et al. (2024), where conservative pre-training learning rates and TPU-based pretraining...' is difficult to follow and seems to describe the authors' own setup rather than a contrast. Please rewrite for clarity.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"This paper is likely to be of interest to the Hebrew NLP community, and the release of a large-scale Hebrew RoBERTa model is valuable. However, the empirical SOTA claim is currently supported only by best-of-10 single-point scores with no variance or significance assessment, and the margins are small. The failure to reproduce HeRo's published results further complicates the baseline comparison. I recommend major revision: the authors should add per-run statistics, significance tests or confidence intervals, and address the baseline-reproduction concern. If the claims are tempered to 'competitive with' instead of 'outperforms,' a more limited publication might be possible, but as written the central claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: HalleluBERT is a real asset for Hebrew NLP, but the headline \"state of the art\" is not yet established. The paper's genuine novelty is a 357M-parameter Hebrew RoBERTa trained from scratch on 49.1GB of deduplicated web text and Wikipedia with a Hebrew-specific byte-level BPE. That fills a concrete gap: until now, no large Hebrew RoBERTa existed. The evaluation covers three native benchmarks (BMC, NEMO, SMCD) and the authors report hyperparameters, compute times, perplexity curves, and sequence-length stats. They also explicitly admit they could not reproduce HeRo's published scores and list limitations (no WWM, no mixed precision, limited grid, no QA/long-context). That honesty is worth something.\n\nThe soft spot is the statistical foundation of the SOTA claim. Table 1 gives each model's best score out of 10 runs selected by validation, with no standard deviation, confidence interval, or significance test. The margins are thin: HalleluBERT-large leads XLM-R-large by 1.46 points on the overall average; HalleluBERT-base leads by 0.34. On datasets with roughly 1,700 test instances (SMCD) and small NER sets, those differences could easily be seed noise, and best-of-N selection makes the reported test score optimistically biased. The authors' own inability to reproduce HeRo's numbers suggests the tuned grid isn't neutral for all baselines. So the paper's central claim is conditional on a stability check it doesn't provide. That is the main thing a referee should push on.\n\nIf the weights are actually released (the abstract says MIT, but the text contains no repository or HuggingFace link — a fixable oversight), this becomes a useful baseline for anyone working on Hebrew. I'd want to see error bars and a direct link before citing the model myself. Still, the contribution is real, the writing is clear, and the authors are transparent about limitations. I'd send it to peer review with a request for variance reporting and a live model link; I wouldn't desk-reject it.","headline":"A genuinely useful large Hebrew RoBERTa resource, but the SOTA claim needs error bars before it should be taken as established.","tokens_in":9916,"tokens_out":2576,"would_cite":false,"duration_ms":23977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HalleluBERT, a Hebrew RoBERTa encoder trained from scratch on 49.1 GB of web and Wikipedia text, outperforms every compared monolingual and multilingual model on native NER and sentiment benchmarks.","keywords":["Hebrew NLP","RoBERTa","pretraining","named entity recognition","sentiment classification","byte-level BPE","monolingual language model","benchmark evaluation"],"falsifier":"Re-running the three benchmarks with every model on the same hyperparameter grid but reporting the full distribution over random seeds (mean, standard deviation, and per-seed best) would settle the matter: if the best-of-ten HalleluBERT-large average falls within one standard deviation of the strongest multilingual baseline, the claimed superiority is not established.","tokens_in":8992,"feed_emoji":"🇮🇱","tokens_out":5410,"duration_ms":46364,"temperature":0.7,"pith_summary":"The paper sets out to show that a Hebrew-specific RoBERTa encoder trained at scale, in both base and large variants, closes the gap left by earlier Hebrew models and multilingual encoders. On the native BMC, NEMO, and SMCD benchmarks, the large model reaches an average of 88.95 across the three tasks, the base model 87.83, and both outperform existing monolingual Hebrew models and multilingual baselines. If correct, this supports the broader claim that fully converged monolingual pretraining with a language-specific tokenizer pays off for a mid-resource language like Hebrew. The work matters because the field lacks a large-scale, reproducible Hebrew encoder: both variants and the tokenizer are released for public use.","feed_headline":"HalleluBERT tops native Hebrew NER and sentiment benchmarks","feed_subtitle":"Large and base variants average 88.95 and 87.83 on the three tasks, beating every compared baseline.","key_machinery":"The load-bearing mechanism is the pretraining recipe: a RoBERTa-style masked-language-model head trained from scratch in full precision for 100k update steps with a global batch size of 8k on about 49.1 GB of deduplicated Hebrew text, using a 52k-subword byte-level BPE vocabulary learned from 20 GB of that text. The argument is that this combination—monolingual data at scale, a Hebrew-specific tokenizer, high batch size, and long training—produces representations that transfer better to downstream Hebrew NER and sentiment than less-trained Hebrew models or multilingual encoders.","core_discovery":"The paper's central claim is that HalleluBERT-large and HalleluBERT-base achieve the highest unweighted mean score across the BMC, NEMO, and SMCD benchmarks among all compared models, with the large variant at 88.95 versus 87.49 for the strongest compared multilingual baseline. The authors attribute this to training a RoBERTa-style encoder from scratch on a 49.1 GB deduplicated Hebrew web and Wikipedia corpus, a Hebrew-specific byte-level BPE vocabulary, high-batch pretraining for about 61 epochs, and scaling to a large variant. They further observe that scaling yields modest but consistent gains on NEMO and sentiment, while the smaller BMC benchmark does not reward the large model.","pith_inferences":["The reported 1.5-point average margin over the strongest multilingual baseline may not be stable: the paper selects the best of ten fine-tuning runs per model without significance tests, so run-to-run noise could shrink the gap if score distributions were compared.","The causal role of the Hebrew-specific tokenizer is inferred rather than demonstrated, since the paper does not quantify its separate impact on compression or downstream performance.","The gains are confined to three tasks; extending the same protocol to question answering, natural language inference, and long-context understanding would show whether the advantage generalizes beyond NER and sentiment.","If the margin survives a distribution-based comparison, it would add to the cross-linguistic pattern that monolingual encoders trained to convergence at moderate scale can beat much larger multilingual encoders on native benchmarks."],"forward_implications":["Fully converged monolingual pretraining can push Hebrew encoder performance past multilingual models despite much larger multilingual training data.","The base variant (126M parameters) is competitive, while the large variant (357M) yields the best average, so scaling helps on two of the three tasks.","A prior Hebrew RoBERTa model may not have been trained long enough or optimally: under the same evaluation protocol, the new base model outperforms it on NER and the overall average.","Releasing both variants and the Hebrew-specific tokenizer under an MIT license lets Hebrew NLP applications start from better representations.","The results suggest that a conservative hyperparameter schedule can still produce strong gains when combined with large-batch, long training."],"fun_headline_variants":["HalleluBERT beats baselines on Hebrew NER and sentiment","Hebrew RoBERTa from scratch tops multilingual encoders","HalleluBERT-large outscores all on Hebrew benchmarks","Hebrew-specific BERT wins NER and sentiment tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's ranking rests on the assumption that selecting the best of ten fine-tuning runs by validation score, with no significance tests or standard deviations, yields fair and stable comparisons across all models; if run-to-run variation exceeds the roughly 1.5-point margins, the state-of-the-art claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["HalleluBERT beats baselines on Hebrew NER and sentiment","Hebrew RoBERTa from scratch tops multilingual encoders","HalleluBERT-large outscores all on Hebrew benchmarks","Hebrew-specific BERT wins NER and sentiment tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1256,"prompt_tokens":652,"completion_tokens":604,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":396,"completion_tokens_details":{"reasoning_tokens":537}},"tokens_in":396,"tokens_out":604,"duration_ms":5650,"temperature":1.0,"reasoning_tokens":537,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:17:11.754250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the three benchmarks with every model on the same hyperparameter grid but reporting the full distribution over random seeds (mean, standard deviation, and per-seed best) would settle the matter: if the best-of-ten HalleluBERT-large average falls within one standard deviation of the strongest multilingual baseline, the claimed superiority is not established.","supporting_citations":[],"review_version":1}