REVIEW 3 cited by
HeRo: RoBERTa and Longformer Hebrew Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
HeRo: RoBERTa and Longformer Hebrew Language Models
read the original abstract
In this paper, we fill in an existing gap in resources available to the Hebrew NLP community by providing it with the largest so far pre-train dataset HeDC4, a state-of-the-art pre-trained language model HeRo for standard length inputs and an efficient transformer LongHeRo for long input sequences. The HeRo model was evaluated on the sentiment analysis, the named entity recognition, and the question answering tasks while the LongHeRo model was evaluated on the document classification task with a dataset composed of long documents. Both HeRo and LongHeRo presented state-of-the-art performance. The dataset and model checkpoints used in this work are publicly available.
Forward citations
Cited by 3 Pith papers
-
HalleluBERT: Let Every Token That Has Meaning Bear Its Weight
HalleluBERT, a Hebrew-only RoBERTa encoder family trained from scratch at scale, reports the highest unweighted mean scores on BMC, NEMO, and SMCD benchmarks, but without statistical significance testing.
-
Hierarchical Section Matching Prediction (HSMP) BERT for Fine-Grained Extraction of Structured Data from Hebrew Free-Text Radiology Reports in Crohn's Disease
HSMP-BERT, using a section-matching pretraining objective and hierarchical prompts, extracts organ-finding labels from Hebrew radiology reports with mean F1 0.83 and 5.1x faster inference.
-
The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
ChristBERT models trained on a 13.5GB German medical corpus via continued pre-training, from-scratch training, and vocabulary adaptation outperform prior general and medical German models on four of five NER and class...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.