Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark

David M. Smiley

REVIEW 3 major objections 5 minor 22 references

Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-trained transformer models can distinguish known parallel verses from non-parallel verses in Biblical Hebrew, with E5 best at detection and AlephBERT best at avoiding false positives.

desk verdict A genuinely first benchmark of transformers for Biblical Hebrew parallel detection, honestly reported, but the missing lexical-overlap control leaves the central claim unproven. read the letter →

arxiv 2506.24117 v2 pith:RK6XIB3B submitted 2025-06-30 cs.CL

David M. Smiley This is my paper

classification cs.CL

keywords intertextualitybiblicalHebrewparallelpassagestransformerembeddingscosinesimilarityWassersteindistanceChroniclesandSamuel/Kingspre-trainedlanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether off-the-shelf pre-trained transformer models, built for modern languages, can identify passages that the book of Chronicles reuses from Samuel and Kings in Biblical Hebrew. It benchmarks four models on 558 known parallel verse pairs drawn from a standard Hebrew text database, scoring each Chronicles verse against its true parallel and against all non-parallel verses in Samuel/Kings. E5 produces the highest similarity scores for true parallels, while AlephBERT separates parallel from non-parallel distributions more cleanly; both reach F1-scores near 0.88 in a closest-match test. The author takes this as evidence that existing models, without any fine-tuning, can support the discovery of intertextual parallels in ancient Hebrew, and offers the results as a first benchmark for future work.

What carries the argument

The load-bearing object is the verse-level embedding generated by a pre-trained transformer, scored by cosine similarity between vectors. Each Chronicles verse is compared with its known Samuel/Kings parallel and with all other Samuel/Kings verses; the difference between the two score distributions is measured with Wasserstein distance, and a nearest-neighbor step converts the similarities into precision, recall, and F1. The underlying identity is that semantic reuse survives rewording, omission, and insertion of the kind shown in the paper's Table 1, so a contextual embedding should rank a genuine parallel above unrelated verses even when the wording differs.

What would settle it

Expand the nearest-neighbour candidate set for each Chronicles verse from Samuel/Kings only to all remaining verses of the Hebrew Bible; if E5 and AlephBERT precision or recall drops sharply, the claim that they are ready for general parallel detection is shown to depend on the restricted search.

Watch

Extended reading notes

Core claim

The central claim is that pre-trained transformer embeddings encode enough semantics of Biblical Hebrew for cosine similarity to sort parallel from non-parallel verses, even though the models were not trained on ancient Hebrew. On the test set, E5 has a mean cosine similarity of 0.966 for true parallels but 0.882 for non-parallels, while AlephBERT has 0.914 for parallels and 0.638 for non-parallels; Wasserstein distances show AlephBERT separates the two distributions more sharply, and E5 has the best recall in a nearest-neighbor classification. The paper interprets these numbers as complementary strengths rather than a single winner: E5 catches more true parallels, AlephBERT produces fewer false positives, and a combined check-and-balance approach is the recommended use.

Load-bearing premise

The whole evaluation assumes that a model which finds the right parallel among the other verses of Samuel and Kings would also find it among every verse of the Hebrew Bible, but that harder search is never run.

Editorial extensions

If this is right

A scholar could use E5 and AlephBERT as a screening tool to generate candidate parallel verses for manual inspection, narrowing the search space for intertextual study.
Because the models work without fine-tuning, the same embedding-and-similarity pipeline can be carried over to other ancient languages in which parallel corpora exist.
Combining E5's recall with AlephBERT's separation should outperform either model alone, since no single model dominates both detection and false-positive avoidance.
Fine-tuning these models on Biblical Hebrew is a plausible next step that the paper expects to improve accuracy and interpretability beyond the zero-shot results.
The benchmark provides a reference point: future Hebrew-specific models will need to beat these F1 and Wasserstein numbers on the same paired-verse task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark restricts the candidate pool to Samuel/Kings, so the known parallel is always present; applying the same models to a whole-canon search would likely show lower precision and would be the real test of unknown-parallel discovery.
AlephBERT's sharper separation suggests that modern-Hebrew morphological knowledge transfers to ancient Hebrew better than multilingual scale does, but the paper does not test how much of E5's recall comes from lexical overlap rather than semantic understanding.
A natural extension would be to reuse the same evaluation on other known biblical parallels, such as between Isaiah and Kings or between the Psalter and Chronicles, to see whether the observed thresholds transfer across genres.
Given E5's high scores for non-parallel verses, an editor assembling a digital tool should tune a similarity threshold carefully or use AlephBERT as the precision gate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper is exactly what its title says: a first benchmark of four pre-trained transformer models for detecting known Samuel/Kings–Chronicles parallels in Biblical Hebrew. That is a modest but real contribution. The ground truth is external (Enders et al.), no parameters are fitted, the models are genuinely pre-trained, and the code and data are promised on GitHub. The writing is clear and the limitations section is honest about genre scope and the black-box issue.

The narrow empirical finding holds up: within a closed set of Samuel/Kings verses, E5 and AlephBERT assign higher cosine similarity to known parallel pairs than to non-parallels, and the nearest-neighbor results beat chance. The Wasserstein distance adds a useful second view. I believe those numbers.

The soft spot is the one the stress-test flags: there is no lexical-overlap baseline. The paper motivates transformers by saying rule-based methods rely too heavily on surface-level lexical matching, but it never runs a TF-IDF, word-overlap, or n-gram baseline on the same pairs. That matters because the known parallels are textual reuse with substantial exact wording, and the candidate pool is restricted to Samuel/Kings, so the true parallel is always in the list. A surface-matching baseline would likely assign high similarity to the true parallel and low similarity to unrelated verses, producing F1 scores in the same range. Without that control, the p-values and F1 scores in Tables 2 and 4 do not show that contextual representations enhance detection; they are consistent with the models just recovering lexical identity.

Two smaller issues. First, the t-tests treat all non-parallel pairs as independent, but each Chr verse contributes many non-parallel similarities, so the sample sizes are inflated and the p-values are meaningless as significance claims. Second, the evaluation is restricted to one source pair; the actual use case of finding parallels anywhere in the Hebrew Bible is not tested. The limitations section acknowledges the narrative-only scope but does not mention this closed-search issue or the missing baseline.

None of this makes the paper worthless. For a digital-humanities reader who wants to know whether off-the-shelf embeddings are worth trying, this is a useful starting point, and the limitations are stated more candidly than in most papers. But the central claim—that transformers do better than lexical methods—is unestablished.

I'd send it to peer review because it's a legitimate first dataset and benchmark, and the flaw is fixable: add a lexical baseline, run a whole-canon retrieval experiment, and report proper statistical tests. A referee who asks for those will get a much stronger paper. I wouldn't cite it for the transformer claim, but I might cite it as the first benchmark if I were working in this niche.

Referee Report

3 major / 5 minor

Summary. This paper evaluates four pre-trained transformer models (E5, AlephBERT, MPNet, LaBSE) as off-the-shelf encoders for detecting known parallel passages between Samuel/Kings and Chronicles in the Hebrew Bible. For each Chronicles verse, the author computes cosine similarity against the known parallel and against non-parallel Samuel/Kings verses, compares parallel and non-parallel similarity distributions with a t-test and Wasserstein distance, and reports nearest-neighbor classification metrics. The paper concludes that E5 and AlephBERT are promising for parallel detection, with E5 favoring recall of parallels and AlephBERT favoring separation from non-parallels, and claims this is the first comprehensive benchmark of transformer-based models for biblical parallel detection. Section 7 acknowledges limitations regarding genre coverage, interpretability, and the lack of a BH-specific model.

Significance. If the central claim were fully supported, the paper would be a useful empirical contribution to digital biblical studies: it uses an external scholarly ground truth (Enders et al.), the evaluation is parameter-free in the sense that no model is fine-tuned, and the code and data are promised in a public repository. The comparative ranking of four models on a new ancient-language task is also of interest to the NLP community. However, the significance is conditional on establishing that the transformer embeddings add value over simple surface matching. The manuscript motivates transformers by contrasting them with rule-based and lexical methods that rely on surface matching, but it never includes a lexical baseline, and the candidate pool is restricted to Samuel/Kings so that every true parallel is always present. As a result, the reported p-values and F1 scores currently are consistent with the possibility that the models are only recovering lexical identity rather than semantic relationships.

major comments (3)

[5.4, Tables 1 and 4] The closest-match evaluation searches only within the Samuel/Kings candidate pool, and by construction (§4.1) every Chronicles verse has its known parallel in that pool. Because the known parallels in this corpus are textual reuses that retain substantial verbatim wording (see the 2 Sam 7:14 / 1 Chr 17:13 example in Table 1), a simple lexical baseline such as word-overlap Jaccard, TF-IDF cosine similarity, character n-gram overlap, or Levenshtein-based matching would likely also assign the highest score to the true parallel and low scores to unrelated verses. The paper reports no such baseline. Consequently, the p-values in Table 2 and the F1 scores in Table 4 do not establish that contextual transformer representations enhance detection over surface-level lexical matching, which is exactly the comparison the introduction in §2 uses to motivate the study. At minimum, one or more lexical baselines must be evaluated on the same candidate pool and the same ground-truth set.
[3, 5.2] The non-parallel distribution is defined as Samuel/Kings verses that are not known parallels of a given Chronicles verse, so the evaluation measures discrimination within a single source pair, not detection of parallels anywhere in the Hebrew Bible. The stated objective in §3 is to estimate suitability for finding unknown connections traditionally identified through manual methods, and §6 projects generalization to the whole canon and to other ancient languages. With the current design, every true parallel is always in the candidate list, and the distractors come from one book pair rather than from the full range of biblical genres (poetry, wisdom, law, prophecy). This setup can overstate detection ability for a canon-wide search task. A more appropriate test would include non-parallel candidates from other books or genres, or at least a held-out set of books, and report whether the known parallel is ranked at the top of that extended pool.
[5.2, Table 2] The t-test in Table 2 treats all non-parallel cosine pairs as independent observations, but the 558 Chronicles verses each contribute many non-parallel scores computed from the same verse vector; the observations are therefore clustered and the reported p-values are artificially extreme. A paired analysis by Chronicles verse, or a bootstrap resampled at the verse level, would give statistically defensible significance statements. The qualitative conclusion that parallel scores exceed non-parallel scores would likely survive such a test, but the p-values as reported (e.g., 1.06e-284 for E5) are not reliable evidence on their own.

minor comments (5)

[1, Abstract] The text contains multiple missing spaces between words, e.g., "potentialofpre-trained" in §1 and "oftransformer" in the abstract; the manuscript needs a careful proofreading pass.
[4.1, References] Section 4.1 attributes the BHSA corpus to reference [19] (van Peursen and Talstra, 2007), but the database citation is reference [20] (van Peursen, Sikkel, and Roorda, 2015); the citation should be corrected.
[5.4] The classification section reports micro-averaged precision, recall, and F1 but does not state the exact decision rule used to label a prediction as a true positive, e.g., whether a Chronicles verse is counted as correctly detected only when its maximum-cosine Samuel/Kings verse is the known parallel; the rule should be stated explicitly for reproducibility.
[5.3, Table 3] Table 3 would be easier to read if the models were ordered by Wasserstein distance, since the text then consistently compares the lowest and highest values without referring to row position.
[7] The limitations section is candid about genre coverage, opacity of embeddings, and the absence of a BH-specific model, but it omits the lexical-baseline limitation and the closed-candidate-pool limitation that directly affect the interpretation of the reported metrics.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark compares off-the-shelf embeddings against an external gold standard with no fitted parameters.

full rationale

The paper's derivation chain is self-contained and empirically grounded. The ground-truth parallel set is taken from an external scholarly source (Enders et al., Chronicles and Its Synoptic Parallels in Samuel, Kings, and Related Biblical Texts), not derived from the models' outputs. The transformer embeddings are used as pre-trained, off-the-shelf encoders, and no parameter is fitted to the evaluation data or to the known-parallel labels, so the reported similarities, p-values, Wasserstein distances, and F1 scores are genuine measurements rather than constructed equivalences. The definition of non-parallel passages as the complement of the known-parallel list is a standard evaluation convention and does not make the detection result circular; at most, it assumes the external list's completeness, which is a corpus-validity limitation, not a circularity. The absence of a lexical-overlap baseline is a meaningful correctness concern about whether the transformer advantage is established, but it is not a circularity: the paper does not claim to derive transformer performance from lexical overlap, nor does it fit a model to the answer. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled via citation appears. The central empirical claim, that E5 and AlephBERT produce embeddings that separate known parallel from non-parallel verses, is directly tested against external data and therefore is not equivalent to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the accuracy and completeness of the external parallel list (Enders et al.), the validity of cosine similarity as a proxy for intertextual reuse, and the assumption that models pre-trained on modern or multilingual text are usable for Biblical Hebrew without adaptation. No parameters are fitted, and no new entities are introduced.

assumptions (4)

domain assumption The 558 recognized parallels from the Enders et al. synopsis are correct and complete ground truth.
Section 4.1 adopts this list without independent verification or discussion of its criteria.
domain assumption Cosine similarity between pooled verse embeddings is a valid measure of textual parallelism in Biblical Hebrew.
Section 5.1 posits that cosine captures semantics despite additions and omissions, but this connection to the philological notion of parallel passages is asserted, not tested.
domain assumption Pre-trained tokenizers and models are adequate for Biblical Hebrew without any fine-tuning or Hebrew-specific preprocessing.
Section 4.2 selects models trained on modern Hebrew or multilingual data; the paper notes their BH performance is untested.
domain assumption Verses in the non-parallel set (complement of known parallels) are truly unrelated to each Chr verse, so the non-parallel distribution is uncontaminated by undiscovered parallels.
Section 5.2 defines non-parallels as all Sam/Kgs verses not in the known parallel list, assuming the list is exhaustive.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark." pith.science (2026). https://pith.science/paper/RK6XIB3B

@misc{pith2026250624117,
  author       = {Pith},
  title        = {Pith review of: Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RK6XIB3B}},
  note         = {Machine review of arXiv:2506.24117}
}

read the original abstract

Identifying parallel passages in biblical Hebrew (BH) is central to biblical scholarship for understanding intertextual relationships. Traditional methods rely on manual comparison, a labor-intensive process prone to human error. This study evaluates the potential of pre-trained transformer-based language models, including E5, AlephBERT, MPNet, and LaBSE, for detecting textual parallels in the Hebrew Bible. Focusing on known parallels between Samuel/Kings and Chronicles, I assessed each model's capability to generate word embeddings distinguishing parallel from non-parallel passages. Using cosine similarity and Wasserstein Distance measures, I found that E5 and AlephBERT show promise; E5 excels in parallel detection, while AlephBERT demonstrates stronger non-parallel differentiation. These findings indicate that pre-trained models can enhance the efficiency and accuracy of detecting intertextual parallels in ancient texts, suggesting broader applications for ancient language studies.

Figures

Figures reproduced from arXiv: 2506.24117 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 17 canonical work pages

[1]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019) “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 4171–4186.https://doi.org/10.18653/ v1/N19-1423

work page 2019
[2]

(1998).Chronicles and Its Synoptic Parallels in Samuel, Kings, and Related Biblical Texts

Enders, J., Millar, W., & Burns, J. (1998).Chronicles and Its Synoptic Parallels in Samuel, Kings, and Related Biblical Texts. The Liturgical Press. 10

work page 1998
[3]

Language-agnostic BERT Sentence Embed- ding

Feng, F., Yang, Y., Cer, D., et al., (2022). Language-agnostic BERT Sentence Embed- ding. https://doi.org/10.48550/arXiv.2007.01852

arXiv doi:10.48550/arxiv.2007.01852 2022
[4]

Fewell, D. (1992). Reading between Texts: Intertextuality and the Hebrew Bible . Louisville: Westminster John Knox Press

work page 1992
[5]

(1985).Biblical Interpretation in Ancient Israel

Fishbane, M. (1985).Biblical Interpretation in Ancient Israel. New York: Oxford Press

work page 1985
[6]

Gomez, J., and Vázquez, P. (2022). An Empirical Evaluation of Document Embeddings and Similarity Metrics for Scientific Articles.Applied Sciences, 12(11), 5664. https: //doi.org/10.3390/app12115664

work page doi:10.3390/app12115664 2022
[7]

(2004).Retelling the Torah: The Deuteronomistic Historian’s Use of Tetra- teuchal Narratives

Harvey, J. (2004).Retelling the Torah: The Deuteronomistic Historian’s Use of Tetra- teuchal Narratives. New York: Bloomsbury

work page 2004
[8]

Kalimi, I.(2004).The Reshaping of Ancient Israelite History in Chronicles.Eisenbrauns

work page 2004

Show all 22 references

[9]

doi.org/10.2139/ssrn.4368733

Leo, J., Ge, E., and Li, S., (2023) Wasserstein Distance in Deep Learning.http://dx. doi.org/10.2139/ssrn.4368733

2023 doi
[10]

Mars, M. (2022). From Word Embeddings to Pre-Trained Language Models: A State- of-the-Art Walkthrough.Applied Sciences, 12(17), 8805. https://doi.org/10.3390/ app12178805

2022
[11]

Miller, G. (2011). Intertextuality in Old Testament Research.Currents in Biblical Re- search, 9, pp. 283–309

2011
[12]

Rainio, O., Tueho, J., and Klén, R. (2024). Evaluation Metrics and Statistical Tests for Machine Learning.Scientific Reports, 14, 6086. https://doi.org/10.1038/ s41598-024-56706-x

2024
[13]

(2021).Old Testament Use of the Old Testament

Schnittjer, G. (2021).Old Testament Use of the Old Testament. Grand Rapids: Zon- dervan

2021
[14]

AlephBERT: A Hebrew Large Pre- Trained Language Model to Start-off your Hebrew NLP Application With

Seker, M., Bandel, E., Barekat, D., et al., (2021). AlephBERT: A Hebrew Large Pre- Trained Language Model to Start-off your Hebrew NLP Application With. In Pro- ceedings of the 2022 Conference on Computational Natural Language Processing, pp. 321–330. https://doi.org/10.48550/...

2021 doi arXiv
[15]

Shmidman, A., Koppel, M., and Porat, E. (2018). Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus.Journal of Data Mining and Digital Human- ities. https://doi.org/10.46298/jdmdh.1388

2018 doi
[16]

Shmidman, A. (2022). Automatic Identification of Biblical Citations and Allusions in Hebrew Texts. InJewish Studies in the Digital Age, pp. 335-348. https://doi.org/ 10.1515/9783110744828-015

2022 doi
[17]

Singh, N., and Habash, N. (2012). Hebrew Morphological Preprocessing for Statistical Machine Translation. InProceedings of the 16th EAMT Conference, pp. 43-50. 11

2012
[18]

InProceedings of the 34th International Conference on Neu- ral Information Processing Systems, pp

Song, K., Tan, X., Qin, T., etal., (2020).MPNet: MaskedandPermutedPre-trainingfor Language Understanding. InProceedings of the 34th International Conference on Neu- ral Information Processing Systems, pp. 16857–16867. https://doi.org/10.48550/ arXiv.2004.09297

2020 doi arXiv
[19]

van Peursen, W., and Talstra, E. (2007). Computer-Assisted Analysis of Parallel Texts in the Bible.Vetus Testamentum, 57(1), pg. 45–72

2007
[20]

van Peursen, W., Sikkel, C., and Roorda, D. (2015). Hebrew Text Database ETCBC4b. https://doi.org/10.17026/dans-z6y-skyh

2015 doi
[21]

Attention is All You Need

Vaswani, A., Shazeer, N., Parmar, N., et al., (2017). Attention is All You Need. In Advances in Neural Information Processing Systems 30(NIPS 2017), pp. 5998–6008. https://doi.org/10.48550/arXiv.1706.03762

2017 doi arXiv
[22]

Multilingual E5 Text Embeddings: A Technical Report.https://doi.org/10.48550/arXiv.2402.05672 12

Wang, L., Yang, N., Huang, X., et al., (2024). Multilingual E5 Text Embeddings: A Technical Report.https://doi.org/10.48550/arXiv.2402.05672 12

2024 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019) “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 4171–4186.https://doi.org/10.18653/ v1/N19-1423

work page 2019

[2] [2]

(1998).Chronicles and Its Synoptic Parallels in Samuel, Kings, and Related Biblical Texts

Enders, J., Millar, W., & Burns, J. (1998).Chronicles and Its Synoptic Parallels in Samuel, Kings, and Related Biblical Texts. The Liturgical Press. 10

work page 1998

[3] [3]

Language-agnostic BERT Sentence Embed- ding

Feng, F., Yang, Y., Cer, D., et al., (2022). Language-agnostic BERT Sentence Embed- ding. https://doi.org/10.48550/arXiv.2007.01852

arXiv doi:10.48550/arxiv.2007.01852 2022

[4] [4]

Fewell, D. (1992). Reading between Texts: Intertextuality and the Hebrew Bible . Louisville: Westminster John Knox Press

work page 1992

[5] [5]

(1985).Biblical Interpretation in Ancient Israel

Fishbane, M. (1985).Biblical Interpretation in Ancient Israel. New York: Oxford Press

work page 1985

[6] [6]

Gomez, J., and Vázquez, P. (2022). An Empirical Evaluation of Document Embeddings and Similarity Metrics for Scientific Articles.Applied Sciences, 12(11), 5664. https: //doi.org/10.3390/app12115664

work page doi:10.3390/app12115664 2022

[7] [7]

(2004).Retelling the Torah: The Deuteronomistic Historian’s Use of Tetra- teuchal Narratives

Harvey, J. (2004).Retelling the Torah: The Deuteronomistic Historian’s Use of Tetra- teuchal Narratives. New York: Bloomsbury

work page 2004

[8] [8]

Kalimi, I.(2004).The Reshaping of Ancient Israelite History in Chronicles.Eisenbrauns

work page 2004