Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

A new benchmark gives Latin intertextuality detection a standardized testbed: 545 expert-verified parallels in ~172k text segments, plus baseline retrieval and classification results.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:01 UTC pith:RDJ4MY3H

load-bearing objection Useful benchmark, real resource, but the missing lexical baseline leaves the semantic-similarity claim untested — and the stress-test overstates the n-gram bias. the 3 major comments →

arxiv 2601.07533 v2 pith:RDJ4MY3H submitted 2026-01-12 cs.IR cs.CLcs.DL

Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature

Julian Schelb , Michael Wittweiler , Marie Revellio , Barbara Feichtinger , Andreas Spitz This is my paper
classification cs.IR cs.CLcs.DL
keywords Latin literatureintertextuality detectionbenchmark datasettext reuseinformation retrievallanguage modelsclassical philologydense retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper addresses a concrete gap: computational methods for detecting intertextuality in Latin literature exist, but no standardized benchmark lets researchers compare them. Loci Similes is that benchmark—a curated corpus of ~172k text segments from Late Antique authors (Jerome, Lactantius) and canonical classical authors (Virgil, Cicero, Ovid, and others), together with 545 expert-verified intertextual links. The authors also define an evaluation framework that treats detection as a whole-document retrieval and alignment problem, scored by error-based metrics that account for the huge number of true negatives. Their baselines show a two-stage retrieve-and-rerank pipeline can capture roughly 79% of references at a retrieval depth of 100 while cutting a scholar's manual review workload by about 99%. If the benchmark holds, it gives the field a common ruler and a concrete target for models that go beyond verbatim matching.

Core claim

The central claim is that Loci Similes provides the first standardized resource for evaluating intertextuality detection in Latin. The authors show that a modern pipeline—dense retrieval with a multilingual embedding model followed by cross-encoder reranking—substantially outperforms a retrieval-only baseline in precision and error rates, recovering most true links while discarding the vast majority of candidates. They argue that this demonstrates both the value of semantic (rather than purely lexical) matching and the remaining open problem: distinguishing genuine allusions from coincidental overlap of common phrases.

What carries the argument

The load-bearing object is the benchmark itself: a query corpus of ~83k segments from Jerome and Lactantius, a source corpus of ~88k segments from ten classical authors, and a ground-truth set of 545 directional intertextual links. Around it, the paper builds an evaluation framework that frames detection as segment-wise retrieval between whole documents and defines error-based metrics (Segment-Misclassification Rate, Global False-Positive Rate, Global False-Negative Rate) normalized by total pairs, appropriate for a setting where true negatives vastly outnumber positives. The baseline system is a retrieve-and-rerank pipeline: a bi-encoder generates candidate source passages by cosine similar

Load-bearing premise

The 545 ground-truth links are treated as a representative sample of intertextuality in the corpus, but half were derived from a rule-based n-gram pipeline that favors lexical overlap; if subtle, lexically-distinct allusions are under-represented, the benchmark measures mostly verbatim quote and paraphrase detection rather than the full phenomenon.

What would settle it

Assemble a held-out test set of well-documented Jerome and Lactantius allusions that share no n-grams with their sources (e.g., thematic echoes identified by philologists). If the current best models perform near chance on this set while scoring well on the existing ground truth, the paper's claim that the benchmark captures semantic similarity beyond lexical matching would be refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can now benchmark new intertextuality-detection methods against a fixed dataset instead of ad-hoc collections, making progress measurable.
  • The framework shifts practical workflows: at k=100, a scholar reviews ~780 candidate passages to find 85 genuine links, versus ~93,700 for retrieval-only—a ~99% workload reduction.
  • The dataset's taxonomy (verbatim quote, paraphrase, allusion) provides a way to analyze where automated methods succeed and fail along the spectrum of reuse.
  • The error-based metrics offer a template for other retrieval tasks with extreme class imbalance, e.g., historical text reuse in other languages.
  • The results indicate that large multilingual models outperform Latin-specific ones, at least with current fine-tuning, guiding future model selection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ground truth's half-and-half construction—prior scholarship plus a rule-based n-gram pipeline—may make the benchmark lexically biased; a future version should add a separate set of allusions with minimal lexical overlap to test true semantic generalization.
  • The same dataset could support related tasks beyond retrieval, such as automatically classifying links into verbatim/paraphrase/allusion categories, potentially refining theories of intertextuality.
  • The pipeline itself could be applied to other low-resource historical language pairs if new ground truth is provided, making this a reusable evaluation infrastructure rather than a Latin-only artifact.
  • A testable extension: incorporating the annotators' 'conduit function' criterion—whether the reused phrase imports meaning from the source—as an explicit training signal might improve precision beyond the current best F1 of 0.5.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Loci Similes, a benchmark for Latin intertextuality detection comprising ~172k text segments and 545 expert-verified parallels linking Late Antique authors (Jerome, Lactantius) to classical Latin authors. The ground truth is constructed from two sources: 270 instances from prior scholarship (Schropp et al. 2024b) and 275 instances identified via a rule-based n-gram matching pipeline and then manually verified by domain experts. The authors propose an evaluation framework with error-based metrics (SMR, FPR, FNR) and baseline experiments for dense retrieval, binary classification, and a retrieve-and-rerank pipeline using multilingual and Latin-specific language models. They report that E5-large performs best for retrieval (Recall@10 = 0.61) and XLM-RoBERTa Large for classification (F1 = 0.50), and that retrieve-and-rerank substantially reduces false positives compared to retrieval-only, yielding a 99% reduction in manual review workload while retaining nearly 80% of true links.

Significance. If the benchmark is taken at face value, it fills a genuine gap by providing the first standardized evaluation resource for computational intertextuality detection in Latin, a task of direct relevance to classical philology and digital humanities. The authors ship the dataset, the open-source locisimiles package, and a GUI, which are concrete contributions that will enable future comparisons. The expert annotation process and the explicit acknowledgement of non-exhaustive ground truth and labeling ambiguity are strengths. The evaluation framework, with its focus on whole-document comparison and error-based metrics, is well motivated for the philological workflow. However, the benchmark's validity hinges on the representativeness of the ground truth and on whether the experiments demonstrate capability beyond lexical matching; these points require substantial revision.

major comments (3)
  1. [Section 3.2 / Abstract / Conclusion] The ground-truth construction procedure guarantees that every positive pair shares lexical overlap. The 275 additional instances are explicitly identified via a two-word n-gram matching pipeline, and the 270 instances from Schropp et al. (2024b) derive from the same group's earlier n-gram-based approaches (Section 2.1). Thus the benchmark cannot contain allusions with zero lexical overlap, yet the abstract and Section 9 claim it evaluates 'semantic similarity beyond exact lexical matching.' This sampling bias undermines the central claim. Please either supplement the ground truth with non-lexical-overlap parallels (e.g., from scholarly indices referencing thematic or semantic echoes), or explicitly reposition the benchmark as covering lexically grounded intertextuality.
  2. [Section 7 / Tables 3-4] No lexical baseline is provided. Since all positives satisfy a lexical-overlap condition, a sparse retriever (e.g., Tesserae two-word overlap, lemma-overlap, or BM25) is a necessary control. Without it, the reported recall of E5-large (0.61 at Recall@10) is uninterpretable as evidence of semantic capability. Please add at least one non-neural lexical baseline and compare it against the dense models under the same evaluation protocol.
  3. [Appendix B.1] The metric definitions are internally inconsistent. The formulas SMR=(FP+FN)/N, FPR=FP/N, FNR=FN/N define global rates over all pairs, but the text states 'We calculate these metrics individually for each query segment and report the mean value averaged over all queries.' These two cannot both be true. If the metrics are global, remove the per-query sentence; if per-query, provide the correct per-query formulas. This affects the validity of Tables 3, 5, and 6.
minor comments (6)
  1. [Section 3.2] The construction of the 270 instances from Schropp et al. (2024b) is not described in detail; specify how that dataset was created and whether it also relies on n-gram candidate generation.
  2. [Section 6.1] The selection of the 937 query and 880 source segments used for evaluation is not described. Please specify the sampling procedure to ensure reproducibility and demonstrate that the subset is representative of the full corpus.
  3. [Tables 5 and 6] The confusion matrices appear inconsistent with the reported FPR/SMR values. For example, in Table 5, the random-pairs row shows FP=831 and N≈1,061,646, which would give FPR≈0.00078, not the reported 0.0491. Please verify the numbers.
  4. [Abstract] The phrase 'comprising of' is nonstandard; use 'comprising' or 'consisting of'.
  5. [Figure 5] The '# Classifications' line is included in the legend but the caption does not explain which axis it uses. If it is on a secondary y-axis, label the axes accordingly.
  6. [Section 5.1] The 'Online Contrastive Loss' is mentioned without a reference. If this is a standard loss from a specific source, cite it; otherwise, describe the loss function.

Circularity Check

0 steps flagged

No circularity in the evaluation logic; the self-sourced ground-truth provenance is a benchmark-validity caveat, not a derivation-level loop.

full rationale

The paper's quantitative claims are not derived from the data-generation rule. The 545 positive pairs are expert-verified (Sections 3.2-3.3), and the retrieval/classification baselines are trained on folds and evaluated on held-out positive/negative pairs (Section 6.1), so no fitted parameter is renamed as a prediction and no reported metric is forced by construction. The self-citations to Schropp et al. (2024a,b) and Revellio (2022) provide an earlier dataset and a candidate-generation pipeline, but the pipeline outputs were independently screened by four Latin experts, and the prior dataset entries were also reviewed and partly rejected by the present annotators. These citations are not invoked as uniqueness theorems or as a substitute for the benchmark content. The skeptic's concern - that all 275 added positives descend from a shared-token n-gram matcher, so the benchmark may overrepresent lexical overlap and lacks a lexical baseline - is a legitimate validity/coverage limitation, acknowledged only partially in the Limitations section ('the labeled dataset likely omits some valid references'). That concern affects what the benchmark measures, but it does not make any derivation equivalent to its input. Hence score 1 rather than 0 only as a nod to the heavily self-sourced provenance of the ground truth.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

No free parameters are fitted to produce the central claim; the benchmark is a dataset. The key axioms are the trustworthiness of expert annotation and the sentence-level segmentation. No new theoretical entities are introduced.

axioms (2)
  • domain assumption Expert annotations determine true intertextuality based on the three criteria (uncommon vocabulary, attested frequency, conduit function).
    Section 3.3: these criteria are used to validate candidate links; if the criteria are wrong or inconsistently applied, the ground truth is wrong.
  • domain assumption The segmentation at 'approximate sentence level' preserves the units of intertextuality.
    Section 3.2: consolidating poetic verses into sentence-like segments may merge or split references, affecting evaluation.

pith-pipeline@v1.3.0-alltime-deepseek · 19383 in / 9000 out tokens · 85436 ms · 2026-08-03T11:01:49.185920+00:00 · methodology

0 comments
read the original abstract

Tracing connections between historical texts is an important part of intertextual research, enabling scholars to reconstruct the virtual library of a writer and identify the sources influencing their creative process. These intertextual links manifest in diverse forms, ranging from direct verbatim quotations to subtle allusions and paraphrases disguised by morphological variation. Language models offer a promising path forward due to their capability of capturing semantic similarity beyond lexical overlap. However, the development of new methods for this task is held back by the scarcity of standardized benchmarks and easy-to-use datasets. We address this gap by introducing Loci Similes, a benchmark for Latin intertextuality detection comprising of a curated dataset of ~172k text segments containing 545 expert-verified parallels linking Late Antique authors to a corpus of classical authors. Using this data, we establish baselines for retrieval and classification of intertextualities with state-of-the-art LLMs.

Figures

Figures reproduced from arXiv: 2601.07533 by Andreas Spitz, Barbara Feichtinger, Julian Schelb, Marie Revellio, Michael Wittweiler.

Figure 1
Figure 1. Figure 1: Example of intertextual reference. Reuse of a classic Vergilian phrase for speechlessness by Jerome. While retaining the semantic core, the author alters the word order to adapt the expression to a different context. Recovering such textual reuses is not merely a matter of identifying sources. It facilitates research on broader cultural-historical phenomena (Tangher￾lini and Chen, 2024). In particular, it … view at source ↗
Figure 2
Figure 2. Figure 2: Spectrum of intertextuality. References manifest in diverse forms, spanning from easily de￾tectable verbatim quotations to adapted paraphrases and subtle allusions where only a semantic core remains. cross-lingual reuse in English-Urdu news articles by combining machine translation with classifiers. 2.3 Further Related Tasks The detection of intertextualities shares concepts with other NLP tasks that ident… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of confirmed references. Man￾ually verified intertextual links in the annotated dataset by citing author (Jerome, Lactantius) and source author (including Virgil, Cicero, and others). and reused material. Consequently, we consoli￾dated multiple entries into a single instance in cases where citations span two or more verses. Further￾more, our expert annotators rejected links from pre￾vious scho… view at source ↗
Figure 4
Figure 4. Figure 4: Retrieve-and-rerank pipeline. Stage 1: The input text segment acts as a query to retrieve potential candidates from the database. Stage 2: To verify the ref￾erence, the query and source candidate are concatenated into a single input sequence to train a binary classifier. we generate contrastive examples by augmenting the assigned positive pairs with n negative samples. In the evaluation phase, we reconstru… view at source ↗
Figure 5
Figure 5. Figure 5: Performance vs. efficiency trade-off in the retrieve-and-rerank pipeline. The pipeline first generates k candidates via embedding cosine similarity, followed by a binary classification stage to label pairs as Reference or No Reference. We compare this against a “Retrieval Only” baseline where the top k candidates are treated as positive predictions, mimicking a scholar manually reviewing the top results. R… view at source ↗
Figure 6
Figure 6. Figure 6: Recall@k comparison across embedding model families. Performance of E5, Granite, BGE-M3, and the domain-specific SPhilBerta on the Latin intertextuality retrieval task. We observe that larger models show better performance, with multilingual E5-large achieving the highest scores overall. Error bands show ±1 std across 5-fold cross-validation. @1 @5 @10 @20 @100 @1000 @10000 k 0.0 0.2 0.4 0.6 0.8 1.0 RECALL… view at source ↗
Figure 7
Figure 7. Figure 7: Recall@k performance of the best embedding model across individual folds. We show recall at varying cutoff values (k ∈ {1, 5, 10, 20, 100, 1000, 10000}) for the top-performing model on the Latin intertextuality retrieval task. Each bar represents a single fold from 5-fold cross-validation, with the orange line indicating mean ± std. Performance improves substantially with larger k, approaching near-perfect… view at source ↗
Figure 8
Figure 8. Figure 8: Impact of training data imbalance on retrieval. Heatmap showing recall, MAP, MRR, and NDCG scores at different cutoff values (@10, @100, @1000) across negative sampling proportions (1:1 to 1:10). Values are averaged over 5 cross-validation folds. Colors are normalized per column to highlight relative performance differences. Best values per metric are shown in bold with an orange underline. Higher ratios o… view at source ↗
Figure 9
Figure 9. Figure 9: Hyperparameter sensitivity for the dense retriever. We visualize retrieval metrics (recall, MRR, MAP, and NDCG at k = 1000) across varying learning rates and training epochs. Results are averaged across 5 folds, where annotated values represent the mean score ± standard deviation. 1:1 1:2 1:3 1:4 1:5 1:6 1:7 1:8 1:9 1:10 Negative Sampling Proportion 0.0 0.2 0.4 0.6 Value F1 Score Across Negative Sampling P… view at source ↗
Figure 10
Figure 10. Figure 10: Impact of training data imbalance on classification performance. The plot illustrates the recall metric on the evaluation set as the ratio of negative-to-positive training samples is increased from 1:1 to 1:10. Shaded areas represent the standard deviation across the 5 cross-validation folds [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Hyperparameter sensitivity analysis for the classification model. We visualize key performance metrics (F1 score, accuracy, precision, and recall) across varying learning rates and training epochs. Results are averaged across 5 folds, where annotated values represent the mean score ± standard deviation [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Graphical user interface workflow. (A) Data Upload: Users ingest query and source documents via CSV files. (B) Configuration: The pipeline is customized by selecting pre-trained models and tuning the retrieval depth (top-k) and classification confidence threshold. (C) Result Exploration: The dashboard presents query segments alongside retrieved source candidates, displaying both cosine similarity and clas… view at source ↗
Figure 13
Figure 13. Figure 13: Example references. Three instances of text reuse by Jerome included in the ground truth dataset [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories

    cs.CL 2026-07 conditional novelty 8.0

    A tool-constrained LLM extracts span-grounded, typology-labeled intertextual pairs; expert-adjudicated validation and a 65,380-comparison run across the Twenty-Four Histories yield stable citation composition but decl...

  2. Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke

    cs.CL 2026-05 unverdicted novelty 7.0

    Semantic search retrieves substantially more implicit receptions of Locke's work than lexical baselines in 18th-century corpora, yet remains constrained by lexical gatekeeping.

  3. From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages

    cs.CL 2026-07 conditional novelty 6.0

    Unsupervised TSDAE and CSE adaptation of specialized Latin/Greek LMs yields corpus-specific sentence encoders that outperform multilingual, distilled, and supervised baselines on biblical reuse detection and retrieval.

  4. Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke

    cs.CL 2026-05 unverdicted novelty 5.0

    Semantic search retrieves substantially more implicit receptions of Locke's ideas than lexical methods but remains partly limited by surface vocabulary overlap.

Reference graph

Works this paper leans on

8 extracted references · 2 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Data Upload:Users ingest custom query and source documents via CSV files

  2. [2]

    Configuration:The pipeline is customized by selecting pre-trained models and tuning retrieval parameters (e.g., retrieval depth k and classification confidence thresholds)

  3. [3]

    Quo usque tandem abutere, Catilina, patientia nostra?

    Result Exploration:The interactive dash- board presents query segments alongside retrieved source candidates, displaying co- sine similarity and classification probability scores, with functionality to export confirmed matches. 8https://github.com/julianschelb/ locisimiles # Predictions Retrieval Only Retrieve+Rerank FPR×10 4 SMR×104 kRet. Rer. TP FP FN F...

    arXiv 2036

  4. [208]

    Charith Rajitha, Lakmali Piyarathna, Dilan Sachintha, and Surangika Ranathunga

    Association for Computational Linguistics. Charith Rajitha, Lakmali Piyarathna, Dilan Sachintha, and Surangika Ranathunga. 2021. Metric learning in multilingual sentence similarity measurement for document alignment. InProceedings of the Inter- national Conference on Recent Advances in Natural Language Processing (RANLP 2021), Held Online, 1-3 September, ...

    2021

  5. [2019]

    On the feasibility of automated detection of allusive text reuse. InProceedings of the 3rd Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, LaTeCH@NAACL-HLT 2019, Min- neapolis, MN, USA, June 7, 2019, pages 104–114. Association for Computational Linguistics. Marc Marone, Orion Weller, Wi...

    2019

  6. [2021]

    InProceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, *SEM 2021, Online, August 5-6, 2021, pages 52–66

    Recovering lexically and semantically reused texts. InProceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, *SEM 2021, Online, August 5-6, 2021, pages 52–66. Association for Computational Linguistics. Adnen Mahmoud and Mounir Zrigui. 2022. Siamese arabert-lstm model based approach for arabic para- phrase detection. I...

    2021

  7. [2023]

    over-generating

    Cross-lingual text reuse detection at document level for english-urdu language pair.ACM Trans. Asian Low Resour. Lang. Inf. Process., 22(6):173:1– 173:22. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text em- beddin...

    Pith/arXiv arXiv 2023

  8. [2025]

    Hadar Miller, Tsvi Kuflik, and Moshe Lavee

    mmbert: A modern multilingual encoder with annealed language learning.CoRR, abs/2509.06888. Hadar Miller, Tsvi Kuflik, and Moshe Lavee. 2025. Text alignment in the service of text reuse detection.Ap- plied Sciences, 15(6). Francesco Molfese, Andrei Stefan Bejgu, Simone Tedeschi, Simone Conia, and Roberto Navigli. 2024. Crocoalign: A cross-lingual, context...

    Pith/arXiv arXiv 2025