{"id":"dc6ecc6f-1131-400f-aefb-ea132c7dea84","arxiv_id":"2601.07533","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark dataset and evaluation framework for detecting intertextual references in Latin literature, with baseline results showing moderate performance of dense retrieval and classification models.","lead":"This paper introduces Loci Similes, a benchmark dataset of ~172k Latin text segments with 545 expert-verified intertextual links between Late Antique and classical authors. It also evaluates retrieval and classification baselines using language models, showing that dense retrieval finds about 61% of true references at rank 10.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth is built entirely from n-gram-matched candidates, so every positive pair shares lexical overlap; without a lexical baseline the benchmark's semantic-similarity claim is untested.","rationale":"The reader's weakest assumption correctly identifies the ground-truth selection bias. I concur and sharpen it: because both candidate sources are n-gram-based, every positive pair is guaranteed to contain a shared two-word sequence, so the benchmark cannot test the paper's central claim of evaluating semantic similarity beyond lexical matching. The absence of a lexical baseline compounds the problem, as there is no control condition to show that LLMs add value beyond simple lexical overlap. Additionally, I noticed internal inconsistencies in Table 3: given the reported TP/FP/FN values, the F1 scores are arithmetically impossible (e.g., at k=5, retrieval-only TP=81, FP=4595, FN=27 yields F1≈0.03, not 0.26; retrieve+rerank yields F1≈0.49, not 0.66). This indicates a reporting/computation error that must be corrected for the baseline results to be credible. Nevertheless, the dataset may still be valuable if its scope is described honestly and a lexical baseline is added. The current CONDITIONAL verdict remains appropriate; the paper should be revised to address the lexical-overlap bias and the table discrepancies before acceptance.","tokens_in":19578,"tokens_out":10491,"duration_ms":107356,"concrete_test":"Run a lexical baseline (e.g., Tesserae's n-gram overlap scorer or BM25 over lemmatized segments) on the same 5-fold splits used in Section 6.1, and compute recall@10 and F1 against the reported E5-large retrieval-only/pipeline results. Additionally, count the fraction of the 545 positive pairs sharing at least one content-word bigram after lemmatization. If the lexical baseline's recall@10 approaches 0.61 and nearly all pairs share a bigram, the benchmark is lexically saturated and fails to test semantic similarity beyond lexical matching.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The 545 ground-truth parallels are not a representative sample of Latin intertextuality. The 270 instances from Schropp et al. trace back to two-word n-gram matching, and the 275 additional instances are generated by the same rule-based n-gram pipeline described in Section 3.2. This guarantees that every positive pair shares at least one two-word lexical overlap. The paper's stated goal is to evaluate 'semantic similarity beyond exact lexical matching' (Abstract and Section 9), yet a benchmark whose positives all satisfy a lexical-overlap criterion cannot distinguish semantic models from lexical retrievers. The absence of any lexical baseline (e.g., Tesserae n-gram overlap or BM25) in Sections 6–7 leaves the reported LLM results without a control. If a simple n-gram retriever already achieves recall comparable to E5-large, the benchmark does not measure what the paper claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Loci Similes, a benchmark for Latin intertextuality detection comprising ~172k text segments and 545 expert-verified parallels linking Late Antique authors (Jerome, Lactantius) to classical Latin authors. The ground truth is constructed from two sources: 270 instances from prior scholarship (Schropp et al. 2024b) and 275 instances identified via a rule-based n-gram matching pipeline and then manually verified by domain experts. The authors propose an evaluation framework with error-based metrics (SMR, FPR, FNR) and baseline experiments for dense retrieval, binary classification, and a retrieve-and-rerank pipeline using multilingual and Latin-specific language models. They report that E5-large performs best for retrieval (Recall@10 = 0.61) and XLM-RoBERTa Large for classification (F1 = 0.50), and that retrieve-and-rerank substantially reduces false positives compared to retrieval-only, yielding a 99% reduction in manual review workload while retaining nearly 80% of true links.","tokens_in":19803,"tokens_out":8908,"duration_ms":87845,"significance":"If the benchmark is taken at face value, it fills a genuine gap by providing the first standardized evaluation resource for computational intertextuality detection in Latin, a task of direct relevance to classical philology and digital humanities. The authors ship the dataset, the open-source locisimiles package, and a GUI, which are concrete contributions that will enable future comparisons. The expert annotation process and the explicit acknowledgement of non-exhaustive ground truth and labeling ambiguity are strengths. The evaluation framework, with its focus on whole-document comparison and error-based metrics, is well motivated for the philological workflow. However, the benchmark's validity hinges on the representativeness of the ground truth and on whether the experiments demonstrate capability beyond lexical matching; these points require substantial revision.","major_comments":[{"comment":"The ground-truth construction procedure guarantees that every positive pair shares lexical overlap. The 275 additional instances are explicitly identified via a two-word n-gram matching pipeline, and the 270 instances from Schropp et al. (2024b) derive from the same group's earlier n-gram-based approaches (Section 2.1). Thus the benchmark cannot contain allusions with zero lexical overlap, yet the abstract and Section 9 claim it evaluates 'semantic similarity beyond exact lexical matching.' This sampling bias undermines the central claim. Please either supplement the ground truth with non-lexical-overlap parallels (e.g., from scholarly indices referencing thematic or semantic echoes), or explicitly reposition the benchmark as covering lexically grounded intertextuality.","section":"Section 3.2 / Abstract / Conclusion"},{"comment":"No lexical baseline is provided. Since all positives satisfy a lexical-overlap condition, a sparse retriever (e.g., Tesserae two-word overlap, lemma-overlap, or BM25) is a necessary control. Without it, the reported recall of E5-large (0.61 at Recall@10) is uninterpretable as evidence of semantic capability. Please add at least one non-neural lexical baseline and compare it against the dense models under the same evaluation protocol.","section":"Section 7 / Tables 3-4"},{"comment":"The metric definitions are internally inconsistent. The formulas SMR=(FP+FN)/N, FPR=FP/N, FNR=FN/N define global rates over all pairs, but the text states 'We calculate these metrics individually for each query segment and report the mean value averaged over all queries.' These two cannot both be true. If the metrics are global, remove the per-query sentence; if per-query, provide the correct per-query formulas. This affects the validity of Tables 3, 5, and 6.","section":"Appendix B.1"}],"minor_comments":[{"comment":"The construction of the 270 instances from Schropp et al. (2024b) is not described in detail; specify how that dataset was created and whether it also relies on n-gram candidate generation.","section":"Section 3.2"},{"comment":"The selection of the 937 query and 880 source segments used for evaluation is not described. Please specify the sampling procedure to ensure reproducibility and demonstrate that the subset is representative of the full corpus.","section":"Section 6.1"},{"comment":"The confusion matrices appear inconsistent with the reported FPR/SMR values. For example, in Table 5, the random-pairs row shows FP=831 and N≈1,061,646, which would give FPR≈0.00078, not the reported 0.0491. Please verify the numbers.","section":"Tables 5 and 6"},{"comment":"The phrase 'comprising of' is nonstandard; use 'comprising' or 'consisting of'.","section":"Abstract"},{"comment":"The '# Classifications' line is included in the legend but the caption does not explain which axis it uses. If it is on a secondary y-axis, label the axes accordingly.","section":"Figure 5"},{"comment":"The 'Online Contrastive Loss' is mentioned without a reference. If this is a standard loss from a specific source, cite it; otherwise, describe the loss function.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The ground-truth construction is heavily dependent on the authors' own prior work (Schropp et al. 2024a/b), which raises a question of independence. The manual verification step mitigates this, but the benchmark would be more convincing with an external source of positives (e.g., scholarly indices from other research groups). The missing lexical baseline is the most pressing technical issue; it is easily fixable and will determine whether the benchmark actually supports the paper's semantic-similarity claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: Loci Similes is a genuinely useful resource for Latin intertextuality detection, and it deserves a serious referee. The dataset is well documented, the code and data are public, and the whole-document evaluation framing is a sensible break from sentence-pair IR. My main reservation is not the one in the stress-test note. That note claims all 545 positives trace back to two-word n-gram matching, but the paper says only the 275 added links come from the n-gram pipeline; the 270 from Schropp et al. (2024b) are a manually curated gold standard. So the ground truth is a mix, and the lexical-overlap bias is partial, not total.\n\nThe real gap is the missing lexical baseline. The paper evaluates E5, BGE-M3, SPhilBerta, and others, but no n-gram overlap or BM25. Since over half the positives were originally surfaced by n-gram matching, a simple lexical retriever might already get decent recall. Without that control, the abstract's claim that LLMs capture 'semantic similarity beyond exact lexical matching' is not actually tested. This is easy to fix: add a Tesserae-style or BM25 baseline to the retrieval tables. If the lexical baseline is clearly worse, the neural models are validated; if it is competitive, the benchmark still stands as a resource, but the framing needs to change.\n\nThe soft spots otherwise are minor. The error-based metrics are appropriate for the task, and the authors are honest about non-exhaustive ground truth and labeling ambiguity. The citation pattern is reasonable; they build on their own prior work and say so explicitly.\n\nWho it's for: computational philology and anyone building tools for historical text reuse. It's a solid benchmark for comparing future methods, with the caveat that the positive set is anchored in lexically detectable reuse. I'd send it out, and the main revision requests would be the lexical baseline and a clearer statement of ground-truth composition.","headline":"Useful benchmark, real resource, but the missing lexical baseline leaves the semantic-similarity claim untested — and the stress-test overstates the n-gram bias.","tokens_in":20263,"tokens_out":5687,"would_cite":true,"duration_ms":57589,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark gives Latin intertextuality detection a standardized testbed: 545 expert-verified parallels in ~172k text segments, plus baseline retrieval and classification results.","keywords":["Latin literature","intertextuality detection","benchmark dataset","text reuse","information retrieval","language models","classical philology","dense retrieval"],"falsifier":"Assemble a held-out test set of well-documented Jerome and Lactantius allusions that share no n-grams with their sources (e.g., thematic echoes identified by philologists). If the current best models perform near chance on this set while scoring well on the existing ground truth, the paper's claim that the benchmark captures semantic similarity beyond lexical matching would be refuted.","tokens_in":19507,"feed_emoji":"📜","tokens_out":5047,"duration_ms":50452,"temperature":0.7,"pith_summary":"The paper addresses a concrete gap: computational methods for detecting intertextuality in Latin literature exist, but no standardized benchmark lets researchers compare them. Loci Similes is that benchmark—a curated corpus of ~172k text segments from Late Antique authors (Jerome, Lactantius) and canonical classical authors (Virgil, Cicero, Ovid, and others), together with 545 expert-verified intertextual links. The authors also define an evaluation framework that treats detection as a whole-document retrieval and alignment problem, scored by error-based metrics that account for the huge number of true negatives. Their baselines show a two-stage retrieve-and-rerank pipeline can capture roughly 79% of references at a retrieval depth of 100 while cutting a scholar's manual review workload by about 99%. If the benchmark holds, it gives the field a common ruler and a concrete target for models that go beyond verbatim matching.","feed_headline":"New benchmark tests AI on Latin literary allusions","feed_subtitle":"545 expert-verified parallels let researchers compare how well systems spot quotes, paraphrases, and subtle echoes.","key_machinery":"The load-bearing object is the benchmark itself: a query corpus of ~83k segments from Jerome and Lactantius, a source corpus of ~88k segments from ten classical authors, and a ground-truth set of 545 directional intertextual links. Around it, the paper builds an evaluation framework that frames detection as segment-wise retrieval between whole documents and defines error-based metrics (Segment-Misclassification Rate, Global False-Positive Rate, Global False-Negative Rate) normalized by total pairs, appropriate for a setting where true negatives vastly outnumber positives. The baseline system is a retrieve-and-rerank pipeline: a bi-encoder generates candidate source passages by cosine similar","core_discovery":"The central claim is that Loci Similes provides the first standardized resource for evaluating intertextuality detection in Latin. The authors show that a modern pipeline—dense retrieval with a multilingual embedding model followed by cross-encoder reranking—substantially outperforms a retrieval-only baseline in precision and error rates, recovering most true links while discarding the vast majority of candidates. They argue that this demonstrates both the value of semantic (rather than purely lexical) matching and the remaining open problem: distinguishing genuine allusions from coincidental overlap of common phrases.","pith_inferences":["The ground truth's half-and-half construction—prior scholarship plus a rule-based n-gram pipeline—may make the benchmark lexically biased; a future version should add a separate set of allusions with minimal lexical overlap to test true semantic generalization.","The same dataset could support related tasks beyond retrieval, such as automatically classifying links into verbatim/paraphrase/allusion categories, potentially refining theories of intertextuality.","The pipeline itself could be applied to other low-resource historical language pairs if new ground truth is provided, making this a reusable evaluation infrastructure rather than a Latin-only artifact.","A testable extension: incorporating the annotators' 'conduit function' criterion—whether the reused phrase imports meaning from the source—as an explicit training signal might improve precision beyond the current best F1 of 0.5."],"forward_implications":["Researchers can now benchmark new intertextuality-detection methods against a fixed dataset instead of ad-hoc collections, making progress measurable.","The framework shifts practical workflows: at k=100, a scholar reviews ~780 candidate passages to find 85 genuine links, versus ~93,700 for retrieval-only—a ~99% workload reduction.","The dataset's taxonomy (verbatim quote, paraphrase, allusion) provides a way to analyze where automated methods succeed and fail along the spectrum of reuse.","The error-based metrics offer a template for other retrieval tasks with extreme class imbalance, e.g., historical text reuse in other languages.","The results indicate that large multilingual models outperform Latin-specific ones, at least with current fine-tuning, guiding future model selection."],"fun_headline_variants":["New benchmark for Latin intertextuality detection","First standardized test for spotting Latin literary allusions","Benchmark with 545 verified parallels for AI evaluation","Reranking boosts AI precision on Latin literary parallels","AI retriever-reranker beats baseline on Latin allusions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 545 ground-truth links are treated as a representative sample of intertextuality in the corpus, but half were derived from a rule-based n-gram pipeline that favors lexical overlap; if subtle, lexically-distinct allusions are under-represented, the benchmark measures mostly verbatim quote and paraphrase detection rather than the full phenomenon.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark for Latin intertextuality detection","First standardized test for spotting Latin literary allusions","Benchmark with 545 verified parallels for AI evaluation","Reranking boosts AI precision on Latin literary parallels","AI retriever-reranker beats baseline on Latin allusions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000445,"raw_usage":{"total_tokens":2039,"prompt_tokens":647,"completion_tokens":1392,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":391,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":391,"tokens_out":1392,"duration_ms":10221,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:01:49.185920+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Assemble a held-out test set of well-documented Jerome and Lactantius allusions that share no n-grams with their sources (e.g., thematic echoes identified by philologists). If the current best models perform near chance on this set while scoring well on the existing ground truth, the paper's claim that the benchmark captures semantic similarity beyond lexical matching would be refuted.","supporting_citations":[],"review_version":1}