Pith. sign in

REVIEW 3 major objections 4 minor 17 references

This paper argues that fine-grained intertextuality—how a text reuses another, not just where—can be extracted at scale by an LLM agent whose every proposed pair is grounded in exact character spans and labeled under a five-dimension typolo

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 05:01 UTC pith:Z6DK54F4

load-bearing objection A careful, genuinely useful systems paper whose headline historical claims overreach the evidence: recall is never measured, and that weakens the diachronic conclusions. the 3 major comments →

arxiv 2607.27595 v1 pith:Z6DK54F4 submitted 2026-07-30 cs.CL cs.AIcs.DL

Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories

classification cs.CL cs.AIcs.DL
keywords intertextualitytext reuseLLM agentspan groundingclassical ChineseAnalectsTwenty-Four Historiescultural attraction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper recasts intertextuality extraction from a similarity-scoring problem into a grounded relation-extraction task: an LLM reads one Analects book against one history scroll in full and, through a constrained tool interface, must localize every reused fragment to exact character spans on both sides and label it on five dimensions—form, aspect, source-marking, function, stance. The authors validate the protocol on an exhaustive comparison of the Analects with the Book of Han, where three experts adjudicate 3,489 pooled candidates into a 2,533-pair gold standard, then scale the validated extractor to all Twenty-Four Histories, producing 5,766 pairs. The central finding is diachronic: the interpretive composition of Analects citation shows no systematic change across eighteen centuries, yet the same passage is quoted less and less literally. The paper argues this stability-in-the-aggregate with drift-in-the-individual is exactly what a cultural-attraction account predicts, and that this structure is inexpressible in a similarity score.

Core claim

The paper's central claim is that a promptable LLM, constrained by a tool interface that verifies every annotation at write time, can transform the detection of text reuse into describable reception history. Each intertextual pair is committed only after exact substring lookup grounds both fragments and a schema check enforces the five-dimension typology; the final submission is checked against the committed set, so the output is auditable rather than parsed from model prose. Validated on an expert-adjudicated gold standard of 2,533 pairs, the protocol scales to 5,766 pairs across all Twenty-Four Histories. With this record, the paper establishes that the way the Analects is cited—its form,

What carries the argument

The load-bearing mechanism is the span-grounded agentic extraction protocol: an LLM agent reads both text units in full, then interacts with tools (exact substring positioning, pair commitment with verbatim re-slice and schema validation, list/remove revision, and a guarded final submit) so that a prediction is a correct annotation only if both fragment texts occur verbatim at their stated offsets. The five-dimension typology (form, aspect, source-marking, function, stance) converts each grounded pair into a describable reuse event. The validate-then-scale design—adjudicated gold on one source pair, then deployment of the chosen model to the full corpus—is what turns a small expert effort in

Load-bearing premise

The load-bearing premise is that the extractor's precision—and more importantly its sensitivity to paraphrase—measured on the Analects–Book of Han pair transfers unchanged to all twenty-four histories, so that the observed diachronic patterns are properties of the texts rather than artifacts of which pairs the model happens to surface.

What would settle it

A second expert-adjudicated benchmark on a late history (for example, the Ming History) with an independent enumeration of intertextual pairs—including a count of how many pairs share no character trigram—would settle the transfer assumption. If the model's recall on such a benchmark is measurably lower than on the Book of Han, the declining-fidelity and stable-composition patterns would be artifacts of differential detection sensitivity across time, not properties of the tradition.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Similarity scores cannot substitute for typology: distributions of form and source-marking are annotatable with confidence, while function and stance are contested; any corpus-scale claim about reuse should be built on the reliable dimensions.
  • Corpus-scale reception history becomes auditable: every one of the 65,380 tasks ended in a well-defined outcome, and every pair is a write-time-checked commitment, so a reader can verify a claim by inspecting any pair in context.
  • The null result for interpretive composition, combined with the fidelity decline, provides quantitative evidence for a cultural-attraction account of transmission: aggregate stability coexists with individual drift.
  • Marking and fidelity are distinct axes: named citations are more literal than unmarked ones even among direct quotations, so a similarity score cannot stand in for the marking label.
  • Fixed segmentation misses asymmetric reuse: many-to-one relations, where one history span answers to two or more disjoint Analects passages, are expressible only with free span localization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the protocol transfers beyond the Analects–histories pair, the same validate-then-scale design could be applied to other canonical corpora (the Odes, the Zhuangzi, or Latin and Greek texts), enabling cross-tradition comparisons of how citation practice evolves—or fails to.
  • The reliability gradient implies that future annotation designs should treat function and stance as multi-label or as targets for contested-judgment adjudication, rather than as forced single-choice dimensions, if these intent-laden categories are to carry load-bearing claims.
  • The fidelity decline held per passage suggests a testable 'audience effect': when an authority is named, wording fidelity rises; a cross-linguistic replication could reveal whether this is a property of canonical transmission generally or specific to the Chinese historiographic tradition.
  • A practical consequence left implicit: because precision and cost vary independently across models, deployment choices for scholarly extraction should be made on precision-per-dollar and calibration curves, not on benchmark rank alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper recasts fine-grained intertextuality extraction as an agentic LLM task: a model reads two full text units, grounds each proposed reuse in exact character spans through a constrained tool interface, and labels it along five interpretive dimensions. The method is validated on the Analects–Book of Han pair, where two experts plus an arbiter adjudicate a pooled multi-model candidate set into a 2,533-pair gold standard. Twelve LLMs are compared on precision, cost, latency, and calibration. The validated extractor is then scaled to all Twenty-Four Histories (65,380 chunk-pair tasks, 5,766 extracted pairs), and the paper claims that the interpretive composition of Analectscitation is stable across eighteen centuries while literal fidelity to a given passage declines. The protocol, benchmark, and code/data are released.

Significance. The methodological contribution is strong and valuable. The tool-enforced grounding with exact character spans, write-time schema/duplicate checks, hermetic execution, and explicit abstention produce auditable, verifiable annotations rather than parsed model prose. The expert-adjudicated gold standard with a reported agreement gradient provides a careful evaluation resource, and the twelve-model precision/cost/calibration comparison is informative. The oracle retrieval analysis convincingly shows the limitations of similarity-based prefiltering for paraphrase. If the corpus-scale diachronic findings are robust, the paper would be a significant advance for digital humanities and computational text reuse. However, those findings currently rest on an unmeasured recall assumption, and the 'no systematic change' claim is a null result without equivalence testing; the central historical contribution therefore needs additional support or more careful qualification.

major comments (3)
  1. [Scaling to the Twenty-Four Histories / Table 2 / Discussion and Limitations] The corpus-scale historical claims rest on an unmeasured and, on the validation set, low recall. The deployment model (deepseek-v4-flash) contributes 390 of the 2,533 adjudicated gold pairs (15.4%; Table 2 vs. the gold-size figure), and because the gold pool is itself the union of the twelve models' proposals, the benchmark cannot bound what the models miss. Precision alone cannot validate a diachronic distribution: if detection sensitivity varies with composition date, genre, or fidelity, the stable-composition result in Table 3 and the declining-fidelity result in Figure 4 could be artifacts of which pairs the extractor surfaces. The confidence≥0.9 subset is a precision-correlated filter, not a recall-bias diagnostic. The authors acknowledge precision transfer but not recall transfer. I ask for a recall estimate on at least one additional adjudicated source pair or a sampling-based det
  2. [Scaling to the Twenty-Four Histories / Table 3] The 'no systematic change' conclusion is an unsupported null. Table 3 reports Spearman p-values of .12–.28 and a permutation test for Jensen–Shannon divergence, but failure to reject the null is not evidence of stability; no confidence intervals, equivalence bounds, or power analysis are given. With 24 histories and one composition date per multi-century work, the test is coarse. Please report effect sizes with confidence intervals or an explicit equivalence margin, and in the Abstract use 'no significant trend was detected' rather than 'shows no systematic change'.
  3. [Abstract / Conclusion] The headline claim, 'Across eighteen centuries the interpretive composition of citation shows no systematic change while the same passage is quoted ever less literally,' is stronger than the evidence supports. The composition claim inherits the recall/representativeness issue (Major 1) and the null-result issue (Major 2); the fidelity decline is estimated on model-surfaced pairs from a single extractor without a recall correction. Additionally, Table 3 includes function and stance, which the paper itself labels exploratory (Discussion and Limitations) because of low inter-annotator agreement; the Abstract should specify that the stable-composition claim rests on form and source-marking, or should mark the other dimensions explicitly as exploratory.
minor comments (4)
  1. [Abstract] The phrase 'exhaustive comparison of the Analects with the Book of Han' should be clarified: the chunk-pair comparison is exhaustive, but the gold standard is a candidate pool, not an exhaustive reference. Suggest 'exhaustive chunk-pair comparison' to avoid overstatement.
  2. [Scaling to the Twenty-Four Histories] The fidelity measure is described only as 'character-level longest common subsequence' in one sentence. Please specify the normalization (e.g., divided by source or target length) and how span-boundary differences are handled, since Figure 4's quantitative claims depend on it.
  3. [Expert-Adjudicated Evaluation / Table 1] Table 1 is informative, but the 'Chance' column would be clearer if it stated that chance is computed from the annotators' marginals, as the text already says. Consider adding a footnote to the table so the column is self-contained.
  4. [Discussion and Limitations] The statement 'corpus totals are lower bounds' is too weak: the issue is not only totals but distributional representativeness. Please state explicitly that lower-bound status applies to counts, not to proportions or trends.

Circularity Check

0 steps flagged

No significant circularity: the benchmark is expert-adjudicated, the scaling output is explicitly labeled as model judgments, and the main threats are recall/transfer validity rather than constructional circularity.

full rationale

The paper's derivation chain is: define the extraction task; build the gold standard by expert adjudication of pooled model candidates; measure twelve LLMs on precision and calibration; select one extractor; scale to the Twenty-Four Histories; and then compute diachronic distributional statistics. No step in this chain reduces to its inputs by construction. The benchmark is not model-defined validity: the 2,533 gold pairs are expert judgments on the 3,489 pooled candidates, and precision is the human-confirmed share of each model's own committed pairs. The pool is the union of the tested models' proposals, which makes the gold incomplete and recall undefined or low, but the paper acknowledges this ('the pool is the union of twelve models' candidates ... so corpus totals are lower bounds') and it is a standard pooled-evaluation limitation, not a circular derivation. At scale, the paper explicitly frames the output as 'a distribution of calibrated model judgments' and states the precision-transfer limitation ('corpus-scale extraction uses a single model calibrated on the Book of Han'). The historical findings are not fitted to the model: no parameter is tuned on the diachronic trends, expert adjudication is external human judgment, and the LLMs are applied without fine-tuning. The self-citations (Duan et al. 2023; Wang et al. 2024a) appear only as related-work background and are not load-bearing. The remaining concerns—unmeasured recall, possible detection bias over centuries, and overgeneralization from the Book of Han calibration—are threats to external validity and should be assessed as correctness risk, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims require no fitted numeric parameters; the method rests on design choices (typology, chunking, thresholds) and domain assumptions about expert judgment and edition fidelity. The main load-bearing unverified premise is the transfer of precision from the Analects–Book of Han pair to all Twenty-Four Histories.

free parameters (3)
  • confidence threshold 0.9
    Hand-chosen threshold for a robustness subset; not optimized against the target finding.
  • minimum distinct composition years per cluster (5)
    Inclusion threshold for the fidelity-vs-year analysis; selected to retain enough clusters for stable correlation, not tuned to produce significance.
  • chunk granularity (Analects book, history scroll)
    Design choice defining task units; affects extraction scope and cost, not a fitted number.
axioms (5)
  • domain assumption The five-dimension typology is a valid, jointly exhaustive schema for intertextual reuse.
    Central protocol design; the reliability gradient (Table 1) shows only some dimensions are reliably annotatable, so the schema is partly validated empirically.
  • domain assumption Expert judgments on context-window evidence are acceptable ground truth for intertextual validity.
    The gold standard rests on three experts' forced-choice validity and label judgments; no external source of truth is used.
  • domain assumption The Shidian Guji digital edition faithfully represents the Analects and Book of Han.
    All spans and offsets are anchored to this edition; errors would propagate to the benchmark and corpus run.
  • domain assumption Single composition dates for each of the Twenty-Four Histories adequately represent their creation time.
    Diachronic correlations use one year per history; the authors acknowledge multi-decade composition and follow received periodization.
  • standard math Standard statistics (Spearman, permutation tests, bootstrap) apply to cluster-level summaries.
    Used to compute p-values and CIs for the fidelity-decline and stability null results.

pith-pipeline@v1.3.0-daily-deepseek · 13645 in / 14711 out tokens · 152023 ms · 2026-08-01T05:01:36.546387+00:00 · methodology

0 comments
read the original abstract

Computational approaches to intertextuality have advanced from string matching to neural retrieval, yet their outputs, similarity scores and parallel-passage lists, identify where texts reuse one another without characterizing how or why. We recast fine-grained intertextuality extraction as an agentic task in which a large language model (LLM) reads two text units in full and, through a constrained tool interface, must ground each proposed reuse in exact character spans on both sides and label it under a five-dimension typology of reuse (form, aspect, source-marking, function, stance). We validate the approach on an exhaustive comparison of the Analects with the Book of Han, where three domain experts adjudicate a pooled multi-model candidate set into a benchmark of 2,533 intertextual pairs. Against this standard we study twelve LLMs, reporting precision (56%-93%), a 51$\times$ cost spread at comparable quality, and how well their confidence is calibrated. Expert agreement traces a reliability gradient: dimensions legible on the textual surface are annotated consistently, while those requiring inference of intent are contested, delimiting the claims such annotation supports. Scaling the validated extractor to the full Twenty-Four Histories (65,380 comparisons, 5,766 pairs) recovers corpus-level structure a similarity score cannot express. The interpretive composition of citation shows no systematic change across eighteen centuries, yet the same passage is quoted less and less literally. Stability in the aggregate with drift in the individual case is what a cultural-attraction account expects. We release the extraction protocol and the expert-adjudicated benchmark.

Figures

Figures reproduced from arXiv: 2607.27595 by Jun Wang, Wanyu Si, Zhaoji Wang.

Figure 1
Figure 1. Figure 1: Framework overview. (a) The extraction agent for one chunk pair and its task tools (names abbreviated): [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Twelve LLMs on the 2,533-pair gold standard: [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Calibration of verbalized confidence, pooled over [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Literal fidelity declines with the citing history’s [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 5 linked inside Pith

  1. [6]

    arXiv:2510.27045

    Quantitative Intertextuality from the Digital Humanities Perspective: A Survey. arXiv:2510.27045. Duan,S.;Wang,J.;Yang,H.;andSu,Q.2023.Disentangling the cultural evolution of ancient China: a digital humanities perspective.Humanities and Social Sciences Communica- tions, 10:

  2. [8]

    arXiv:2410.15145

    Mining Asymmetric Intertextuality. arXiv:2410.15145. Lee, J

  3. [12]

    arXiv:2601.07533

    Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature. arXiv:2601.07533. Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.;Hambro,E.;Zettlemoyer,L.;Cancedda,N.;andScialom, T

  4. [502]

    Wenxin diaolong [The Literary Mind and the Carving of Dragons]. Ca. 501–502; chapter “Shilei” (on allusion) cited; verified against the Guangming Daily Press edition (Beijing, 2014), p

  5. [1832]

    Qingdynasty;chapter“Shuolin”cited; verified against the Guji Chubanshe edition (Beijing, 1956, punctuated by Liu Gongchun), pp

    Wenshi tongyi [General Principles of Lit- eratureandHistory]. Qingdynasty;chapter“Shuolin”cited; verified against the Guji Chubanshe edition (Beijing, 1956, punctuated by Liu Gongchun), pp. 116–118. Zhang,Z.2008.CongShiji,Hanshu,HouHanshukanLunyu zaiLiangHandeLiuchuan[TheTransmissionoftheAnalects in the Two Han Dynasties as Seen from the First Three His- ...

  6. [1960]

    DeepSeek-AI.2026.DeepSeek-V4:TowardsHighlyEfficient Million-Token Context Intelligence

    A Coefficient of Agreement for Nominal Scales.EducationalandPsychologicalMeasurement,20(1): 37–46. DeepSeek-AI.2026.DeepSeek-V4:TowardsHighlyEfficient Million-Token Context Intelligence. arXiv:2606.19348. Deng, Z.; Yang, H.; and Wang, J

  7. [1985]

    Sturgeon,D.2018

    Anthropology and Psychology: Towards an Epidemiology of Representations.Man (New Series), 20(1): 73–89. Sturgeon,D.2018. Unsupervisedidentificationoftextreuse in early Chinese literature.Digital Scholarship in the Hu- manities, 33(3): 670–684. Tian, K.; Mitchell, E.; Zhou, A.; Sharma, A.; Rafailov, R.; Yao, H.; Finn, C.; and Manning, C

  8. [1990]

    The problems of two paradoxes.Journal of Clinical Epidemiology, 43(6): 543–549

    High agreement but low kappa: I. The problems of two paradoxes.Journal of Clinical Epidemiology, 43(6): 543–549. Forstall, C. W.; and Scheirer, W. J. 2019.Quantitative In- tertextuality: Analyzing the Markers of Information Reuse. Springer. GLM-5Team.2026. GLM-5:FromVibeCodingtoAgentic Engineering. arXiv:2602.15763. Gu,J.1923.YuQianXuantongxianshenglungus...

  9. [2016]

    InProceedings of the 2016 Confer- enceonEmpiricalMethodsinNaturalLanguageProcessing, 1849–1859

    Non-literal text reuse in historical texts: An approach to identify reuse transformations and its appli- cation to Bible reuse. InProceedings of the 2016 Confer- enceonEmpiricalMethodsinNaturalLanguageProcessing, 1849–1859. Association for Computational Linguistics. Nehrdich, S

  10. [2017]

    Cameron,E.2026

    What are cultural attractors?Biology & Philosophy, 32(3): 377–394. Cameron,E.2026. ‘TheOrderintheHorse’sHeart’:ACase StudyinLLM-AssistedStylometryfortheDiscoveryofBib- licalAllusioninModernLiteraryFiction.arXiv:2604.19447. Chen, J.; Xiao, S.; Zhang, P.; Luo, K.; Lian, D.; and Liu, Z

  11. [2018]

    ArXiv:1602.08715

    Identifica- tion of Parallel Passages Across a Large Hebrew/Aramaic Corpus.Journal of Data Mining and Digital Humanities. ArXiv:1602.08715. Smith,D.A.;Cordell,R.;andDillon,E.M.2013. Infectious Texts:ModelingTextReuseinNineteenth-CenturyNewspa- pers. In2013 IEEE International Conference on Big Data. IEEE. Sommerschield,T.;Assael,Y.;Pavlopoulos,J.;Stefanak,...

  12. [2019]

    A BLAST-based, Language-agnostic Text Reuse Algorithm with a MARKUS Implementation and Sequence Alignment Optimized for LargeChineseCorpora.JournalofCulturalAnalytics,4(2). Wang, J. 2014.Hanshu yin Lunyu kao [A Study of the Book of Han’s Quotations of the Analects]. Master’s thesis, Qufu Normal University. Wang, J.; Duan, S.; Fu, B.; Gao, L.; and Su, Q. 2...

  13. [2021]

    InProceedings of the 2021 Con- ferenceoftheNorthAmericanChapteroftheAssociationfor ComputationalLinguistics:HumanLanguageTechnologies, 4900–4907

    Profiling of Intertextuality in Latin Literature Using Word Embeddings. InProceedings of the 2021 Con- ferenceoftheNorthAmericanChapteroftheAssociationfor ComputationalLinguistics:HumanLanguageTechnologies, 4900–4907. Association for Computational Linguistics. Buskell, A

  14. [2023]

    InProceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, 5433–5442

    Just Ask for Cali- bration:StrategiesforElicitingCalibratedConfidenceScores from Language Models Fine-Tuned with Human Feedback. InProceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, 5433–5442. Associa- tion for Computational Linguistics. Umphrey,R.;Roberts,J.;andRoberts,L.2024. Investigating Expert-in-the-LoopLLMDisc...

  15. [2024]

    InFindings of the Association for Computational Linguistics: ACL 2024, 2318–2335

    M3-Embedding: Multi-Linguality, Multi- Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. InFindings of the Association for Computational Linguistics: ACL 2024, 2318–2335. As- sociation for Computational Linguistics. Coffee,N.;Koenig,J.-P.;Poornima,S.;Forstall,C.W.;Osse- waarde,R.;andJacobson,S.L.2013. TheTesseraeProject...

  16. [2025]

    Buckley,C.;andVoorhees,E.M.2004

    Contextualizing ancient texts with gen- erative neural networks.Nature, 645(8079): 141–147. Buckley,C.;andVoorhees,E.M.2004. RetrievalEvaluation with Incomplete Information. InProceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 25–32. ACM. Burns, P. J.; Brofos, J. A.; Li, K.; Chaudhuri, P...

  17. [2026]

    https://openai.com/ index/gpt-5-5-system-card

    GPT-5.5 System Card. https://openai.com/ index/gpt-5-5-system-card. Periti, F.; Cassotti, P.; Montanelli, S.; Tahmasebi, N.; and Schlechtweg,D.2024. TRoTR:AFrameworkforEvaluating theRe-contextualizationofTextReuse.InProceedingsofthe 2024ConferenceonEmpiricalMethodsinNaturalLanguage Processing, 13972–13990. Association for Computational Linguistics. Pi, X....