Pith. sign in

REVIEW 3 major objections 6 minor 15 references

Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH

T0 review · 3 major / 6 minor · reviewed 2026-07-08 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GraphRAG for SSH: grounding LLMs in scholarly practice, not just text

desk verdict Design document for a GraphRAG-based SSH search system — no results yet, but the compliance framework and infrastructure integration are the real contribution read the letter →

arxiv 2607.05956 v1 pith:JGNJH7MP submitted 2026-07-07 cs.AI cs.CL

Adam Faci , Alessio Miaschi , Anne Combe , Pascal Cuxac , Francesca Frontini , Nicolas Larrousse , Stéphane Pouyllau This is my paper · ORCID
classification cs.AIcs.CL
keywords researchcaseepistemicevaluationframeworkhumanitiesliteraturellms
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a design — not yet a validated system — for adapting large language models to the specific epistemic and methodological needs of Social Sciences and Humanities (SSH) research. The central object is a GraphRAG (knowledge-graph-enhanced retrieval-augmented generation) architecture built on top of ISIDORE, an existing French SSH search engine. The architecture has three coupled layers: a retrieval model fine-tuned on historical user click data from ISIDORE so that ranking reflects observed scholarly search behaviour rather than pure textual similarity; a knowledge-graph enrichment layer using Wikidata and OpenAIRE for entity disambiguation, structured navigation, and document-level comparison; and a generative model instruction-tuned on SSH corpora to produce literature syntheses and state-of-the-art overviews that remain traceable to explicitly retrieved sources. The authors argue that this combination can support core SSH workflows — literature review, thematic exploration, comparative analysis — while preserving multilingualism (French, Italian, English), source traceability, and compliance with European regulatory frameworks including the EU AI Act, GDPR, and copyright law. The paper frames the entire effort as a corrective to dominant AI-powered discovery tools (Semantic Scholar, Elicit, Scopus) that privilege English-language journal articles, citation-based metrics, and STEM epistemic norms, arguing that SSH scholarship requires tools sensitive to disciplinary diversity, multilingual sources, non-article outputs (monographs, editions, datasets, blogs), and interpretive plurality — the same Dante text read differently by a literary scholar, a philologist, a historian, and a digital humanist. The project is in an advanced preparatory phase: corpora are being consolidated, fine-tuning has not yet begun, and the evaluation framework — combining quantitative benchmarking with expert panels from Franco-Italian Digital Humanities communities — is under development.

What carries the argument

The load-bearing mechanism is the coupling of three components: (1) a retrieval model fine-tuned on ISIDORE historical query–click data, where clicked documents are promoted and ignored results demoted — this interaction-aware ranking is the most novel and infrastructure-specific element; (2) a knowledge-graph layer drawing on Wikidata and OpenAIRE for entity disambiguation, semantic expansion, and structured navigation across authors, themes, institutions, and citations; (3) a generative model instruction-tuned on SSH corpora (approximately 3 million documents from the ISTEX SSH subset, supplemented by Italian DH conference proceedings and journals) to produce coherent literature syntheses.

What would settle it

The interaction-aware retrieval component — the most novel part of the architecture — would be falsified if fine-tuning on ISIDORE click data produces retrieval rankings that expert panels judge no better than, or worse than, purely semantic retrieval baselines. More broadly, the entire proposal would be undermined if the GraphRAG pipeline, once implemented, produces literature syntheses that expert DH panels rate as epistemically unreliable or if hallucination rates exceed acceptable thresholds in the independent evaluation. Since no results exist yet, the falsifier is prospective: the first

Watch

Extended reading notes

Core claim

The paper does not report experimental results. Its contribution is an architectural proposal: that a GraphRAG pipeline combining (1) interaction-aware retrieval fine-tuned on real scholarly click data, (2) public knowledge-graph enrichment for entity disambiguation and structured navigation, and (3) instruction-tuned generation on curated multilingual SSH corpora can produce a domain-adaptive LLM system that grounds its outputs in traceable sources while respecting the epistemic plurality, multilingualism, and regulatory constraints characteristic of SSH research. The authors explicitly state this is ongoing work in a preparatory phase, with the first fine-tuning experiments scheduled to ''

Load-bearing premise

The design assumes that historical click data from ISIDORE — which documents users clicked on and which they ignored — is a reliable signal for scholarly relevance. But clicks may reflect visibility, ranking bias, or convenience rather than true scholarly value, and no evidence is provided that this signal improves retrieval quality. The authors themselves acknowledge this component is replaceable by ''purely semantic retrieval models'' if behavioural signals are unavailable,

Editorial extensions

If this is right

  • If the click-data fine-tuning produces retrieval rankings that genuinely align with scholarly relevance, the approach could be ported to any research infrastructure that logs user interaction data — making interaction-aware retrieval a transferable design pattern beyond ISIDORE.
  • The architecture''s modularity (the authors note the click-data component is replaceable by purely semantic retrieval, and TEI/XML is replaceable by plain text with structured metadata) means the minimal viable version of this system could be deployed by institutions with far less infrastructure than ISIDORE, broadening access.
  • If expert-panel evaluation reveals that SSH scholars systematically disagree with quantitative benchmarks on what counts as relevant, this would constitute evidence that generic IR/LLM evaluation protocols are epistemically mismatched to humanities research — a finding with implications beyond this project.
  • The compliance-by-design approach (treating GDPR, AI Act, and copyright constraints as architectural components rather than post-hoc constraints) could serve as a template for other domain-adaptive LLM projects operating under European regulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents ReSearch_SSH, an ongoing use case within the LLMs4EU project, proposing a GraphRAG architecture for domain-adaptive LLMs in the Social Sciences and Humanities (SSH). The system extends the French ISIDORE search platform by combining (i) continued pre-training on the ISTEX SSH corpus (~3M documents), (ii) retrieval fine-tuning on historical ISIDORE query–click data, (iii) knowledge-graph enrichment via Wikidata and OpenAIRE, and (iv) instruction-tuned generation for scholarly synthesis tasks. The paper also describes a compliance-by-design framework (GDPR, EU AI Act, copyright/TDM) and a hybrid evaluation protocol combining quantitative benchmarking with expert-panel assessment. The project is in a preparatory phase; no models have been trained, no experiments run, and no results are reported.

Significance. The manuscript addresses a genuine gap: most AI-powered scholarly discovery tools are English-centric and citation-driven, poorly aligned with SSH practices that emphasize multilingualism, book-based scholarship, and qualitative evaluation. The choice of data sources is well-motivated and concrete (ISTEX SSH subset, AIUCD proceedings, Umanistica Digitale), and the regulatory discussion is substantive rather than boilerplate. The interaction-aware retrieval component (§3.2) is the most architecturally novel element. However, the paper is a project description with no empirical validation, no trained models, no baselines, and no quantitative findings of any kind. The significance therefore rests entirely on the design's plausibility, which has not been tested.

major comments (3)
  1. §3.2, retrieval fine-tuning paragraph: The interaction-aware retrieval component — the most novel architectural element — is proposed without any discussion of known biases in click-based implicit feedback. Learning-to-rank from click data is well-documented to suffer from position bias, selection bias, and presentation bias (e.g., Craswell et al. 2008; Joachims et al. 2017). The paper itself critiques commercial platforms for letting ranking bias shape scholarly visibility (§1), yet proposes using ISIDORE click data without addressing how the same biases might affect its own retrieval model. This is load-bearing because this component is identified as the most infrastructure-specific and novel part of the architecture. The authors should at minimum discuss what debiasing strategies (click models, counterfactual estimation, propensity weighting) will be considered, or justify why they相信o
  2. §3.1, Table 2: The token distribution shows English at 16,000–21,000M tokens versus Italian at 10–15M — a ratio exceeding 1,000:1. The paper claims the corpus 'supports large-scale alignment to SSH discourse while preserving multilingual exposure' and that Italian DH corpora enable 'cross-lingual bridging between Italian, French and English SSH traditions.' At this imbalance, it is unclear whether the Italian data is sufficient for any meaningful cross-lingual transfer, especially for continued pre-training. The paper should discuss whether this volume is adequate for the stated multilingual objectives or whether the Italian component is effectively limited to task-tuning rather than domain alignment.
  3. The paper contains no experimental results, no trained models, no baseline comparisons, and no quantitative evaluation of any component. While §5 transparently states the project is in a 'preparatory phase,' the manuscript as submitted does not constitute a research contribution in the conventional sense — it is a design document. For a journal publication, at minimum a proof-of-concept experiment (e.g., domain alignment pilot, retrieval quality with vs. without click-based fine-tuning, or a small-scale GraphRAG pipeline demonstration) would be needed to substantiate the design's feasibility. The authors should clarify what minimal empirical evidence they can provide to validate the architecture before submission.
minor comments (6)
  1. §3.1: 'the use case may introduce also scientific publications and academic blog posts from the Hypotheses platform, together with research data and metadata from the Nakala repository, will be incorporated' — the sentence structure is broken; 'may introduce' and 'will be incorporated' conflict syntactically.
  2. §5: 'attention will devoted to metadata normalisation' — missing 'be' ('will be devoted').
  3. §3.2: The reference to 'GPT-5' as a candidate for synthetic data generation appears premature given the project's timeline; the authors should either confirm availability or use a more generic reference to frontier commercial models.
  4. Table 2: The English token range (16,000–21,000M) is unusually wide for an estimate; clarifying the basis for this range would improve credibility.
  5. §3.2: The base model is not specified ('will be drawn from the pool of open multilingual foundation models developed or endorsed within the European AI ecosystem'). While the authors commit to reporting this later, a shortlist of candidates under consideration would strengthen the design discussion.
  6. The paper would benefit from a related-work section positioning the GraphRAG approach (Edge et al. 2024) against other domain-adaptive RAG systems, particularly in non-English or non-STEM application contexts.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive reading of our manuscript. The report correctly identifies the paper as a design document for an ongoing project (ReSearch_SSH within LLMs4EU) and raises three substantive points: (1) the absence of debiasing discussion for click-based retrieval fine-tuning, (2) the adequacy of the Italian corpus at the documented token imbalance, and (3) the lack of any empirical validation. We address each below. On the first point, we agree the omission is significant and will revise. On the second, we will clarify the role of Italian data and acknowledge its limitations more explicitly. On the third, we accept that a proof-of-concept experiment is needed and will commit to including one in the revised submission, while being transparent about the timeline constraints of the project.

read point-by-point responses

  1. Referee: §3.2, retrieval fine-tuning paragraph: The interaction-aware retrieval component is proposed without any discussion of known biases in click-based implicit feedback (position bias, selection bias, presentation bias). The paper critiques commercial platforms for ranking bias yet proposes using ISIDORE click data without addressing how the same biases might affect its own retrieval model.

    Authors: The referee is correct that this is a genuine gap in the manuscript. The irony of critiquing commercial platforms for ranking bias in §1 while proposing click-based fine-tuning in §3.2 without discussing debiasing is a fair and important observation. We will add a dedicated paragraph in §3.2 discussing the known bias sources (position bias, selection bias, presentation bias) and the debiasing strategies we plan to evaluate. Specifically, we intend to investigate: (a) propensity-weighted inverse propensity scoring based on result rank position, following the framework of Joachims et al. (2017); (b) click model-based estimation of relevance (e.g., position-based and cascade models) to derive debiased training signals; and (c) counterfactual estimation techniques for offline policy evaluation. We will also note that ISIDORE's interface design differs from commercial platforms in ways that may partially mitigate some biases (e.g., less aggressive personalization, no advertising-driven ranking), but we agree this does not eliminate the need for explicit debiasing. The revised manuscript will cite the relevant literature (Craswell et al. 2008; Joachims et al. 2017) and frame the debiasing strategy as an integral part of the retrieval fine-tuning design rather than an afterthought. revision: yes

  2. Referee: §3.1, Table 2: The token distribution shows English at 16,000–21,000M tokens versus Italian at 10–15M — a ratio exceeding 1,000:1. The paper claims the corpus supports multilingual exposure and cross-lingual bridging, but it is unclear whether the Italian data is sufficient for meaningful cross-lingual transfer, especially for continued pre-training.

    Authors: The referee raises a valid concern about the adequacy of the Italian component at the documented scale. We agree that 10–15M tokens of Italian in the ISTEX SSH subset is insufficient for meaningful domain alignment via continued pre-training on its own. The manuscript's claim that the corpus 'supports large-scale alignment to SSH discourse while preserving multilingual exposure' is accurate for French (450–600M tokens) but overstated for Italian. We will revise the text to clarify that: (1) the Italian ISTEX data contributes marginally to continued pre-training and is not expected to produce robust Italian-language domain alignment on its own; (2) the targeted Italian DH corpora (AIUCD proceedings, Umanistica Digitale) are primarily intended for task-tuning and genre/terminology adaptation rather than large-scale domain alignment; and (3) meaningful cross-lingual bridging for Italian will depend on the multilingual transfer capabilities of the base model (e.g., EuroLLM) rather than on Italian-specific pre-training volume. We will also note that the project's current phase focuses on enabling Italian and English querying over predominantly French materials, which is a more modest and achievable goal than full multilingual domain alignment. The revised manuscript will reframe the Italian component's role accordingly and avoid overstating its contribution to cross-lingual transfer. revision: yes

  3. Referee: The paper contains no experimental results, no trained models, no baseline comparisons, and no quantitative evaluation of any component. For a journal publication, at minimum a proof-of-concept experiment would be needed to substantiate the design's feasibility.

    Authors: We accept this criticism. The manuscript as submitted is a design document, and while §5 transparently states the project is in a preparatory phase, we agree that a journal publication requires at least minimal empirical evidence. We will commit to including a proof-of-concept experiment in the revised submission. The most feasible pilot within our timeline is a small-scale GraphRAG pipeline demonstration over a subset of the ISIDORE corpus, comparing retrieval quality with and without knowledge-graph enrichment (Wikidata/OpenAIRE entity linking). This would directly validate the most architecturally novel element the referee identifies as load-bearing. We will also include a preliminary domain-alignment pilot: continued pre-training of a small open multilingual model on a French SSH subset, evaluated via perplexity reduction and domain-specific term recognition compared to the base model. We cannot commit to full retrieval fine-tuning with click-based debiasing experiments at this stage, as the click data preparation and debiasing pipeline are still under development. We will be transparent about what the pilot does and does not demonstrate, and will frame it as initial feasibility evidence rather than full validation of the architecture. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is a design proposal with no empirical claims, no equations, and no fitted-parameter predictions.

full rationale

The paper describes ReSearch_SSH, an ongoing project proposing a GraphRAG architecture for SSH scholarly discovery. It contains no equations, no fitted parameters, no empirical results, and no predictions that could reduce to inputs by construction. The derivation chain is entirely propositional: (1) domain alignment via continued pre-training on SSH corpora, (2) retrieval fine-tuning on ISIDORE click data, (3) instruction-tuning for generative tasks, (4) evaluation via external LLMs4EU protocol. No step claims to derive a result from its own inputs. Self-citations are minimal and contextual (Frontini appears in Arachchige et al. 2025 and Branco et al. 2023, both describing SSH infrastructure landscape, not load-bearing for any derivation). The paper explicitly states 'The project is currently ongoing' and 'The first stages of domain alignment and retrieval-oriented fine-tuning are scheduled to begin shortly.' There is no empirical loop, no fitted-parameter-as-prediction pattern, and no self-citation chain that could create circularity. The concern about click-data bias (§3.2) is a correctness/validity risk for future work, not a circularity in the paper as written, since no claim is made that the approach works—only that it is proposed. Score: 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted because no experiments are conducted. No new entities are invented; the system uses existing models, knowledge graphs, and infrastructures. The axioms are domain assumptions about the reliability of click data, the feasibility of domain adaptation without forgetting, and the validity of expert evaluation — all unstated and unvalidated.

assumptions (4)
  • domain assumption Click-through data from ISIDORE user queries is a reliable proxy for scholarly relevance in SSH document retrieval.
    §3.2: 'Training signals are derived from user behaviour: documents effectively clicked and consulted are promoted, while systematically ignored results are ranked lower.' This assumption underpins the interaction-aware retrieval component but is not validated.
  • domain assumption A medium-scale multilingual model can be domain-adapted to SSH discourse via continued pre-training without catastrophic forgetting of general reasoning.
    §3.2: 'A first phase performs domain alignment through continued pre-training on SSH corpora in French and Italian, strengthening linguistic and terminological competence while preserving the general reasoning capacities of the base multilingual model.' No evidence is provided that this preservation holds.
  • standard math GraphRAG architecture (Edge et al. 2024) is applicable to SSH scholarly corpora with metadata-derived relational structures.
    §3.2 references Edge et al. 2024 for the GraphRAG approach. The applicability to SSH metadata graphs is assumed but not demonstrated.
  • domain assumption Expert panel qualitative assessment can reliably evaluate epistemic adequacy of LLM outputs in DH contexts.
    §3.3: 'These panels will evaluate system outputs in terms of scholarly quality, epistemic reliability, methodological adequacy, and practical usefulness.' The reliability of expert panels for LLM evaluation is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH." pith.science (2026). https://pith.science/paper/JGNJH7MP

@misc{pith2026260705956,
  author       = {Pith},
  title        = {Pith review of: Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JGNJH7MP}},
  note         = {Machine review of arXiv:2607.05956}
}
read the original abstract

The integration of Large Language Models (LLMs) into scientific research workflows, particularly for bibliographic discovery and literature synthesis, raises significant methodological, epistemic and regulatory challenges for the Social Sciences and Humanities (SSH), especially with regard to disciplinary diversity, multilingual access to sources and the evaluation of results. This paper presents an on-going use case developed within the European project LLMs4EU and the ALT-EDIC infrastructure, aimed at adapting foundation models to SSH research practices and supporting tasks such as question answering, comparative document analysis and literature review. The evaluation framework follows the LLMs4EU protocol and encompasses both independent quantitative benchmarking (retrieval, summarisation, traceability and hallucination detection) and a qualitative assessment involving a panel of Digital Humanities experts. By embedding model adaptation within research infrastructures and a structured legal and ethical compliance framework, the use case explores how domain-sensitive and regulation-aware generative AI can support SSH scholarship while preserving reliability and epistemic responsibility.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

  1. [1]

    , month = dec, year =

    Fenlon, Katrina S. , month = dec, year =. Thematic research collections:

    work page

  2. [2]

    Journal of Open Humanities Data , author =

    Contextualizing. Journal of Open Humanities Data , author =. 2024 , file =. doi:10.5334/johd.192 , abstract =

    work page doi:10.5334/johd.192 2024

  3. [3]

    JLIS.it , author =

    Multilingualism within. JLIS.it , author =. 2021 , keywords =. doi:10.4403/jlis.it-12672 , abstract =

    work page doi:10.4403/jlis.it-12672 2021

  4. [4]

    Perspective

    Spichtinger, Daniel , month = jan, year =. Perspective. doi:10.5772/intechopen.1013402 , file =

    work page doi:10.5772/intechopen.1013402

  5. [5]

    ITM Web of Conferences , publisher =

    Dumouchel, Suzanne and Blotière, Emilie and Barbot, Laure and Breitfuss, Gert and Chen, Yin and Donato, Francesca Di and Forbes, Paula and Petitfils, Clara and Pohle, Stefanie , year =. ITM Web of Conferences , publisher =. doi:10.1051/itmconf/20203303005 , abstract =

    work page doi:10.1051/itmconf/20203303005

  6. [6]

    Fostering Data Communities -- perspective from a Data Archive Service Provider

    Morselli, Francesca and Touber, Jetze and Scharnhorst, Andrea , month = feb, year =. Fostering. doi:10.48550/arXiv.2502.02321 , abstract =

    work page Pith review arXiv doi:10.48550/arxiv.2502.02321

  7. [7]

    Edge, Darren and Trinh, Ha and Cheng, Newman and Bradley, Joshua and Chao, Alex and Mody, Apurva and Truitt, Steven and Metropolitansky, Dasha and Ness, Robert Osazuwa and Larson, Jonathan , month = feb, year =. From. doi:10.48550/arXiv.2404.16130 , abstract =

    work page Pith review arXiv doi:10.48550/arxiv.2404.16130

  8. [8]

    From Local to Global: A Graph RAG Approach to Query-Focused Summarization

    From local to global: A graph rag approach to query-focused summarization , author=. arXiv preprint arXiv:2404.16130 , doi=

    work page Pith review arXiv

Show all 15 references
  1. [9]

    Vers un corpus optimal pour la fouille de textes : stratégie de constitution de corpus spécialisés à partir d'

    de Salabert, Camille and Barreaux, Sabine , editor =. Vers un corpus optimal pour la fouille de textes : stratégie de constitution de corpus spécialisés à partir d'. Actes de la 6e conférence conjointe. 2020 , keywords =

    2020

  2. [10]

    Manghi, Paolo and Bardi, Alessia and Atzori, Claudio and Baglioni, Miriam and Manola, Natalia and Schirrwagen, Jochen and Principe, Pedro , year =. The

  3. [11]

    Language Resources and Evaluation , author =

    The. Language Resources and Evaluation , author =. 2023 , keywords =. doi:10.1007/s10579-023-09658-z , abstract =

    2023 doi

  4. [12]

    Proceedings of the CLARIN Annual Conference , author =

    The. Proceedings of the CLARIN Annual Conference , author =. 2023 , file =. doi:10.3384/ecp210006 , abstract =

    2023 doi

  5. [13]

    JLIS: Italian Journal of Library, Archives and Information Science= Rivista italiana di biblioteconomia, archivistica e scienza dell'informazione: 12, 2, 2021 , pages=

    Multilingualism within Scholarly Communication in SSH: A literature review , author=. JLIS: Italian Journal of Library, Archives and Information Science= Rivista italiana di biblioteconomia, archivistica e scienza dell'informazione: 12, 2, 2021 , pages=. 2021 , publisher=

    2021

  6. [14]

    Procedia Computer Science , volume=

    Eurollm: Multilingual language models for europe , author=. Procedia Computer Science , volume=. 2025 , publisher=

    2025

  7. [15]

    Imparting

    Mayeur, Ingrid , doi =. Imparting. Expanding. 2017 , pages =

    2017

Pith tools

Reviewed July 8, 2026 · model on record in the stance chip above.