{"id":"be898a0d-10fe-48e1-a05a-22e3d14828d4","arxiv_id":"2607.05956","paper_version":1,"verdict":"UNVERDICTED","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper presents the design of ReSearch_SSH, a GraphRAG-based LLM adaptation for multilingual SSH literature discovery within the ISIDORE platform, currently in a preparatory phase with no experimental results yet.","lead":"This paper describes a plan to adapt open multilingual LLMs for Social Sciences and Humanities research using a knowledge-graph-enhanced RAG system built on the ISIDORE search platform. A smart generalist might read it to understand how European infrastructures are approaching domain-specific, regulation-compliant AI for non-English scholarly corpora.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"No significant objection identified beyond the reader's core assessment: this is a design document with no empirical results, and the click-data retrieval assumption is the most technically novel untested component.","rationale":"The reader's assessment is accurate and well-calibrated. The paper is explicitly an ongoing project description — it states multiple times that no experiments have been run ('The project is currently ongoing,' 'The first stages of domain alignment and retrieval-oriented fine-tuning are scheduled to begin shortly'). The verdict of UNVERDICTED with MODERATE confidence is appropriate: the paper's status as a design document is unambiguous from the text itself. The reader's identification of the click-data retrieval assumption as the weakest link is reasonable. It is the most infrastructure-specific and novel component (§3.2), and it rests on well-known challenges in learning-to-rank from implicit feedback that the paper does not address. However, I would note that this concern is one among many untested assumptions in the architecture — the GraphRAG grounding, the multilingual cross-lingual performance with heavily imbalanced training data, and the instruction-tuning for literature synthesis are all equally unvalidated. The click-data component stands out only because it is the most technically specific and the most dependent on a signal whose quality is questionable. The paper does not overclaim: it consistently frames everything as proposed or planned, and the modular design with fallbacks (e.g., replacing click-based retrieval with semantic retrieval) shows awareness of potential failure modes. The legal and compliance framework (§4) is thorough and internally consistent. The evaluation plan (§3.3) with independent external assessment and expert panels is well-conceived. There is no internal inconsistency, no misrepresentation of the work's status, and no claim that exceeds what is justified by the current state. The paper is simply premature for evaluation as a research contribution, which the reader correctly identified.","tokens_in":10122,"tokens_out":1750,"duration_ms":143473,"concrete_test":"Once the retrieval component is trained, run a controlled comparison on a held-out SSH query set: (1) interaction-aware retrieval model trained on ISIDORE click data without debiasing, (2) same model with position-bias correction (e.g., propensity-weighted loss or a cascade click model), (3) a purely semantic baseline (e.g., multilingual E5 fine-tuned on SSH text). Measure nDCG@10 and recall@100 against expert-annotated relevance judgments from the DH panel. If (1) does not outperform (3) by a meaningful margin, or if (2) substantially outperforms (1), the click-data signal without debiasing is not adding value over semantic retrieval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the paper's status as a design document with no trained models, no experiments, and no results. I agree with the reader's identification of the interaction-aware retrieval component (§3.2) as the most load-bearing untested assumption. The paper proposes fine-tuning a retrieval model on ISIDORE historical query–click data, promoting clicked documents and demoting ignored ones. This is essentially learning-to-rank from implicit feedback, which is known to suffer from position bias (users click higher-ranked results more often regardless of relevance), selection bias (only results shown can be clicked), and presentation bias. The paper does not discuss any debiasing strategy (e.g., click models, counterfactual estimation, or propensity-weighted training). Additionally, in a scholarly search context, clicks may reflect convenience or visibility rather than scholarly relevance — a concern the paper itself raises about commercial platforms but does not address for its own signal source. That said, the paper does not claim this works; it is proposed as a strategy, and the authors acknowledge the component is replaceable by 'purely semantic retrieval models' if behavioral signals are unavailable. The concern is therefore about whether the most novel architectural component will function as intended, not about any internal inconsistency or overclaiming in the paper as written. No other concern rises above this one: the multilingual imbalance (English dominates ISTEX at 16,000–21,000M tokens vs. Italian at 10–15M) is a practical risk but is acknowledged and partially addressed through supplementary Italian DH corpora. The legal/compliance framework is well-structured and internally consistent. The evaluation plan is appropriately external and hybrid. The fundamental issue remains that nothing has been validated.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper presents ReSearch_SSH, an ongoing use case within the LLMs4EU project, proposing a GraphRAG architecture for domain-adaptive LLMs in the Social Sciences and Humanities (SSH). The system extends the French ISIDORE search platform by combining (i) continued pre-training on the ISTEX SSH corpus (~3M documents), (ii) retrieval fine-tuning on historical ISIDORE query–click data, (iii) knowledge-graph enrichment via Wikidata and OpenAIRE, and (iv) instruction-tuned generation for scholarly synthesis tasks. The paper also describes a compliance-by-design framework (GDPR, EU AI Act, copyright/TDM) and a hybrid evaluation protocol combining quantitative benchmarking with expert-panel assessment. The project is in a preparatory phase; no models have been trained, no experiments run, and no results are reported.","tokens_in":10897,"tokens_out":1305,"duration_ms":289017,"significance":"The manuscript addresses a genuine gap: most AI-powered scholarly discovery tools are English-centric and citation-driven, poorly aligned with SSH practices that emphasize multilingualism, book-based scholarship, and qualitative evaluation. The choice of data sources is well-motivated and concrete (ISTEX SSH subset, AIUCD proceedings, Umanistica Digitale), and the regulatory discussion is substantive rather than boilerplate. The interaction-aware retrieval component (§3.2) is the most architecturally novel element. However, the paper is a project description with no empirical validation, no trained models, no baselines, and no quantitative findings of any kind. The significance therefore rests entirely on the design's plausibility, which has not been tested.","major_comments":[{"comment":"§3.2, retrieval fine-tuning paragraph: The interaction-aware retrieval component — the most novel architectural element — is proposed without any discussion of known biases in click-based implicit feedback. Learning-to-rank from click data is well-documented to suffer from position bias, selection bias, and presentation bias (e.g., Craswell et al. 2008; Joachims et al. 2017). The paper itself critiques commercial platforms for letting ranking bias shape scholarly visibility (§1), yet proposes using ISIDORE click data without addressing how the same biases might affect its own retrieval model. This is load-bearing because this component is identified as the most infrastructure-specific and novel part of the architecture. The authors should at minimum discuss what debiasing strategies (click models, counterfactual estimation, propensity weighting) will be considered, or justify why they相信o","section":null},{"comment":"§3.1, Table 2: The token distribution shows English at 16,000–21,000M tokens versus Italian at 10–15M — a ratio exceeding 1,000:1. The paper claims the corpus 'supports large-scale alignment to SSH discourse while preserving multilingual exposure' and that Italian DH corpora enable 'cross-lingual bridging between Italian, French and English SSH traditions.' At this imbalance, it is unclear whether the Italian data is sufficient for any meaningful cross-lingual transfer, especially for continued pre-training. The paper should discuss whether this volume is adequate for the stated multilingual objectives or whether the Italian component is effectively limited to task-tuning rather than domain alignment.","section":null},{"comment":"The paper contains no experimental results, no trained models, no baseline comparisons, and no quantitative evaluation of any component. While §5 transparently states the project is in a 'preparatory phase,' the manuscript as submitted does not constitute a research contribution in the conventional sense — it is a design document. For a journal publication, at minimum a proof-of-concept experiment (e.g., domain alignment pilot, retrieval quality with vs. without click-based fine-tuning, or a small-scale GraphRAG pipeline demonstration) would be needed to substantiate the design's feasibility. The authors should clarify what minimal empirical evidence they can provide to validate the architecture before submission.","section":null}],"minor_comments":[{"comment":"§3.1: 'the use case may introduce also scientific publications and academic blog posts from the Hypotheses platform, together with research data and metadata from the Nakala repository, will be incorporated' — the sentence structure is broken; 'may introduce' and 'will be incorporated' conflict syntactically.","section":null},{"comment":"§5: 'attention will devoted to metadata normalisation' — missing 'be' ('will be devoted').","section":null},{"comment":"§3.2: The reference to 'GPT-5' as a candidate for synthetic data generation appears premature given the project's timeline; the authors should either confirm availability or use a more generic reference to frontier commercial models.","section":null},{"comment":"Table 2: The English token range (16,000–21,000M) is unusually wide for an estimate; clarifying the basis for this range would improve credibility.","section":null},{"comment":"§3.2: The base model is not specified ('will be drawn from the pool of open multilingual foundation models developed or endorsed within the European AI ecosystem'). While the authors commit to reporting this later, a shortlist of candidates under consideration would strengthen the design discussion.","section":null},{"comment":"The paper would benefit from a related-work section positioning the GraphRAG approach (Edge et al. 2024) against other domain-adaptive RAG systems, particularly in non-English or non-STEM application contexts.","section":null}],"recommendation":"major_revision","confidential_remarks":"This is a well-written project description from a credible team within a major European infrastructure initiative. The design is coherent and the regulatory discussion is above average for this type of paper. However, the manuscript has zero empirical content — no experiments, no models, no results. For a serious journal, this is below the threshold for acceptance regardless of how well-motivated the design is. The authors should be encouraged to resubmit once at least a pilot experiment is available. The click-data debiasing concern (major comment 1) is the most important technical issue to address in any revision, as it goes to the heart of the paper's claimed novelty."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive reading of our manuscript. The report correctly identifies the paper as a design document for an ongoing project (ReSearch_SSH within LLMs4EU) and raises three substantive points: (1) the absence of debiasing discussion for click-based retrieval fine-tuning, (2) the adequacy of the Italian corpus at the documented token imbalance, and (3) the lack of any empirical validation. We address each below. On the first point, we agree the omission is significant and will revise. On the second, we will clarify the role of Italian data and acknowledge its limitations more explicitly. On the third, we accept that a proof-of-concept experiment is needed and will commit to including one in the revised submission, while being transparent about the timeline constraints of the project.","responses":[{"response":"The referee is correct that this is a genuine gap in the manuscript. The irony of critiquing commercial platforms for ranking bias in §1 while proposing click-based fine-tuning in §3.2 without discussing debiasing is a fair and important observation. We will add a dedicated paragraph in §3.2 discussing the known bias sources (position bias, selection bias, presentation bias) and the debiasing strategies we plan to evaluate. Specifically, we intend to investigate: (a) propensity-weighted inverse propensity scoring based on result rank position, following the framework of Joachims et al. (2017); (b) click model-based estimation of relevance (e.g., position-based and cascade models) to derive debiased training signals; and (c) counterfactual estimation techniques for offline policy evaluation. We will also note that ISIDORE's interface design differs from commercial platforms in ways that may partially mitigate some biases (e.g., less aggressive personalization, no advertising-driven ranking), but we agree this does not eliminate the need for explicit debiasing. The revised manuscript will cite the relevant literature (Craswell et al. 2008; Joachims et al. 2017) and frame the debiasing strategy as an integral part of the retrieval fine-tuning design rather than an afterthought.","revision_made":"yes","referee_comment":"§3.2, retrieval fine-tuning paragraph: The interaction-aware retrieval component is proposed without any discussion of known biases in click-based implicit feedback (position bias, selection bias, presentation bias). The paper critiques commercial platforms for ranking bias yet proposes using ISIDORE click data without addressing how the same biases might affect its own retrieval model."},{"response":"The referee raises a valid concern about the adequacy of the Italian component at the documented scale. We agree that 10–15M tokens of Italian in the ISTEX SSH subset is insufficient for meaningful domain alignment via continued pre-training on its own. The manuscript's claim that the corpus 'supports large-scale alignment to SSH discourse while preserving multilingual exposure' is accurate for French (450–600M tokens) but overstated for Italian. We will revise the text to clarify that: (1) the Italian ISTEX data contributes marginally to continued pre-training and is not expected to produce robust Italian-language domain alignment on its own; (2) the targeted Italian DH corpora (AIUCD proceedings, Umanistica Digitale) are primarily intended for task-tuning and genre/terminology adaptation rather than large-scale domain alignment; and (3) meaningful cross-lingual bridging for Italian will depend on the multilingual transfer capabilities of the base model (e.g., EuroLLM) rather than on Italian-specific pre-training volume. We will also note that the project's current phase focuses on enabling Italian and English querying over predominantly French materials, which is a more modest and achievable goal than full multilingual domain alignment. The revised manuscript will reframe the Italian component's role accordingly and avoid overstating its contribution to cross-lingual transfer.","revision_made":"yes","referee_comment":"§3.1, Table 2: The token distribution shows English at 16,000–21,000M tokens versus Italian at 10–15M — a ratio exceeding 1,000:1. The paper claims the corpus supports multilingual exposure and cross-lingual bridging, but it is unclear whether the Italian data is sufficient for meaningful cross-lingual transfer, especially for continued pre-training."},{"response":"We accept this criticism. The manuscript as submitted is a design document, and while §5 transparently states the project is in a preparatory phase, we agree that a journal publication requires at least minimal empirical evidence. We will commit to including a proof-of-concept experiment in the revised submission. The most feasible pilot within our timeline is a small-scale GraphRAG pipeline demonstration over a subset of the ISIDORE corpus, comparing retrieval quality with and without knowledge-graph enrichment (Wikidata/OpenAIRE entity linking). This would directly validate the most architecturally novel element the referee identifies as load-bearing. We will also include a preliminary domain-alignment pilot: continued pre-training of a small open multilingual model on a French SSH subset, evaluated via perplexity reduction and domain-specific term recognition compared to the base model. We cannot commit to full retrieval fine-tuning with click-based debiasing experiments at this stage, as the click data preparation and debiasing pipeline are still under development. We will be transparent about what the pilot does and does not demonstrate, and will frame it as initial feasibility evidence rather than full validation of the architecture.","revision_made":"yes","referee_comment":"The paper contains no experimental results, no trained models, no baseline comparisons, and no quantitative evaluation of any component. For a journal publication, at minimum a proof-of-concept experiment would be needed to substantiate the design's feasibility."}],"tokens_in":10123,"tokens_out":1185,"duration_ms":168166,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Bottom line: this is a project description, not a research paper. No models trained, no experiments run, no results. The reader's assessment is accurate on this point. But the paper does have genuine value as a design document, and I think it deserves a fair hearing despite the absence of empirical content.","headline":"Design document for a GraphRAG-based SSH search system — no results yet, but the compliance framework and infrastructure integration are the real contribution","tokens_in":10928,"tokens_out":131,"would_cite":false,"duration_ms":89927,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"GraphRAG for SSH: grounding LLMs in scholarly practice, not just text","keywords":[],"falsifier":"The interaction-aware retrieval component — the most novel part of the architecture — would be falsified if fine-tuning on ISIDORE click data produces retrieval rankings that expert panels judge no better than, or worse than, purely semantic retrieval baselines. More broadly, the entire proposal would be undermined if the GraphRAG pipeline, once implemented, produces literature syntheses that expert DH panels rate as epistemically unreliable or if hallucination rates exceed acceptable thresholds in the independent evaluation. Since no results exist yet, the falsifier is prospective: the first","tokens_in":10423,"feed_emoji":"🔍","tokens_out":1166,"duration_ms":177082,"temperature":0.7,"pith_summary":"This paper proposes a design — not yet a validated system — for adapting large language models to the specific epistemic and methodological needs of Social Sciences and Humanities (SSH) research. The central object is a GraphRAG (knowledge-graph-enhanced retrieval-augmented generation) architecture built on top of ISIDORE, an existing French SSH search engine. The architecture has three coupled layers: a retrieval model fine-tuned on historical user click data from ISIDORE so that ranking reflects observed scholarly search behaviour rather than pure textual similarity; a knowledge-graph enrichment layer using Wikidata and OpenAIRE for entity disambiguation, structured navigation, and document-level comparison; and a generative model instruction-tuned on SSH corpora to produce literature syntheses and state-of-the-art overviews that remain traceable to explicitly retrieved sources. The authors argue that this combination can support core SSH workflows — literature review, thematic exploration, comparative analysis — while preserving multilingualism (French, Italian, English), source traceability, and compliance with European regulatory frameworks including the EU AI Act, GDPR, and copyright law. The paper frames the entire effort as a corrective to dominant AI-powered discovery tools (Semantic Scholar, Elicit, Scopus) that privilege English-language journal articles, citation-based metrics, and STEM epistemic norms, arguing that SSH scholarship requires tools sensitive to disciplinary diversity, multilingual sources, non-article outputs (monographs, editions, datasets, blogs), and interpretive plurality — the same Dante text read differently by a literary scholar, a philologist, a historian, and a digital humanist. The project is in an advanced preparatory phase: corpora are being consolidated, fine-tuning has not yet begun, and the evaluation framework — combining quantitative benchmarking with expert panels from Franco-Italian Digital Humanities communities — is under development.","feed_headline":"GraphRAG for SSH: grounding LLMs in scholarly practice, not just text","feed_subtitle":"A Franco-Italian project proposes fine-tuning retrieval on real scholar click data and enriching with knowledge graphs to keep AI-assisted搜索","key_machinery":"The load-bearing mechanism is the coupling of three components: (1) a retrieval model fine-tuned on ISIDORE historical query–click data, where clicked documents are promoted and ignored results demoted — this interaction-aware ranking is the most novel and infrastructure-specific element; (2) a knowledge-graph layer drawing on Wikidata and OpenAIRE for entity disambiguation, semantic expansion, and structured navigation across authors, themes, institutions, and citations; (3) a generative model instruction-tuned on SSH corpora (approximately 3 million documents from the ISTEX SSH subset, supplemented by Italian DH conference proceedings and journals) to produce coherent literature syntheses.","core_discovery":"The paper does not report experimental results. Its contribution is an architectural proposal: that a GraphRAG pipeline combining (1) interaction-aware retrieval fine-tuned on real scholarly click data, (2) public knowledge-graph enrichment for entity disambiguation and structured navigation, and (3) instruction-tuned generation on curated multilingual SSH corpora can produce a domain-adaptive LLM system that grounds its outputs in traceable sources while respecting the epistemic plurality, multilingualism, and regulatory constraints characteristic of SSH research. The authors explicitly state this is ongoing work in a preparatory phase, with the first fine-tuning experiments scheduled to ''","pith_inferences":[],"forward_implications":["If the click-data fine-tuning produces retrieval rankings that genuinely align with scholarly relevance, the approach could be ported to any research infrastructure that logs user interaction data — making interaction-aware retrieval a transferable design pattern beyond ISIDORE.","The architecture''s modularity (the authors note the click-data component is replaceable by purely semantic retrieval, and TEI/XML is replaceable by plain text with structured metadata) means the minimal viable version of this system could be deployed by institutions with far less infrastructure than ISIDORE, broadening access.","If expert-panel evaluation reveals that SSH scholars systematically disagree with quantitative benchmarks on what counts as relevant, this would constitute evidence that generic IR/LLM evaluation protocols are epistemically mismatched to humanities research — a finding with implications beyond this project.","The compliance-by-design approach (treating GDPR, AI Act, and copyright constraints as architectural components rather than post-hoc constraints) could serve as a template for other domain-adaptive LLM projects operating under European regulation."],"fun_headline_variants":["GraphRAG architecture for multilingual SSH: knowledge graphs meet scholarly corpora","Grounding domain-adaptive LLMs in SSH research infrastructures and knowledge graphs","Knowledge-graph-enriched retrieval for multilingual SSH literature synthesis","Domain-adaptive LLMs for SSH: combining knowledge graphs and multilingual corpora","Traceable generation for SSH: GraphRAG with scholarly click data and knowledge graphs"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The design assumes that historical click data from ISIDORE — which documents users clicked on and which they ignored — is a reliable signal for scholarly relevance. But clicks may reflect visibility, ranking bias, or convenience rather than true scholarly value, and no evidence is provided that this signal improves retrieval quality. The authors themselves acknowledge this component is replaceable by ''purely semantic retrieval models'' if behavioural signals are unavailable,","fun_headline_variants_meta":{"raw":{"variants":["GraphRAG architecture for multilingual SSH: knowledge graphs meet scholarly corpora","Grounding domain-adaptive LLMs in SSH research infrastructures and knowledge graphs","Knowledge-graph-enriched retrieval for multilingual SSH literature synthesis","Domain-adaptive LLMs for SSH: combining knowledge graphs and multilingual corpora","Traceable generation for SSH: GraphRAG with scholarly click data and knowledge graphs"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":637,"prompt_tokens":538,"completion_tokens":99,"prompt_tokens_details":null},"tokens_in":538,"tokens_out":99,"duration_ms":28822,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T19:58:21.307184+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"The interaction-aware retrieval component — the most novel part of the architecture — would be falsified if fine-tuning on ISIDORE click data produces retrieval rankings that expert panels judge no better than, or worse than, purely semantic retrieval baselines. More broadly, the entire proposal would be undermined if the GraphRAG pipeline, once implemented, produces literature syntheses that expert DH panels rate as epistemically unreliable or if hallucination rates exceed acceptable thresholds in the independent evaluation. Since no results exist yet, the falsifier is prospective: the first","supporting_citations":[],"review_version":1}