{"id":"ff53f7f9-288d-4664-bc9b-9b60f910452d","arxiv_id":"2507.04395","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A retrieval-augmented generation system answers religion and spirituality questions from 7,500 UN resolutions, with a 100-question expert evaluation.","lead":"SpiritRAG is a chatbot-style search tool that answers questions about religion and spirituality using 7,500 United Nations resolution documents on health and education. Researchers and policy analysts could use it to find context-sensitive references quickly, but the evidence for its practical value currently rests on a small, two-expert evaluation without a baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation never verifies the factual correctness of generated answers: the five answer-generation dimensions omit any accuracy/faithfulness metric, so the 100-question pilot cannot support the claim of practical value for a Q&A system.","rationale":"The reader's conditional verdict already identifies the central weakness: the usefulness claim rests on two experts' Likert ratings with no baseline, no inter-rater reliability, and no independent factual verification. My stress-test sharpens this into the single most load-bearing gap: the answer-generation evaluation does not contain any metric for factual correctness at all. For a retrieval-augmented Q&A system, coherence and engagement are secondary; a system that produces confident, well-written answers with fabricated resolution citations would receive high scores on all five reported answer dimensions. The paper's own limitations section admits the absence of ground-truth answers, so this is not an inference about reviewer standards but a direct consequence of the reported design. A concrete fact-checking pass over the 100 generated answers would settle whether the pilot demonstrates practical value, because the central claim is not merely that experts found the interface pleasant, but that SpiritRAG enables accurate, context-sensitive retrieval and synthesis from the UN archive. I do not see a reason to reject the paper outright: the corpus construction, modular architecture, and reproducible deployment are real contributions, and the low actionability scores are honestly discussed. But the claimed demonstration of practical value is conditional on passing an objective correctness check, so the reader's CONDITIONAL verdict is appropriate; no verdict change is needed.","tokens_in":12376,"tokens_out":3834,"duration_ms":46088,"concrete_test":"Independently fact-check the 100 answers from the best configuration (Qwen3-Embedding-0.6B retriever + Qwen3-1.7B generator). Two annotators, blind to the original ratings, mark every specific claim in each answer (resolution number, date, policy statement, quote) as supported by the retrieved documents, contradicted, or hallucinated. Report per-question accuracy, the fraction of answers with at least one unsupported citation, and Cohen's kappa. If more than ~10% of answers contain a fabricated resolution number or an unsupported attribution, the 'practical value and usefulness' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4's demonstration of practical value is built entirely on mean Likert ratings from two domain experts (Tables 2 and 3). The five answer-generation dimensions defined in Table 9 are Congruence, Coherence, Relevance, Creativity, and Engagement; none of them measures whether the answer's content is factually correct or properly grounded in the retrieved UN resolutions. The document-retrieval dimension 'Accuracy' (Table 9) asks whether a retrieved document is trustworthy, not whether the generated answer's claims, resolution numbers, dates, or quotations are accurate. The system's value as a Q&A tool therefore depends on an unmeasured property: the 100 answers could be coherent, relevant, and engaging while systematically hallucinating citations or misattributing content, and the reported scores would not change. The paper itself lists the absence of ground-truth answers as a limitation (Section 6, Limitations), confirming that no objective correctness check was performed. Combined with the use of self-written questions by the two expert raters and no reported inter-rater reliability, the pilot evidence is insufficient to establish that SpiritRAG improves on existing archival search for researchers and policymakers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SpiritRAG is a retrieval-augmented generation (RAG) system for question answering over approximately 7,500 United Nations resolution documents related to religion and spirituality (R/S) in the health and education domains. The system combines a sentence-embedding retriever with a reranking step and a Qwen3-based answer generator, packaged as a chat-based web interface. The paper contributes the system architecture, a multilingual corpus of UN resolutions, and an expert-authored test set of 100 questions. The central claim is that the system enables complex, context-sensitive searches of a large archive and that a pilot evaluation with two domain experts demonstrates its practical value and usefulness.","tokens_in":12692,"tokens_out":3382,"duration_ms":34703,"significance":"If the system performs as claimed, it would be a useful proof-of-concept for domain-specific RAG in archival research and policy analysis, offering a low-barrier interface to a large, noisy document collection. The open release of the corpus and code is a concrete strength, as is the modular architecture that eases extension to other domains. The paper also provides a curated 100-question test set, which could support future benchmarking. However, the paper's central claim of practical value rests on a pilot evaluation that does not measure factual correctness, lacks a baseline, and reports only mean Likert scores from two raters who authored the questions. These are significant gaps that must be addressed before the claim can be considered supported.","major_comments":[{"comment":"The evaluation does not verify the factual correctness of generated answers, yet the abstract's claim of 'practical value and usefulness' for a Q&A system depends on answer accuracy. The five answer-generation dimensions (Congruence, Coherence, Relevance, Creativity, Engagement) in Table 9 omit any accuracy or faithfulness metric. The document-retrieval dimension 'Accuracy' asks whether a retrieved document is trustworthy, not whether the generated answer's claims, resolution numbers, dates, or quotations are correct. Consequently, the reported scores would be unchanged even if the system systematically hallucinated citations or misattributed content. The paper itself acknowledges the absence of ground-truth answers in Section 6 (Limitations), confirming that no objective correctness check was performed. I recommend adding an explicit factual-verification component, such as independent expert fact-checking of a sample of answers or an automatic faithfulness metric, and reporting the corresponding error rates.","section":"Section 4, Tables 2 and 3"},{"comment":"The central claim that SpiritRAG improves on existing archival search is not supported by any baseline comparison. The paper states that current UN search interfaces are poorly designed and that general-purpose LLMs lack contextual sensitivity, but the pilot evaluation compares only internal configurations of SpiritRAG (different retrievers and generators). Without a comparison to the existing UN document system, a general-purpose search engine, or a non-RAG LLM baseline, the reported ratings cannot substantiate the claim that SpiritRAG makes large, noisy datasets more accessible. Please include a baseline condition in the evaluation.","section":"Section 4"},{"comment":"The evaluation relies on mean Likert ratings from two domain experts who also authored the 100 test questions, and no inter-rater reliability is reported. With only two raters and self-authored questions, the scores are susceptible to leniency and representativeness bias, and mean ratings alone do not indicate agreement or consistency. Report per-rater scores and a chance-corrected agreement measure (e.g., weighted Cohen's kappa) for each dimension. Ideally, have additional independent experts write or evaluate a subset of questions, or at minimum discuss how the potential bias was mitigated.","section":"Section 4 and Appendix A"},{"comment":"The limitation statement 'in the absence of ground-truth answers and relevant documents, we did not fine-tune the document retriever for optimal performance' also confirms that no ground-truth-based evaluation was performed. This is not merely a limitation for fine-tuning; it directly undermines the abstract's claim that the pilot 'demonstrates the practical value and usefulness of SpiritRAG.' A Q&A system's practical value is inseparable from whether its answers are correct. The paper should either temper the central claim to reflect that only perceived quality was measured, or supply a ground-truth-based evaluation.","section":"Section 6, Limitations"}],"minor_comments":[{"comment":"The sentence 'Evaluators noted the considerable improvement in performance between Sentence-BERT and Qwen3-Embedding as document retriever, and particularly a significant and consistent improvement in answer length and detail of Qwen3-0.6B in comparison with Qwen3-1.7B as answer generator' appears to have the model names reversed; Table 2 shows Qwen3-1.7B outperforming Qwen3-0.6B on all reported dimensions.","section":"Section 4"},{"comment":"The caption states 'Human evaluation results of SpiritRAG on 50 health + R/S test questions,' but the paper reports results for 100 questions (Tables 2 and 3). Clarify whether Figure 1 covers only the health subset or the full set.","section":"Figure 1 caption"},{"comment":"The corpus size is described as 'nearly 7,500 documents' and 'approx. 7,500,' but Table 1 lists per-language document counts that are much higher when summed (overlapping translations). Clarify that the 7,500 figure refers to unique multilingual documents, and state how overlap is handled.","section":"Table 1"},{"comment":"The relevance score r(Di; q) uses L2 (Euclidean) distance while the initial retrieval ranking uses cosine similarity. Clarify whether a lower L2 distance indicates higher relevance and how the two measures are aligned in the ranking.","section":"Section 2.2, Equation (3)"},{"comment":"The example answer in Table 10 is useful, but the paper would benefit from additional examples that illustrate failure cases, particularly with respect to factual grounding, so readers can gauge typical error modes.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper's system and resources are valuable contributions, and the core idea is sound. The evaluation, however, is not yet sufficient for the central claim: it lacks factual verification, a baseline, and inter-rater reliability, and the two expert raters authored the questions. These issues are addressable within the scope of the manuscript (e.g., adding a fact-checking protocol, a baseline comparison, and IRR reporting), so I recommend major revision rather than rejection. I also note a minor mismatch in the text describing the generator comparison that should be corrected. The paper's open-source availability is a strength that supports reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe real contribution here is the resource, not the system. A multilingual corpus of roughly 7,500 UN resolutions on health and education, plus 100 expert-written questions, is a solid, reusable asset for anyone working on religion/spirituality in international policy. The authors are also honest about what they did not do: no fine-tuning, no ground truth, and a clear limitations section. That transparency counts for something.\n\nThe system itself is a standard RAG stack - embedding retriever, max/avg re-ranking, Qwen3 generator - and the technical novelty is modest. That is fine for an application paper, but the evaluation needs to carry weight. It does not. Two domain experts rated their own 100 questions on five-point Likert scales, with no inter-rater agreement reported, no baseline comparison, and no independent check on whether the generated answers actually contain correct resolution numbers, dates, or quotations. The stress-test note is right: the answer-generation dimensions listed in Table 9 - Congruence, Coherence, Relevance, Creativity, Engagement - contain no faithfulness or accuracy dimension. A Q&A system can score well on all five while hallucinating its citations, and the reported numbers would not change. The paper's own acknowledgment that ground-truth answers are missing confirms that this is not an oversight; correctness was simply never measured.\n\nA few smaller issues: the smallest generator scored below 3 on several dimensions, which undercuts the practical value narrative, and the dataset/code links promised in the abstract are absent. The temporal analysis and subject clustering are nice but tangential.\n\nWho should read this? People building RAG systems for specialized archives will find the corpus and the candid write-up useful, and the evaluation design is a good teaching example of what not to do when claiming usefulness. The paper deserves a serious referee - it is coherent, the resource is real, and the authors seem willing to improve the evaluation - but the current submission needs major revision before the value claim is credible. I would suggest the editor send it out with clear instructions that the evaluation must be reworked rather than desk-rejecting it outright.","headline":"A genuinely useful new corpus for a niche domain, wrapped around a standard RAG pipeline whose evaluation, as reported, does not yet establish the paper's central claim of practical value.","tokens_in":682,"tokens_out":1775,"would_cite":false,"duration_ms":31411,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A chat-based Q&A system retrieves and cites religion- and spirituality-related content from 7,500 UN resolutions on health and education, with expert ratings on 100 questions indicating practical usefulness.","keywords":["retrieval-augmented generation","question answering","religion and spirituality","United Nations resolutions","health and education","document retrieval","large language models","digital humanities"],"falsifier":"Run the 100 published test questions through a standard keyword or phrase search over the same UN corpus and have independent evaluators fact-check every SpiritRAG answer against its cited resolutions; if the keyword baseline matches or beats SpiritRAG on relevance and accuracy, or if a substantial share of generated claims cannot be found in the cited documents, the system's claimed practical advantage would be undercut.","tokens_in":12156,"feed_emoji":"🕊️","tokens_out":12572,"duration_ms":112581,"temperature":0.7,"pith_summary":"SpiritRAG is a chat-based question-answering system built to make religion- and spirituality-related content in United Nations archives searchable without specialist archival skills. The paper's central claim is that retrieval-augmented generation can handle the context-dependence of these concepts: users ask natural-language questions, and the system retrieves relevant resolutions from a corpus of about 7,500 health and education documents, generates a synthesized answer, and links it to cited sources. To test this, the authors compiled 100 questions with two domain experts and collected five-point ratings on retrieved documents and generated answers. The best configuration, Qwen3-Embedding-0.6B for retrieval and Qwen3-1.7B for generation, received mean ratings above the midpoint on most dimensions for both health plus R/S and education questions, which the authors take as evidence of practical value. The contribution is a working proof-of-concept, a curated UN resolution corpus, and a reusable expert-written evaluation set.","feed_headline":"Chat-based search opens 7,500 UN resolutions on religion","feed_subtitle":"SpiritRAG retrieves, synthesizes, and cites answers on health and education policy; two experts rated 100 test questions.","key_machinery":"The load-bearing mechanism is retrieval-augmented generation (RAG): retrieve relevant documents from a corpus first, then condition the language model's answer on those documents. The retriever maps the query and each document to embeddings with the same text encoder, pre-fetches the top-$n$ resolutions by cosine similarity, then re-ranks them by a weighted combination of maximum and average Euclidean distance between the query embedding and the sentence embeddings within each document, with the maximum weighted at $\\alpha = 0.7$. The top-$k$ documents, together with an optional user-uploaded parsed PDF, are passed through a fixed prompt template to an instruction-tuned Qwen3 model, which produces a Markdown answer pointing back to cited resolution documents. The corpus side uses keyword-based crawling of the UN Official Document System, Docling-based paragraph parsing, Faiss indexing, and vLLM-accelerated generation.","core_discovery":"On the paper's own terms, the discovery is that a domain-specific retrieval-augmented generation pipeline can operationalize 'religion' and 'spirituality' in archival search. The system retrieves UN resolutions by dense embedding similarity, re-ranks them by sentence-level relevance, and conditions a Qwen3 model on the top documents to write answers that cite original resolutions. The authors claim this makes complex, context-sensitive queries, such as how religious freedom or cultural appropriateness has been framed over time, answerable through an ordinary chat interface. The pilot evaluation reports mean expert ratings of 4.26 for retrieval relevance and 4.18 for answer relevance in the health plus R/S domain, and 3.51 and 3.98 respectively in education, with the larger retriever and generator consistently outperforming the smaller ones. The paper frames SpiritRAG as a proof-of-concept for LLM-augmented work in public policy, historical scholarship, and digital humanities, and as a complement to topic-modeling studies of these archives.","pith_inferences":["Not in the paper: a direct comparison of SpiritRAG with the UN's existing keyword search over the same corpus; running the 100 published questions through both interfaces would quantify how much of the reported usefulness comes from the RAG pipeline rather than from the underlying archive.","Not in the paper: inter-rater reliability or independent fact-checking, so the stability of the mean ratings is unknown; a second set of raters scoring the same 100 answers would test whether the usefulness scores reproduce.","A testable consequence the authors leave implicit: because replacing the retriever produced the largest rating gains, fine-tuning or scaling the retriever may add more practical value than scaling the generator.","The high temporality ratings suggest an undeveloped feature the authors do not spell out: letting users constrain retrieval to explicit time windows would make diachronic questions such as 'since 1946' answerable with greater precision."],"forward_implications":["A researcher can ask multi-part, diachronic questions such as how UN language on religious freedom or cultural appropriateness changed over time and receive an answer with links to the underlying resolutions.","A policymaker without archival training can trace what UN resolutions say about faith-based organizations, religious discrimination, or health equity, because the chat interface handles contextual sensitivity that keyword search misses.","The same modular pipeline can be extended to other UN bodies or non-UN document collections, since the system already accepts user-uploaded PDFs and the architecture is designed for other domain-specific sources.","The 100-question expert test set gives later systems a reusable benchmark for R/S-related retrieval and generation in UN archives.","The paper reports that the best system configuration answers in about seven seconds, suggesting the approach is light enough for real-time use in research and policy settings."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation paradigm that SpiritRAG instantiates.","marker":"Lewis et al. (2020)"},{"why":"Provides Sentence-BERT, the lower-performing document retriever baseline in the evaluation.","marker":"Reimers and Gurevych, 2019"},{"why":"Provides Qwen3-Embedding, the higher-performing document retriever in the evaluation.","marker":"Zhang et al., 2025"},{"why":"Provides the Qwen3 instruction-tuned language models used for answer generation.","marker":"Yang et al., 2025"},{"why":"Supplies the vLLM inference engine used to accelerate answer generation.","marker":"Kwon et al., 2023"},{"why":"Supplies the Docling parser that converts resolution PDFs into paragraph-level chunks for embedding.","marker":"Deep Search Team, 2024"},{"why":"Supplies the modular interface design principle on which SpiritRAG's architecture is modeled.","marker":"Gao et al. (2024)"},{"why":"Documents the history of the 'spiritual dimension' in global health that motivates the domain focus.","marker":"Peng-Keller et al., 2022"}],"fun_headline_variants":["Chat with 7,500 UN resolutions on faith","AI turns 7,500 UN religion resolutions into chat answers","SpiritRAG: Q&A for UN faith documents","UN faith resolutions get a Q&A chatbot","Ask the UN archive about faith in chat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of practical value rests on two domain experts' average five-point ratings of 100 questions they wrote themselves, with no baseline system, no inter-rater agreement statistic, and no independent check that the generated answers' factual claims are supported by the cited resolutions.","fun_headline_variants_meta":{"raw":{"variants":["Chat with 7,500 UN resolutions on faith","AI turns 7,500 UN religion resolutions into chat answers","SpiritRAG: Q&A for UN faith documents","UN faith resolutions get a Q&A chatbot","Ask the UN archive about faith in chat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002022,"raw_usage":{"total_tokens":7890,"prompt_tokens":960,"completion_tokens":6930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":6855}},"tokens_in":576,"tokens_out":6930,"duration_ms":47750,"temperature":1.0,"reasoning_tokens":6855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:47:47.421104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 100 published test questions through a standard keyword or phrase search over the same UN corpus and have independent evaluators fact-check every SpiritRAG answer against its cited resolutions; if the keyword baseline matches or beats SpiritRAG on relevance and accuracy, or if a substantial share of generated claims cannot be found in the cited documents, the system's claimed practical advantage would be undercut.","supporting_citations":[],"review_version":1}