{"id":"e41af93b-1649-471b-82ab-55743fc2f457","arxiv_id":"2608.13410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ParliamentRAG is an authority-aware, multi-view retrieval-augmented system for Italian parliamentary proceedings that achieves perfect verbatim quotation faithfulness and near-perfect group coverage, matching Google NotebookLM on overall satisfaction.","lead":"This paper describes ParliamentRAG, an AI system that summarizes Italian parliamentary debates while ensuring each political group is represented and every quoted sentence comes verbatim from official transcripts. It matters because it demonstrates how domain-specific retrieval design can match a general-purpose AI tool on overall satisfaction while improving source accuracy and political balance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"By-construction quote guarantee only ensures verbatim text, not correct attribution; LLM-chosen offsets can still misquote a real transcript.","rationale":"Good-faith reading: ParliamentRAG is a useful, reproducible system, and the by-construction mechanism does prevent the language model from inventing quotation wording. The reader's conditional acceptance is reasonable. However, the paper's central strongest claim, as quoted by the reader, is that the system 'guarantees by construction that every quotation in its summaries is a verbatim substring of an official transcript.' The mechanism does guarantee verbatimness, but the accompanying inference that this 'prevents hallucinations by construction' goes beyond verbatimness. Since the model generates the placeholder offsets, those pointers are as hallucinable as any other language-model output; the deterministic replacement step only guarantees that whatever offsets the model chose map to real transcript characters. The paper does not state that offsets are constrained to a candidate set from retrieval or that speaker identity is verified. The automated QF metric checks only substring matching, so it cannot detect this failure mode. This is not an accusation: the released code may contain such a constraint, and checking the repository is the appropriate way to settle it. But as written, Section 4.3's guarantee is not established. This concern is different from the reader's weakest assumption about unvalidated authority weights; both support a conditional verdict, but the condition should include offset-to-speaker verification in addition to an authority ablation. Therefore the verdict remains conditional, and the reader's overall judgment is unchanged.","tokens_in":13086,"tokens_out":7642,"duration_ms":73650,"concrete_test":"Instrument the Cite stage to log, for every placeholder, the emitted character offsets, the resolved source chunk, and that chunk's speaker. Re-run the 15 benchmark queries and automatically verify: (1) each resolved quote's chunk speaker matches the MP named in the generated summary, and (2) the chunk was retrieved or selected for that group's position brief. Report any mismatch. Additionally, inspect the open-source code for the placeholder-resolution function: if it validates offsets against a candidate list of chunks with speaker metadata, the concern reduces to a documentation issue; if no such validation exists, the by-construction claim is overstated as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 claims that the Cite stage 'prevents hallucinations by construction' because the language model only inserts placeholders with character offsets that are later replaced with text from the original speeches. This conflates two distinct properties: lexical fidelity (the emitted string is a verbatim substring of some transcript) and attribution correctness (the substring belongs to the speaker and evidence that the surrounding text cites). The pipeline as described has the model generate the placeholder offsets, and nothing in the description constrains those offsets to a verified candidate set. The model could therefore emit an offset for any span of any transcript, and the deterministic replacement would still satisfy the automated Quotation Faithfulness metric, defined in Section 5.1 as the 'fraction of quotations that exactly match substrings of the original parliamentary interventions.' Consequently, QF = 1.00 is compatible with systematic misattribution: a real quote from MP A inserted under MP B's name, or a quote with no evidential connection to the claim it accompanies. This is precisely the 'citation misattribution' risk the introduction says the system addresses. The by-construction guarantee is real but narrower than stated: it rules out fabricated wording, not fabricated pointers, and it does not by itself ensure that 'all claims are supported by verifiable evidence.'","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ParliamentRAG is a retrieval-augmented generation system for Italian Chamber of Deputies proceedings that combines a parliamentary knowledge graph, a query-dependent authority model, and a multi-view generation pipeline. Given a user query, it retrieves speech chunks and legislative acts, reranks evidence using relevance, diversity, group coverage, authority, and salience, selects one expert per parliamentary group, and generates a structured summary in which quotations are introduced as placeholders with character offsets and later replaced by verbatim transcript text. The authors claim this design guarantees quotation faithfulness by construction, achieves near-perfect group coverage, and matches Google NotebookLM on overall satisfaction while being preferred on source-related dimensions. The evaluation uses 15 policy topics, automated metrics, and a blind A/B study with six domain experts, concluding that authority-aware multi-view RAG is a promising approach for faithful parliamentary synthesis.","tokens_in":13322,"tokens_out":3292,"duration_ms":37674,"significance":"If the central claims hold, the paper makes a useful applied contribution to parliamentary NLP and RAG: the offset-based quotation grounding is a genuinely different mechanism from prompt-based citation, the authority model is interpretable and query-dependent, and the system is released with source code and a live deployment. The authors also deserve credit for a blind expert evaluation against a strong commercial baseline and for explicitly acknowledging the absence of internal ablations and the small benchmark size. However, the significance is currently bounded by three issues: the by-construction quotation guarantee is narrower than stated because it does not constrain attribution correctness; the mean-authority metric is computed with the same model used for ranking; and the absence of ablations leaves the authority component's contribution unquantified. These concerns do not invalidate the system, but they weaken the paper's strongest claims as they stand.","major_comments":[{"comment":"The claim that the Cite stage 'prevents hallucinations by construction' conflates lexical fidelity with attribution correctness. The pipeline only guarantees that the emitted string is a verbatim substring of some transcript, because the language model inserts character offsets that are later replaced by source text. Nothing in the description constrains those offsets to a verified candidate set or to the speaker/evidence associated with the claim being supported. A model could select an offset that points to a genuine quote by a different speaker or to an unrelated passage, and the deterministic replacement would still satisfy the Quotation Faithfulness metric defined in Section 5.1 as the fraction of quotations that exactly match substrings of the original interventions. Thus QF = 1.00 is compatible with systematic misattribution, which is precisely the risk the introduction says the system addresses. The by-construction guarantee should be restated as 'no fabricated wording' and supplemented with an attribution evaluation that checks whether each quoted span belongs to the speaker and evidence cited in the summary.","section":"Section 4.3"},{"comment":"The Mean Authority (MA) comparison is partly self-confirming. MA is computed with the same query-dependent authority model that is used in Equation (1) to rerank evidence, and for NotebookLM the context is curated by ParliamentRAG's retrieval pipeline, which includes the authority score. The difference of 0.53 vs. 0.52 therefore cannot be interpreted as evidence that ParliamentRAG selects more authoritative speakers in any model-independent sense. The authors acknowledge this circularity for NotebookLM, but the same issue affects the ParliamentRAG numbers: MA is an internal design metric, not an external validation of expert quality. The paper should either drop MA from the comparative claims or validate the authority model against an independent signal, for example committee membership in the relevant policy area or expert ratings of cited speakers.","section":"Section 5.1, Table 1"},{"comment":"The authority component, which the abstract names as the core contribution, is not isolated by any ablation. The weights w_profession = 0.15, w_education = 0.10, w_committee = 0.25, w_legislative_acts = 0.20, w_speech_interventions = 0.25, and w_institutional_role = 0.05 are set empirically and not learned, and the comparison against NotebookLM varies retrieval, reranking, generation, and backbone LLM simultaneously. Consequently the observed outcome differences cannot be attributed to the authority model. The Limitations section states that internal ablations are future work, but a central architectural contribution should be supported by at least an ablation that varies w_a or replaces authority with a content-based baseline. Without that, the claim that authority awareness improves multi-view synthesis is not demonstrated.","section":"Section 4.2, Equation (2)"},{"comment":"The automated metrics in Table 1 are reported as single means over 15 topics without variance, confidence intervals, or tests, so observed differences such as GQ 0.97 vs. 0.95 and QF 1.00 vs. 0.95 may not be statistically meaningful. The human evaluation in Table 2 is more transparent, but the paper acknowledges that none of the differences survive Holm–Bonferroni correction. The abstract and conclusions state that ParliamentRAG 'achieves higher coverage' and 'is consistently preferred on source-related dimensions'; these statements should be qualified as descriptive patterns with small effect sizes rather than significant advantages. The authors should report distributions or per-topic values for the automatic metrics and clarify which claims are inferential versus exploratory.","section":"Section 5.2, Tables 1 and 2"}],"minor_comments":[{"comment":"There is a typo in the sentence about the authority score: 'reranked considering the authority socre' should read 'authority score'.","section":"Section 5.2"},{"comment":"The name 'ParliamentaryRAG' is used in the quotation-faithfulness paragraph, while the system is called 'ParliamentRAG' elsewhere; please standardize.","section":"Section 2, related work"},{"comment":"The example output is shown in English translation; the paper should clarify whether the live system and the evaluated outputs were in Italian and whether the translation was produced by the system or by the authors.","section":"Figure 1"},{"comment":"The curated keyword-based mapping from queries to parliamentary committees is a free parameter that could bias retrieval; the mapping should be described in more detail or released in full so readers can assess coverage and potential biases.","section":"Section 4.1"},{"comment":"The human evaluation reports N = 67 paired evaluations across six experts with unequal topic coverage, but no inter-annotator agreement or per-evaluator breakdown is provided; this would help assess whether the preference patterns are driven by one or two experts.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a useful deployed system and has a clear, honest limitations section, but the current evaluation does not yet substantiate the strongest claims in the abstract. The most important issue is the overstatement of the by-construction quotation guarantee: as written, Section 4.3 leaves open a systematic misattribution failure mode that the QF metric cannot detect. I would recommend asking the authors to either constrain the placeholder offsets to a verified candidate set (e.g., chunks that were retrieved and attributed to a specific speaker) or to reframe the guarantee and add an attribution-focused evaluation. The lack of ablations is also a concern for an ISWC in-use track, where the practical contribution could be validated with a small controlled study. I do not see grounds for rejection, but revision is needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper deserves a serious referee. The contribution is integration—query-dependent authority scoring over a parliamentary KG, per-group stratified retrieval and generation, and offset-based verbatim quotation—and the authors are honest that the ingredients exist separately in prior work. What earns credit: the architecture enforces per-group coverage by construction, the quotation grounding mechanism is real, and the comparison with NotebookLM is conservative in the right direction, giving the baseline a curated context and a stronger backbone LLM while ParliamentRAG searches the full corpus.\n\nSoft spots, in order of importance:\n\n1. The quotation guarantee is narrower than claimed. Section 4.3 says hallucinations are prevented \"by construction\" because the language model inserts character offsets rather than generating text. That ensures lexical fidelity—every emitted quote is a substring of some transcript. It does not ensure attribution correctness. Nothing in the description forces the model's offsets to point at the evidence chunks that were retrieved, or at the speaker being cited. QF=1.00 is therefore compatible with a real quote from MP A inserted under MP B's name. That is exactly the misattribution risk the introduction says the system addresses. The stress-test note gets this right. It is fixable: validate offsets against the retrieved evidence and the cited speaker before replacement, and report an attribution metric.\n\n2. The authority model is not tested on its own. There is no ablation that removes authority from the reranking or expert selection. Mean Authority is computed with the same model that selected the speakers, so the +0.01 gap is partly self-confirming. The hand-set weights are disclosed and interpretable, which is good, but the central architectural claim remains plausible rather than demonstrated.\n\n3. Human preferences are directionally consistent but do not survive multiple-comparison correction. To the authors' credit, they say this explicitly. Still, the phrase \"stronger expert preferences\" should be read as weak evidence, not a headline result.\n\n4. The NotebookLM comparison is apples-to-oranges in the other direction: its context was assembled using ParliamentRAG's own retrieval and reranking, so NotebookLM's end-to-end retrieval ability is never tested. The authors acknowledge this, but it means the comparison is really about generation under curated input, not system-versus-system.\n\nMinor items: 15 topics, six experts, no error bars on the automated metrics. For an in-use track paper, these are acceptable limits.\n\nBottom line: the system is real, open source, and aimed at a genuine civic information need. The by-construction claim needs tightening and the authority component needs an ablation, but neither issue sinks the paper. I would send it to peer review and expect conditional acceptance: the architecture and evaluation are solid enough, with the authority contribution as the main thing to pressure.","headline":"A real, open-source applied RAG system with a genuine civic-information use case, but its by-construction quotation guarantee is narrower than advertised and the authority model is plausible rather than proven.","tokens_in":13821,"tokens_out":2043,"would_cite":true,"duration_ms":22577,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval-augmented generation system for Italian parliamentary proceedings makes quotation faithfulness a design guarantee: every quote is a verbatim substring of an official transcript.","keywords":["Retrieval-Augmented Generation","Parliamentary NLP","Expert Finding","Knowledge Graphs","Quotation Faithfulness","Multi-Perspective Summarization","Authority-aware ranking","Italian Parliament"],"falsifier":"An ablation of the ranking equation with the authority term removed ($w_a=0$) that shows no drop in group coverage, quotation faithfulness, or expert preference on source-related dimensions would falsify the paper's claim that authority-aware reranking contributes to the system's performance; equally, a benchmark topic on which the authority model ranks a speaker with no legislative record on that topic above all co-signatories would undermine the expertise signal.","tokens_in":12844,"feed_emoji":"🏛️","tokens_out":6461,"duration_ms":56963,"temperature":0.7,"pith_summary":"ParliamentRAG is a retrieval-augmented generation system for the Italian Chamber of Deputies that aims to produce balanced, traceable summaries of parliamentary positions. The paper's central claim is that by making speaker authority query-dependent and by replacing model-generated quotations with placeholder-based verbatim extractions from official transcripts, a RAG system can guarantee that every quotation is faithful to the source while still matching a frontier commercial baseline on overall quality. The system retrieves evidence through a dual channel of dense chunk search and legislative-act graph traversal, reranks by a composite score that includes authority, and generates one section per parliamentary group. Evaluated on 15 policy topics against NotebookLM, ParliamentRAG achieves 0.97 group coverage versus 0.95, 1.00 quotation faithfulness versus 0.95, and is preferred on source-related dimensions but not on prose quality.","feed_headline":"RAG for parliament: every quote is verbatim","feed_subtitle":"Authority-aware retrieval balances all ten political groups while quotations come straight from official transcripts.","key_machinery":"The central machinery is the authority score in Equation 2, a weighted sum of semantic-similarity terms between the query and embeddings of speaker attributes (profession, education, committees, roles) and time-decayed counts of legislative acts and speech interventions, with hand-set weights such as $w_{\\text{committee}}=0.25$ and $w_{\\text{speech interventions}}=0.25$. The other load-bearing mechanism is the Cite stage of the generation pipeline: the language model emits placeholders with character offsets into the original transcript, and a deterministic resolver replaces them with the exact source text, so no generated quotation can differ from the record. Together these mechanisms implement the paper's three principles of multi-view representation, authority awareness, and quotation traceability.","core_discovery":"The paper establishes that the three risks of applying RAG to parliamentary transcripts—speaker dominance, lack of topical expertise weighting, and citation misattribution—can be addressed jointly by architectural design rather than by prompting alone. Its core discovery is a pipeline in which a query-dependent authority score, computed from interpretable weighted components (profession, education, committee membership, legislative acts, speech interventions, institutional role) with temporal decay and coalition-aware invalidation, reranks retrieved evidence and selects one expert per parliamentary group. In the generation stage, the language model never writes quotation text; it inserts character-offset placeholders that a deterministic resolver replaces with verbatim substrings of the original transcripts, making quotation faithfulness 1.00 by construction rather than by measurement. The authors claim this is the first system to integrate structured parliamentary graphs with query-dependent authority modeling and multi-view generation in a unified RAG architecture.","pith_inferences":["The offset-based placeholder mechanism is domain-agnostic; the same Cite stage could be transplanted to any source-grounded generation task where verbatim fidelity matters, such as legal opinion summaries, clinical notes, or financial disclosures, provided the source text is stored with character offsets.","The absence of ablations means the authority model's actual contribution is untested; a natural test is to compare ParliamentRAG against itself with authority reranking disabled, and against a version with static query-independent authority, on the same benchmark.","Because the evaluation used a curated, pre-retrieved context for the commercial baseline, the comparison does not test end-to-end retrieval; a full-corpus baseline would likely widen the gap on source coverage, but this remains to be demonstrated.","The paper implicitly assumes that one expert per parliamentary group is the right granularity; an extension could weight groups by seat share or generate a dissenting-minority view for intra-group factions, which would change the balance semantics."],"forward_implications":["Quotation faithfulness in RAG becomes a hard guarantee rather than a measured property, making the system suitable for journalistic and institutional verification workflows.","Because generation is stratified per parliamentary group, the retrieval stage becomes the sole bottleneck for coverage; improving retrieval recall should directly raise the 0.97 group-coverage figure toward 1.0.","The authority weights and components are explicitly interpretable, so the model can be audited or reweighted for other legislatures, languages, or personalization without retraining.","The comparison suggests that prompt-level instructions alone cannot enforce verbatim quotation or systematic group coverage, implying that civic-information tools should adopt structural constraints rather than relying on stronger base models.","Mean authority of cited speakers is only slightly higher than the baseline (0.53 vs 0.52), a direct corollary of the deliberate low weight (0.05) placed on authority in the reranking equation."],"supporting_citations":[{"why":"Supplies the RDF open data from the Chamber of Deputies that the knowledge graph is built from.","marker":"[6]"},{"why":"Formalizes profile-based versus document-based expert finding, which the query-dependent authority model extends.","marker":"[3]"},{"why":"Establishes comparable parliamentary corpora as a standard resource and domain context for the work.","marker":"[10]"},{"why":"Provides the Italian parliamentary corpus that situates the system's domain and evaluation.","marker":"[11]"},{"why":"Supplies the diversity-based reranking concept that the coverage and diversity terms in Equation 1 draw on.","marker":"[7]"},{"why":"Serves as the post-hoc attribution approach that ParliamentRAG contrasts with its by-construction quotation grounding.","marker":"[12]"},{"why":"Supplies the blind A/B preference evaluation protocol used in the human study.","marker":"[8]"},{"why":"Motivates the over-representation risk that the system addresses by balancing across speakers and groups.","marker":"[24]"},{"why":"Provides the PARADISE evaluation framework that structures the human assessment protocol.","marker":"[26]"}],"fun_headline_variants":["RAG that quotes parliament verbatim","Authority-aware RAG balances party views","Verbatim quotes in parliament RAG by design","First RAG to weight speakers by expertise","ParliamentRAG: perfect quote faithfulness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hand-set authority weights are assumed to correctly identify topical experts; no ablation shows that the authority term improves results over content-based reranking alone.","fun_headline_variants_meta":{"raw":{"variants":["RAG that quotes parliament verbatim","Authority-aware RAG balances party views","Verbatim quotes in parliament RAG by design","First RAG to weight speakers by expertise","ParliamentRAG: perfect quote faithfulness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1418,"prompt_tokens":946,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":406}},"tokens_in":562,"tokens_out":472,"duration_ms":4954,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:34:28.235751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An ablation of the ranking equation with the authority term removed ($w_a=0$) that shows no drop in group coverage, quotation faithfulness, or expert preference on source-related dimensions would falsify the paper's claim that authority-aware reranking contributes to the system's performance; equally, a benchmark topic on which the authority model ranks a speaker with no legislative record on that topic above all co-signatories would undermine the expertise signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RDF open data from the Chamber of Deputies that the knowledge graph is built from."},{"cited_title":"Foundations and Trends in Information Retrieval6, 127–256 (01 2012).https://doi.org/10","cited_arxiv_id":null,"evidence_quote":"Formalizes profile-based versus document-based expert finding, which the query-dependent authority model extends."},{"cited_title":"In: Calzolari, N., Kan, M.Y., Hoste, V., Lenci, A., Sakti, S., Xue, N","cited_arxiv_id":null,"evidence_quote":"Provides the Italian parliamentary corpus that situates the system's domain and evaluation."},{"cited_title":"In: Proceedings of the 41st International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the blind A/B preference evaluation protocol used in the human study."},{"cited_title":"ICML’23, JMLR.org (2023)","cited_arxiv_id":null,"evidence_quote":"Motivates the over-representation risk that the system addresses by balancing across speakers and groups."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PARADISE evaluation framework that structures the human assessment protocol."}],"review_version":1}