{"id":"a7264a8a-6bfb-40ed-baf8-f753c2b31465","arxiv_id":"2511.22858","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A norm-compliant RAG for Japanese medical litigation must restrict knowledge sources, stay faithful to retrieved context, and reference sources valid at the time of the disputed issue.","lead":"This paper proposes three requirements for a retrieval-augmented language model (RAG) system that helps with Japanese medical lawsuits: use only court-approved knowledge sources, keep answers faithful to retrieved documents, and cite sources valid at the time relevant to the dispute. It is a design position paper, not an implemented system, but worth reading as a normative checklist for legal AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Requirement (3) is under-specified: 'validity period' of medical knowledge is assumed machine-determinable from timestamps and citation networks, which is not established and is central to the three-axis framework.","rationale":"The paper's central claim is a conditional normative statement: if a RAG is to support Japanese litigation in a norm-compliant way, it must satisfy three requirements. The derivation from adversarial procedure, knowledge updating, and issue-specific reference time is internally coherent. The paper also honestly identifies implementation as future work. However, the claim is only meaningful if each requirement is operationalizable. Requirement (1) (source control) and requirement (2) (faithfulness/attribution) have at least plausible technical proxies (filtering, faithfulness scores, DA). Requirement (3) (temporal validity) lacks even a definition of 'validity period.' The paper suggests timestamps and citation networks, but this conflates publication date with legal validity. The standard of care at a past time is a socially constructed legal determination, not a property of a document. The paper itself notes that conflicting knowledge streams coexist during transitions, undermining a simple timestamp-based approach. Therefore the most load-bearing concern is that requirement (3) is not implementable as stated, which means the three-axis evaluation framework cannot be applied. A concrete test would be to build a small expert-annotated benchmark of past standards of care and see whether a timestamp/citation algorithm can recover them. This test would settle whether the concern lands. Since the reader already marked CONDITIONAL based on related assumptions, our verdict remains UNCHANGED. We partially agree with the reader's weakest_assumption: we sharpen it from corpus availability to the more fundamental non-machine-determinability of validity periods.","tokens_in":6388,"tokens_out":7640,"duration_ms":74934,"concrete_test":"Select 5–10 Japanese medical malpractice cases where the standard of care at the time of the alleged negligence was disputed. Have a panel of medical-legal experts independently annotate, for each case and each candidate source (guideline version, textbook edition, peer-reviewed article), whether that source was part of the applicable standard at the relevant time. Then implement a timestamp-plus-citation-network algorithm as sketched in §4.3 and measure agreement with the expert panel (e.g., Cohen's κ). If agreement is not high (κ < 0.7), the operationalization fails and requirement (3) cannot ground an evaluation metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central normative derivation is plausible, but the paper's third requirement (Section 4.3) is load-bearing and under-specified. The claim that a legal RAG must reference sources with 'appropriate timestamps' presupposes that the applicability of medical/legal knowledge at a given past time can be read off publication dates and citation networks. This is not established and is unlikely to hold: the 'standard of care' at time T is a legal construct determined by courts through evaluation of guidelines, textbooks, practice, and expert testimony. A document's publication date is neither necessary nor sufficient for its legal validity at T—a source may be published but not yet community-accepted, or may remain authoritative after being superseded by later literature. The paper itself acknowledges that 'during the transitional period, multiple streams of expert knowledge that contradict each other may coexist' (Section 2.3), yet proposes no method to represent or resolve this coexistence within a timestamp-based retrieval system. Because requirement (3) is one of the three axes by which any norm-compliant RAG is to be judged, failure to operationalize it leaves the entire evaluation framework untestable. The paper's own statement that implementation is future work does not mitigate this: the requirements are the contribution, and a requirement that cannot be operationalized is not a requirement but a desideratum.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a set of three requirements for a Retrieval-Augmented Generation (RAG) system intended to support Japanese medical litigation procedures while complying with legal norms. Drawing on the adversarial principle, the prohibition on judges' use of privately acquired expert knowledge, and the need for issue-specific reference times, the authors argue that a norm-compliant legal RAG must (1) restrict retrieval to sources that have been critically validated by the relevant expert community and made equally available to both parties, (2) generate responses that are attributable to and faithful to the retrieved context, and (3) use sources with timestamps appropriate to the legal question at issue. The paper presents these three aspects as the core contribution, with implementation and evaluation left to future work.","tokens_in":6695,"tokens_out":5722,"duration_ms":51646,"significance":"If validated, the proposed framework would move evaluation of legal RAG systems beyond raw accuracy and factuality, directing attention to procedural compliance and source temporal validity. The derivation is grounded in Japanese legal scholarship (e.g., [5], [6], [7]) and is internally coherent: the three requirements do follow from the adversarial principle and the issue-specific reference-time doctrine. The paper is honest about its scope, explicitly deferring methods and experiments to future work, and it does not overclaim empirical results. Its main value is as a conceptual standpoint for a domain where evaluation frameworks are still nascent.","major_comments":[{"comment":"Requirement (3) is under-specified in a way that is load-bearing for the entire framework. The paper asserts that source validity can be managed 'by utilizing timestamps and citation networks' (Sec. 4.3), but it does not explain how legal validity at a given time is to be determined. The standard of care at time T is a legal construct, not a document-level metadata property: a source may be published but not yet accepted, or superseded but still authoritative. The paper itself acknowledges that contradictory expert knowledge can coexist during a transitional period (Sec. 2.3), yet offers no method to represent, retrieve, or reason about such coexisting sources. Without a concrete model of temporal validity and a procedure for deriving the reference time from the legal issue, requirement (3) cannot be evaluated or implemented. The statement that implementation is future work does not miti","section":"Section 4.3 / Section 2.3"},{"comment":"The evaluation of requirement (1) is circular. The paper states that outputs derived from sources that deviate from 'the predefined scope' can be labeled inappropriate, but the 'predefined scope' is never defined. Section 2.1 gives a general principle—sources must be critically validated by the relevant expert community and equally available to both parties—but does not specify how this principle translates into a concrete corpus or retrieval filter. In particular, the paper does not address how to treat party-submitted evidence, which is a central element of the adversarial process. Without a definition of admissible source scope, requirement (1) cannot be tested, leaving the first axis of conformance ungrounded.","section":"Section 4.1"},{"comment":"Requirement (2), while plausible, is also under-specified for the legal setting. The paper mentions prompting and chain-of-verification as possible enforcement mechanisms, and Ragas as an evaluation metric, but it does not address situations where the retrieved context contains conflicting authorities—an acknowledged possibility in Sec. 2.3. A faithful response may nevertheless be legally non-compliant if it silently selects one line of authority at the expense of another, or if it omits a minority view that is relevant to the issue. The paper should clarify whether faithfulness is to be interpreted as claim-level entailment with respect to the entire retrieved context, and how conflicting sources should be handled within the framework.","section":"Section 4.2"}],"minor_comments":[{"comment":"The statistic 'approximately twenty percent of such data becomes outdated within five years' is attributed to a professor but has no citation. Since this is used as motivation, please provide a source or clearly label it as an informal estimate.","section":"Section 2.2"},{"comment":"The components 'Maintenance of timestamp' and 'Identifying reference time' are shown in the retrieval module but are not defined in the text. Please explain what these components do and how they relate to requirements (2) and (3).","section":"Figure 2"},{"comment":"The phrase 'chain-of-verification steps' is used without a citation. If this refers to a specific method (e.g., self-verification [11] or a similar approach), please add a reference.","section":"Section 4.2"},{"comment":"Reference [16] appears to list the same authors as [13] but with a different title ('Reducing Hallucination in Structured Outputs via Retrieval-Augmented Generation'). Please verify the title, venue, and authors; the citation may be incorrect.","section":"References"},{"comment":"The abstract states that the retrieval module must reference external knowledge with 'appropriate timestamps corresponding to the issues at hand,' while requirement (3) in Sec. 4.3 focuses on the 'valid time period' of sources. These are distinct concepts: one is about the issue's reference time, the other about a source's validity interval. Please clarify the relationship between them.","section":"Abstract and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a position paper, not a full system paper, and that is acceptable for its stated scope. The central contribution—deriving three norm-compliance requirements from Japanese civil procedure—is defensible and should be preserved. However, the operationalization of requirement (3) is not merely an implementation detail; it is conceptually unfinished. The paper should either provide a more precise definition of source validity (including how to handle transition periods and legal determination of the standard of care) or explicitly reframe the contribution as a set of high-level desiderata rather than requirements. I recommend major revision so the authors can strengthen this load-bearing part of the framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a workshop-grade position paper, and it's a decent one. The core move is reframing three familiar RAG concerns — source control, faithfulness/attribution, temporal validity — as consequences of Japan's adversarial civil procedure. That reframing is real and not something I've seen done before. The paper reads as careful: it grounds the source-control requirement in Japanese scholarship on judges' use of private knowledge, and the reference-time requirement in the distinction between the standard of care at the time of the alleged act versus knowledge at the close of oral proceedings.\n\nWhat it does well: the requirements follow from the stated norms without circularity; the paper is explicit that implementation and evaluation are future work; and it doesn't oversell the feasibility of replacing expert commissioners with LLMs. It even cites work showing RAG reduces but doesn't eliminate hallucinations. That's honest.\n\nSoft spots, in order of softness. First, requirement (3) is the load-bearing one and the least specified. The paper says valid time periods could be managed 'by utilizing timestamps and citation networks,' but the standard of care at time T is a legal construct, not something that falls out of publication dates. A source can be published but not yet accepted, or authoritative after being superseded. The paper itself notes that contradictory expert knowledge can coexist during transitions, but doesn't propose a representation for that coexistence. The stress-test note is right about this. I'd only push back on one implication: because the paper is a requirements proposal, not a claimed solution, a hard-to-operationalize axis is a weakness, not a fatal flaw. The requirements still have value as evaluation targets.\n\nSecond, the 'approximately twenty percent of data becomes outdated within five years' statistic in Section 2.2 comes from an acknowledged personal communication with Professor Yonemura but has no citation. For a paper that leans on it to motivate the need for retrieval, that's a thin reed.\n\nThird, the novelty claim — no prior research on legal-norm conformance in RAG — is asserted with 'to the best of our knowledge.' It's plausible, but the paper doesn't document a systematic search of Japanese-language literature. Given the authors cite Japanese legal scholarship, they know the terrain; it's a minor gap.\n\nOverall, the central argument holds at the level it's actually stated. The paper doesn't claim to have built the system. If the title is read as promising a system, that's misleading; but the body is clear that this is a design discussion.\n\nWho gets value: people building legal RAG evaluation benchmarks, especially in civil-law jurisdictions, and researchers thinking about how procedural norms should constrain retrieval. It deserves a serious referee for a workshop or a short paper venue; it would not carry a full journal on its own.\n\nMy recommendation: engage with it, but treat it as a requirements paper. If a referee report is requested, accept the assignment. The paper is honest, citeable, and raises a real question about whether RAG evaluation should be norm-grounded.","headline":"A clear, honest position paper that derives three RAG evaluation axes from Japanese civil-procedure norms; the derivation is the contribution and it holds, but the third axis is a desideratum rather than an implementable requirement.","tokens_in":7156,"tokens_out":2703,"would_cite":true,"duration_ms":25643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Legal RAG must obey three norms, not just answer accurately.","keywords":["Retrieval-Augmented Generation","Legal norms","Japanese civil procedure","Medical litigation","Faithfulness","Data attribution","Temporal validity","Expert knowledge"],"falsifier":"Find a real medical negligence case where the standard of care at the time of the alleged act is contested between two streams of expert knowledge and no document carries a verifiable validity timestamp; if such a case is normatively unresolvable by retrieval, the temporal requirement (3) lacks a principled implementation. Alternatively, run an experiment where a RAG system answers a negligence query using only documents from the wrong decade, and show that standard faithfulness/accuracy metrics still score it as correct—demonstrating that existing evaluation misses the paper's dimension.","tokens_in":6288,"feed_emoji":"⚖️","tokens_out":4325,"duration_ms":35863,"temperature":0.7,"pith_summary":"Working toward an AI that helps judges in Japanese medical lawsuits, this paper argues that a retrieval-augmented generation (RAG) system must satisfy three legal-norm requirements, not just answer accurately. First, the system may only draw on expert knowledge that is community-validated and equally accessible to both parties, respecting the civil-procedure prohibition on a judge's private knowledge. Second, every answer must be traceable to and faithful within the retrieved context; a correct-sounding answer drawn from the model's pretraining memory is norm-violating. Third, the system must use the knowledge valid at the legally relevant time—the standard of care when the alleged negligence happened versus the latest knowledge at the close of oral proceedings—managed through timestamps and citation networks. The paper proposes how each requirement could be evaluated, laying groundwork for a 'norm-compliant RAG' design.","feed_headline":"Legal RAG needs three norm checks, not just accuracy","feed_subtitle":"For court use in Japanese medical litigation: vetted sources, traceable answers, knowledge dated to the issue at hand.","key_machinery":"The carrying mechanism is a RAG pipeline equipped with three norm-enforcing components. A source-control layer restricts the document database to expert-community-validated sources and filters retrieval results so both parties have equal access. A faithfulness layer uses prompting constraints, chain-of-verification steps, and claim-level natural-language-inference scoring to force generated claims to originate in the retrieved context. A temporal layer attaches publication and validity timestamps to sources and navigates citation networks so that the system can select knowledge valid at the dispute-relevant time. Together these components turn ordinary retrieval-augmented generation into wha","core_discovery":"The central claim is that a legal RAG system supporting expert testimony in Japanese medical litigation can be designated norm-compliant only if it meets three linked conditions: (1) controlled knowledge sourcing—retrieval is restricted to documents that have passed critical verification by the relevant expert community and are equally available to both parties; (2) attribution and faithfulness—generated responses must derive from the retrieved context and be faithful to it, with any reliance on pre-training knowledge treated as inappropriate and any contradiction of the source treated as a failure; and (3) issue-specific reference time—the system must know when each source was published and","pith_inferences":["A natural testable extension: on a dataset of medical negligence questions, compare standard RAG faithfulness scores against human judgments about whether the answer would be admissible in court; the paper predicts mismatches whenever the answer is grounded in retrieved context that is temporally or procedurally inappropriate.","Implicit but not stated: if the 'equally available to both parties' requirement is taken seriously, the retrieval index must be a shared, inspectable artifact—this turns the RAG system from a judge's private assistant into an element of the adversarial procedure, with implications for transparency and audit.","The paper's own acknowledgment that hallucinations cannot be fully eliminated suggests the strongest version of the claim is about procedural legitimacy rather than total reliability; even partial substitution of expert commissioners may be acceptable only if a human expert verifies the final output."],"forward_implications":["If the three aspects are accepted, evaluations of legal RAG systems must add source-scope, attribution, and temporal-validity metrics alongside accuracy and factuality.","Responses that are factually correct but rely on the model's pretraining knowledge would be ruled norm-violating, changing the error taxonomy of legal AI.","The same norm-compliance lens applies to other expert-commissioner domains such as construction, intellectual property, and system-development litigation.","The timestamp requirement implies that RAG development for law requires building and maintaining document corpora with validated publication and validity dates, not just better retrieval ranking."],"fun_headline_variants":["Legal RAG: three tests for court-ready answers","RAG for litigation: vetted sources, faithful text, dated facts","Legal RAG must pass three checks: sourcing, faithfulness, timing","Expert RAG for court: no private knowledge, only dated sources","RAG in Japanese litigation: three binding constraints"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the legally relevant state of medical and legal knowledge at any given moment can be captured as retrievable documents with accurate publication and validity timestamps—so that which knowledge was 'valid at the time of the act' can be decided by a machine; the paper offers no evidence such timestamped, community-validated corpora exist or that validity periods can be automatically determined.","fun_headline_variants_meta":{"raw":{"variants":["Legal RAG: three tests for court-ready answers","RAG for litigation: vetted sources, faithful text, dated facts","Legal RAG must pass three checks: sourcing, faithfulness, timing","Expert RAG for court: no private knowledge, only dated sources","RAG in Japanese litigation: three binding constraints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2496,"prompt_tokens":681,"completion_tokens":1815,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":1730}},"tokens_in":425,"tokens_out":1815,"duration_ms":10914,"temperature":1.0,"reasoning_tokens":1730,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:38:07.826794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a real medical negligence case where the standard of care at the time of the alleged act is contested between two streams of expert knowledge and no document carries a verifiable validity timestamp; if such a case is normatively unresolvable by retrieval, the temporal requirement (3) lacks a principled implementation. Alternatively, run an experiment where a RAG system answers a negligence query using only documents from the wrong decade, and show that standard faithfulness/accuracy metrics still score it as correct—demonstrating that existing evaluation misses the paper's dimension.","supporting_citations":[],"review_version":1}