{"id":"910dfd4f-45cd-4b45-a8dd-a5d8c555f106","arxiv_id":"2508.00581","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-stage LLM framework using atomic assertions and clustered causal networks generates pre-consultation questionnaires from EMRs, reporting 84.2% personal key-fact coverage versus 42.1% for direct LLM prompting.","lead":"The paper describes a three-stage LLM pipeline that turns electronic medical records into pre-consultation questionnaires by first extracting atomic facts, then building causal networks, then clustering them by disease to guide question generation. It reports higher key-fact coverage than direct LLM prompting and comparable quality to manual expert questionnaires in about a third of the time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Key-fact coverage is a prompt-compliance score: Stage 1 defines the answer key and Stage 3 is required to cover it, so the headline comparison conflates completeness with instruction following.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the evaluation treats GPT-4o's Stage 1 atomic assertions as the ground-truth key facts, while the Stage 3 prompt is required to cover all of those assertions. I agree this is the central weakness. The coverage metric is circular in a precise, demonstrable way: the answer key is produced by the same model that is given the key as input, and the generation prompt instructs the model to reproduce it. Therefore the 84.2% versus 42.1% result largely reflects prompt compliance and an asymmetric division of labor, not clinically verified completeness. The paper deserves credit for a clearly described three-stage pipeline and for evaluating on a real EMR dataset, but the reported evidence does not establish the abstract's claim of superior information coverage. The expert-rated relevance and understandability scores are also presented without statistical support, so they do not independently rescue the claim. Because the reader already reached REJECT for essentially these reasons, my stress-test does not change the verdict. A clinician-defined gold-standard reannotation is the concrete check that would settle whether the concern actually lands; if the coverage advantage survives that test, the central claim would be substantially strengthened.","tokens_in":14136,"tokens_out":2426,"duration_ms":26088,"concrete_test":"Select 50 EMRs from the dataset; have two clinicians who did not design the framework independently write the exhaustive set of key facts for each EMR (and, for 5 diseases, for the corpora). Without changing either system, recompute C_personal and C_disease against this clinician-defined gold set instead of Stage 1 atomic assertions, and report inter-rater agreement (e.g., Cohen's kappa). If the proposed method's coverage advantage over direct LLM shrinks substantially or reverses on clinician-defined facts, the reported coverage superiority is an artifact of a self-generated answer key; if it persists with acceptable inter-rater agreement, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the framework is superior in information coverage rests on C_personal (84.2% vs 42.1%) and C_disease (92.2% vs 90.8%). The key-fact coverage metric is defined in Section 5.2 as the proportion of key facts extracted from the EMR or EMR corpus that appear in the questionnaire. In the implementation, those key facts are exactly the atomic assertions produced by GPT-4o in Stage 1, and the Stage 3 prompt in Figure 5 explicitly requires that the questionnaire cover all atomic assertions, with a termination condition confirming that all atomic assertions appear. Thus the numerator and denominator are generated by the same model, and the denominator is handed to the generator as input. C_personal therefore measures whether the generator echoes its input, not whether clinically complete information is captured. There is no independent verification, by clinicians or otherwise, that Stage 1 extraction is complete or correct; the paper's own limitation section acknowledges dependence on LLM capabilities. Against the direct-LLM baseline, the comparison is also asymmetric: the baseline must discover facts from raw text, while the proposed method is given the extracted fact list. This does not refute the architecture, but it means the reported coverage advantage cannot currently be attributed to clinical completeness. The expert-rated metrics (R and U) are reported as point means from 5 experts without variance, significance testing, or inter-rater reliability, so they cannot carry the claim on their own. The load-bearing assumption is that atomic assertions equal ground-truth key facts, and the paper provides no test of that assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage LLM-driven framework for generating pre-consultation questionnaires from electronic medical records: Stage 1 extracts atomic assertions from a single EMR, Stage 2 constructs personal causal networks and clusters them to synthesize disease-level knowledge, and Stage 3 generates personal and disease-specific questionnaires from these structured representations. The authors evaluate on 3,000 real-world EMRs, comparing against direct LLM prompting (for personal questionnaires) and manual expert generation (for disease-specific questionnaires). Reported metrics are key-fact coverage, diagnostic relevance, understandability, and generation time. The headline claim is that the framework achieves superior performance on all four metrics. The core finding, however, is that the key-fact coverage metric is defined so that the Stage 3 prompt explicitly requires covering the same atomic assertions that define the metric, making the reported coverage advantage a prompt-compliance score rather than an independent measure of clinical completeness. Expert-rated relevance and understandability are reported as point means without statistical support, and the generation-time comparison is incomplete. The underlying architecture is interesting, but the current evidence does not substantiate the stated claims.","tokens_in":14304,"tokens_out":5124,"duration_ms":50338,"significance":"The problem of automating pre-consultation questionnaire generation from complex EMRs is timely and practically important. The proposed structured multi-stage design, with atomic assertions and causal networks as intermediate representations, is a reasonable and potentially valuable contribution to clinical NLP. The collected dataset of 3,000 EMRs, if made available, would be a useful resource. However, the evaluation design is the key weakness. Because the coverage metric is defined by the same model that generates both the answer key and the questionnaire, the headline coverage results (C_personal 84.2% vs 42.1%, C_disease 92.2% vs 90.8%) do not establish clinical superiority. The expert-rated metrics lack statistical rigor, and the comparisons are asymmetric. If the authors rework the evaluation with an independent gold standard for key facts, statistical testing, and fair baselines, the framework could make a meaningful contribution. As presented, the central claims are not supported.","major_comments":[{"comment":"The key-fact coverage metric is circular. C_personal is defined as the proportion of key facts from the EMR that appear in the questionnaire, and in the implementation those key facts are exactly the atomic assertions extracted by GPT-4o in Stage 1. The Stage 3 prompt (Figure 5) explicitly requires that the questionnaire cover all atomic assertions, with a termination condition to enforce this. Therefore the reported 84.2% vs 42.1% gap measures whether the generator echoes the input fact list, not whether the extracted list is clinically complete. There is no independent verification that Stage 1 extraction is complete or correct. The authors should have clinical experts (not the LLM) independently enumerate the key facts in each EMR and compute coverage against that gold standard, and should also report Stage 1 extraction precision/recall against the same gold standard.","section":"Section 5.2, Figure 5, Figure 3"},{"comment":"Diagnostic relevance (R) and understandability (U) are reported as point means from a panel of 5 clinical experts, with no confidence intervals, significance tests, or inter-rater reliability statistics. For instance, the claim that R=8.5 is 'markedly superior' to R=3.2 requires a hypothesis test with the actual rating distribution; with only 5 experts, the variance could be large. For disease-specific questionnaires, the paper asserts that R=9.2 and U=9.1 are only 'slightly lower' than the manual baseline (9.5, 9.5), but this is not tested statistically. Please report per-questionnaire scores, variance, and agreement metrics such as ICC or Cohen's kappa, and specify the number of questionnaires rated and the rating protocol.","section":"Sections 5.3 and 5.4"},{"comment":"The personal-questionnaire comparison is asymmetric. The direct LLM baseline receives only the raw EMR text, whereas the proposed framework receives the extracted atomic assertions as input at Stage 3. Thus the coverage advantage may be due to information being handed to the generator, not to the multi-stage architecture. To isolate the contribution of the causal network and clustering, include a baseline where GPT-4o is given the same atomic assertions but no causal network, and another where it is given the raw text plus an unstructured list of facts. Without such controls, the coverage comparison conflates the value of the structured representation with the value of simply providing the extracted facts.","section":"Section 5.3"},{"comment":"The abstract claims superior generation time, but the paper only reports a generation-time comparison for the disease-specific task (10.4 min vs 33.8 min for manual generation). No generation-time data are given for the personal questionnaire task (our method vs direct LLM). Since the abstract lists generation time as one of the four dimensions of superiority, the omission is load-bearing. Please report wall-clock times for all methods and both tasks, including variance across runs.","section":"Abstract, Sections 5.3 and 5.4"},{"comment":"The clustering procedure relies on 'cutting the dendrogram at a fixed cut-off' to obtain the final clusters, but the paper neither reports the cut-off value nor provides a sensitivity analysis. The number of clusters directly determines the set of representative causal networks used for disease-specific questionnaires, so the choice is a free parameter that affects the main results. Please state the value used and test whether the disease-specific R, U, and C_disease are stable to reasonable variations in the cut-off.","section":"Section 4, Equation (3)"}],"minor_comments":[{"comment":"The heading of Section 4's second stage is misspelled as 'Satge 2' and should read 'Stage 2'.","section":"Section 4 heading"},{"comment":"Please add error bars or confidence intervals to the bar charts and state the number of questionnaires and raters involved in each comparison.","section":"Figures 9 and 10"},{"comment":"The definitions of Relevance to Diagnosis (R) and Understandability (U) would benefit from a concrete rubric or evaluation form, ideally provided in an appendix, so that the expert ratings are reproducible.","section":"Section 5.2"},{"comment":"The reference 'Winston, C., Winston, C., Winston, C., & Winston, C. (2024)' appears to have an erroneous repeated author list; please check the correct citation. This work is also directly relevant to the topic, and the paper's novelty claim should be discussed in light of it.","section":"References"},{"comment":"The literature review on pre-consultation questionnaire generation with LLMs is thin; the 'Winston et al.' work and any other LLM-in-context-learning-based questionnaire generation papers are not discussed, which weakens the novelty argument.","section":"Section 2"},{"comment":"The graph similarity as the average cosine similarity over all edge pairs can be dominated by nodes with many edges; consider a matching-based or degree-normalized similarity measure, or at least report the distribution of edge counts.","section":"Equation (2)"},{"comment":"The demonstration example states that the sample EMR contains '38 key clinical facts' and that the proposed method covers 32 of them, but does not specify who identified the 38 facts or how they were counted; please clarify the source of this reference count.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies on a private dataset that is not released, and the evaluation design will need significant rework to support the claims. The authors appear to be from a university and an industry health-technology company; no conflicts are declared, but the industry connection may be relevant to the editor's assessment of the dataset provenance. The main issue is the circular coverage metric, which is a load-bearing flaw; I would encourage the editor to require an independent clinical annotation of key facts and proper statistical analysis before considering the paper for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the architecture is worth reading, but the headline evaluation does not measure what it claims. If you pick this up, read it as a system paper with a pipeline worth borrowing, not as evidence that the framework captures more clinical facts.\n\nWhat is actually new: a three-stage decomposition—atomic assertion extraction, per-patient causal network construction, and clustering representative networks into disease knowledge—is a sensible assembly for EMR-to-questionnaire generation, and I do not know prior work doing exactly this. The 3,000-EMR real hospital dataset is a real asset, and the RQ1 worked example shows the pipeline end to end. The idea of using network clustering to synthesize disease-level questionnaires is reasonable and worth trying on other tasks.\n\nThe soft spot is the one the stress-test identified, and I think it lands. C_personal and C_disease are defined by the key facts from Stage 1, and the Stage 3 prompt explicitly requires covering every atomic assertion. So coverage measures prompt compliance, not clinical completeness. The 84.2% versus 42.1% gap is mostly the difference between handing the model the fact list and asking it to discover facts from raw text. That is not nothing—a cheap pre-extraction step may help—but it does not support the abstract's \"superior information coverage\" claim. The paper never checks whether Stage 1 extraction is complete or correct; the limitation section admits LLM dependence but does not flag the circularity. Expert R and U are given as point means with no variance, significance tests, or inter-rater reliability, so they cannot carry the claim alone. The disease-level comparison uses only the manual baseline, and the coverage edge (92.2 vs 90.8) is tiny.\n\nTwo smaller issues: the reference list has an entry with the same author listed four times (Winston et al. 2024), and there is no code or data release, which is a real barrier to reuse and verification.\n\nWho this is for: people building pre-consultation or EMR-summarization systems. The pipeline is a plausible template and the failure modes are instructive. It deserves a serious referee—the core idea is good enough—but the evaluation needs major revision before it supports the claims. As it stands, I would not cite it as evidence of clinical completeness.","headline":"Solid pipeline, circular evaluation: the coverage metric measures prompt compliance, not clinical completeness, so the paper needs major revision before its claims hold.","tokens_in":14928,"tokens_out":2379,"would_cite":false,"duration_ms":24656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage LLM pipeline, built from atomic assertions and causal networks, turns EMRs into pre-consultation questionnaires that capture more key facts than direct LLM prompting, with disease-level coverage close to manual expert writing.","keywords":["Large Language Model","Electronic Medical Record","Pre-consultation Questionnaire Generation","Atomic Assertion Extraction","Causal Network","Clinical Knowledge Representation","Medical Questionnaire","LLM multi-stage framework"],"falsifier":"Take 50 EMRs from the same hospital, ask two independent clinicians to list every key fact without seeing the model output, run Stage 1 on the same records, and recompute the questionnaire's coverage against the clinicians' list instead of the model's own assertions; if that independently measured coverage falls well below the reported 84.2% and 92.2%, the central completeness claim fails.","tokens_in":13868,"feed_emoji":"🩺","tokens_out":10550,"duration_ms":91509,"temperature":0.7,"pith_summary":"The paper tries to establish that the right way to turn an unstructured electronic medical record into a pre-consultation questionnaire is not to ask a large language model to do it in one pass, but to run it through three explicit stages: split the record into atomic assertions (single medical facts each with a relative time), wire those assertions into a personal causal network, and only then generate questions from that structure. The claimed payoff is that the questionnaire covers the facts that matter, follows a logical order, and can be tailored to one patient or generalized to a disease. On a real 3,000-record hospital corpus, the authors report that personal questionnaires cover 84.2% of key facts versus 42.1% for direct LLM prompting, and that disease-specific questionnaires approach manual expert coverage (92.2% versus 90.8%) while cutting generation time from about 34 minutes to about 10 minutes. The underlying bet is that explicit, intermediate clinical knowledge, not raw prompt engineering, is what direct LLM approaches lack for this task.","feed_headline":"LLM pipeline lifts questionnaire fact coverage from 42% to 84%","feed_subtitle":"Structured causal networks catch facts direct prompting misses, nearly matching expert-written questionnaires.","key_machinery":"The load-bearing object is the atomic assertion, a single indivisible medical fact paired with a relative time, produced by a prompted LLM. It is the unit that makes coverage measurable and causal structure possible. On top of it, the paper builds personal causal networks whose edges are cause–effect links between assertions; edge embeddings are computed by concatenating vectors from BCEmbedding, a bilingual embedding model, for the cause and effect assertions, network similarity is the average cosine similarity over all edge pairs, and hierarchical clustering with average linkage groups networks into typical disease pathways. Each cluster is represented by its most central network, and the cluster's size gives the pathway's weight. The final generation prompt is explicitly required to cover every atomic assertion, which is how the questionnaire inherits the completeness of the earlier stages.","core_discovery":"The central claim is that a questionnaire is only as good as the structure built between the raw record and the final text. The framework therefore decomposes the task into three stages: Stage 1 has an LLM extract atomic assertions, defined as the smallest indivisible medical fact with its relative time; Stage 2 builds a personal causal network for each patient, then clusters those networks per disease (grouped by ICD-10) into representative causal structures weighted by prevalence; Stage 3 prompts the LLM to turn the assertions and networks into personal and disease-specific questionnaires, with instructions to cover every assertion and to order questions along the causal structure. The authors argue that this structured pipeline solves the failure modes of direct generation—missed facts, broken logical order, and the inability to synthesize a whole corpus into a single disease-level questionnaire—and they support the argument with expert-scored comparisons showing large gains on personal coverage and relevance, and near-parity with manual experts on disease questionnaires at a fraction of the time.","pith_inferences":["An independent clinician re-extraction of key facts from a sample of the same EMRs would test whether the reported coverage reflects true clinical completeness or only how faithfully the LLM's own extracted assertions are restated as questions.","Because the clustering and generation stages are disease-agnostic, the pipeline should transfer to other specialties and languages wherever a bilingual embedding model and an ICD-10-style grouping are available; the paper only demonstrates it on the Shanghai hospital corpus.","The Stage 1 extractor is effectively a plug-in: swapping the chosen LLM for a cheaper or stronger model would change all downstream results, so the framework's gains are a property of the architecture and the model together, not of the architecture alone.","The slightly lower diagnostic relevance and understandability scores for disease questionnaires (9.2 and 9.1 versus 9.5 for manual generation) suggest that coverage and clinical sharpness can diverge when the model is instructed to restate every assertion; weighting assertions by clinical importance before generation could close that gap."],"forward_implications":["Personal pre-consultation questionnaires for follow-up visits can be generated automatically from an existing EMR, cutting the manual chart-review burden that precedes many appointments.","Disease-specific questionnaires can be produced for whole EMR corpora within minutes, with questions ordered by how often each clinical pathway occurs in the data.","Because coverage is enforced at the assertion level, every question in the final questionnaire can be traced back to a specific extracted fact or causal edge, which makes the output auditable by clinicians.","The large gap between the framework and direct LLM prompting (84.2% versus 42.1% personal coverage) implies that decomposing the task into explicit structures, rather than relying on larger context windows or better prompts alone, is what drives completeness on long, multi-condition records."],"supporting_citations":[{"why":"Supplies the GPT-4o model that performs atomic-assertion extraction, causal-network construction, and questionnaire generation, so every reported result depends on it.","marker":"OpenAI, 2024"},{"why":"Provides BCEmbedding, whose vectors are concatenated to form the edge embeddings that define causal-network similarity for clustering.","marker":"NetEase Youdao, 2023"},{"why":"Provides the hierarchical agglomerative clustering method used to group personal causal networks into representative disease pathways.","marker":"Murtagh & Legendre, 2014"},{"why":"Supplies the ICD-10 classification used to group EMRs by primary disease before clustering and disease-specific questionnaire generation.","marker":"WHO & OMS, 1992"}],"fun_headline_variants":["Structured pipeline outperforms direct LLM for EMR questionnaires","Causal networks drive LLM pre-consult questionnaire gains","Multi-stage LLM framework nearly matches expert questionnaires","LLM framework turns EMRs into tailored questionnaires","Expert-validated LLM pipeline improves questionnaire fact coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the atomic assertions the LLM extracts in Stage 1 are a complete and correct inventory of the clinically meaningful facts in each EMR, so a questionnaire that covers all of them is judged comprehensive; if extraction silently drops or distorts facts, the reported coverage measures prompt compliance rather than clinical completeness.","fun_headline_variants_meta":{"raw":{"variants":["Structured pipeline outperforms direct LLM for EMR questionnaires","Causal networks drive LLM pre-consult questionnaire gains","Multi-stage LLM framework nearly matches expert questionnaires","LLM framework turns EMRs into tailored questionnaires","Expert-validated LLM pipeline improves questionnaire fact coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001282,"raw_usage":{"total_tokens":5226,"prompt_tokens":915,"completion_tokens":4311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":4232}},"tokens_in":531,"tokens_out":4311,"duration_ms":29670,"temperature":1.0,"reasoning_tokens":4232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:03:20.973684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 50 EMRs from the same hospital, ask two independent clinicians to list every key fact without seeing the model output, run Stage 1 on the same records, and recompute the questionnaire's coverage against the clinicians' list instead of the model's own assertions; if that independently measured coverage falls well below the reported 84.2% and 92.2%, the central completeness claim fails.","supporting_citations":[],"review_version":1}