{"id":"d3107d74-932b-4cdb-89cc-72fd6f43a4ea","arxiv_id":"2504.16394","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ConTextual filters clinical notes to attention-important tokens, augments them with a patient-specific knowledge graph, and generates summaries that outperform several baselines on MIMIC-BHC and SOAP summarization benchmarks.","lead":"ConTextual is a method that compresses clinical notes by keeping only the most attention-important tokens and adding facts from a knowledge graph before an LLM writes the summary. The authors report higher summary quality and higher throughput on two clinical datasets, but the main comparison table appears to use differently tuned settings than the stated experimental setup.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 6 defines token importance as a row sum of attention; since every attention row sums to 1, all tokens receive identical I_i, making CPTF's top-k selection degenerate.","rationale":"The reader's conditional verdict is more generous than the text supports. The most load-bearing condition for the abstract's claim is that CPTF actually selects contextually important tokens. That condition fails at Eq. 6 for mathematical reasons, not empirical uncertainty: every row of a softmax attention matrix sums to 1, so the proposed importance score is position-independent. This makes the core novelty of the paper degenerate as written. The reader identified the attention proxy as unvalidated and noticed the Table 3 hyperparameter mismatch, but not this formal degeneracy, so my agreement is partial. A matched-configuration rerun could repair Table 3, but it would not repair Eq. 6 unless the importance definition is changed, for example to column sums, gradient-based attribution, or explicitly unnormalized attention scores. For the submitted version, the central claim is not supported, so I recommend REJECT. If the authors can show that their code uses a different attention definition or correct the formula and regenerate all tables, conditional acceptance would be worth revisiting. No judgment is made about author intent; the issue is internal to the equations.","tokens_in":19725,"tokens_out":6479,"duration_ms":69249,"concrete_test":"Use the released code to extract the per-token importance vector I from Eq. 6 on one MIMIC-BHC note with LLaMA 3.2 attention. Check whether I has zero variance across positions; if it does, compute the CPTF-reduced text and compare it with simple truncation to the first floor(r * n) tokens. If the two outputs match or I is constant, the proposed importance ranking is a no-op, and the central claim about context-preserving token filtering fails.","verdict_should_be":"REJECT","load_bearing_attack":"Equation 6 defines I_i = sum_l w_l * (1/n) * sum_j \\bar A_l[i,j]. In standard transformer attention, each row of \\bar A_l is a softmax distribution over keys, so sum_j \\bar A_l[i,j] = 1 for every token i, and causal masking preserves this property. Consequently I_i = (1/n) * sum_l w_l, a constant independent of i. The top-k step in Eq. 7 and Algorithm 1 then has no content: every subset of size k is equally 'important', and the reconstructed text is either the first k tokens or an arbitrary tie-break. This is not merely a question of whether attention is a good proxy for clinical importance; as written, the proxy cannot rank tokens at all. The paper nowhere defines attention as pre-softmax logits, and Fig. 2 and section 3.1 describe attention weights, so the natural reading is softmax probabilities. If so, the contribution of CPTF to the reported gains is unestablished; the gains would have to come from length reduction or KG context alone. This degeneracy is prior to the Table 3/protocol mismatch and the inconsistent latency and throughput numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConTextual, a framework for clinical text summarization that combines two components: Context-Preserving Token Filtering (CPTF), which selects a subset of input tokens using attention-derived importance scores, and a domain-specific knowledge graph (KG) whose entities are retrieved by patient identifier and concatenated with the reduced text before summary generation. The method is evaluated on the MIMIC-IV-Ext-BHC and SOAP summary datasets using LLaMA 3.2 1B, with comparisons to several baselines and ablations, and the authors claim consistent improvements in BLEU, ROUGE-L, BERTScore, LLM-as-a-judge scores, and efficiency.","tokens_in":19917,"tokens_out":4991,"duration_ms":47114,"significance":"If the reported gains were valid, ConTextual would offer a scalable, lightweight approach to clinical summarization: attention-based token filtering plus KG-augmented context could reduce computational cost while preserving fidelity, which is practically valuable for resource-constrained healthcare settings. The paper also ships a public code repository and evaluates on two public benchmark datasets, which is a strength. However, the central technical claims are currently undermined by a degeneracy in the token-importance definition, an inconsistent evaluation protocol, and a possible data-leakage channel through the knowledge graph, so the significance cannot be assessed until these issues are resolved.","major_comments":[{"comment":"The token importance score I_i is constant across all token positions as written. In standard transformer attention, each row of the attention matrix \\bar A_l is a softmax distribution, so \\sum_j \\bar A_l[i,j] = 1 for every i, including under causal masking. Equation (6) then gives I_i = (1/n) \\sum_l w_l, which is independent of i, and the top-k selection in Eq. (7) and Algorithm 1 is degenerate, choosing an arbitrary subset of size k. This means the claimed attention-based filtering cannot rank tokens at all, and the CPTF component's contribution to the reported results is unestablished. The paper needs to define a non-degenerate importance measure (for example, based on pre-softmax attention logits, attention gradients, or head-wise statistics) and rerun the ablation; the current Table 1 also shows that adding CPTF to LLaMA 3.2 lowers ROUGE-L on MIMIC-BHC from 8.85 to 7.08, which further indicates that the mechanism as described is not preserving the most relevant content.","section":"Section 3.1, Eq. (6) and Algorithm 1"},{"comment":"The experimental protocol is inconsistent across tables. Section 4.1 states that all experiments use a fixed generation budget of 200 tokens and temperature 0.7, but the ConTextual results in Table 3 (MIMIC-BHC BLEU-1 12.63, BLEU-2 4.65, ROUGE-L 11.04; SOAP BLEU-1 11.55, BLEU-2 6.09, ROUGE-L 10.70) match the Appendix D configuration Token=300, Temp=0.1, not the stated 200/0.7. Meanwhile Table 1 reports ConTextual on MIMIC-BHC with BLEU-1 9.06, BLEU-2 3.35, ROUGE-L 9.98, and BERT-F1 81.48, which corresponds to a different configuration. The baselines are therefore not compared under the same settings as ConTextual, and the paper presents two different sets of numbers for the same named model. The authors must fix one protocol, report the configuration clearly, and rerun all baselines under identical conditions.","section":"Section 4.1 and Table 3 / Appendix D"},{"comment":"The knowledge graph appears to leak information from the test cohort. The KG is constructed from clinical records of the same patients whose notes are being summarized, and the retrieval function \\eta(d) uses the unique patient identifier in the input note to retrieve entities directly linked to that patient. As a result, the retrieved context C(d) can contain information derived from the very note being summarized, or from other notes of the same patient, that is not actually available in the input at inference time. This makes the reported gains in clinical fidelity and factual consistency difficult to interpret. The KG must be built exclusively from the training split, and the authors should confirm that no test-patient entities or relations are included in retrieval.","section":"Section 3.2, Eq. (11), and Appendix B"},{"comment":"The efficiency numbers are internally inconsistent. On MIMIC-BHC, LLaMA 3.2 has throughput 36.72 summaries/s and latency 3.61 s, while LLaMA 3.2 + CPTF has throughput 139.10 summaries/s and latency 12.38 s, and ConTextual has throughput 142.87 summaries/s and latency 14.29 s. Higher throughput with substantially higher latency is not explained by the text, which attributes the latency increase to KG retrieval but does not account for the throughput increase. The authors should specify the measurement protocol (batch size, hardware, warmup, whether latency is per summary or per batch) and provide a consistent interpretation of these metrics.","section":"Table 4"}],"minor_comments":[{"comment":"The first sentence of the abstract contains the typo 'Contextual' instead of 'ConTextual'.","section":"Abstract"},{"comment":"The text writes 'through CPFT' where the acronym should be 'CPTF' to match the method name.","section":"Section 3, paragraph after Eq. (3)"},{"comment":"The citation 'Authors, 2024' is an incomplete placeholder with 'Journal Name, Volume(Number):Page Range, 2024' and should be filled in or removed.","section":"Related Work, reference list"},{"comment":"The relationship counts do not add up: 6,760 + 5,469 + 3,214 = 15,443, but the text reports 11,443 total edges.","section":"Appendix B, Knowledge Graph Construction"},{"comment":"The x-axis label in Figure 4 appears to be missing the symbol for \\alpha, and the caption should explicitly state which metric each panel shows.","section":"Figure 4 and Appendix C"},{"comment":"The few-shot row in Table 7 reports numbers identical to the ConTextual Token=300, Temp=0.1 row in Table 5; clarify whether the prompting-strategy comparison was run under the final configuration or a different one to avoid confusion.","section":"Table 7 and Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a practically important problem and has a public code release, but the current version contains a degeneracy in the core token-importance formula that invalidates the CPTF mechanism as written, and the evaluation protocol has inconsistencies and a likely data-leakage channel. These are addressable in principle, so I recommend major revision rather than rejection, but the authors will need to rerun the experiments with a corrected importance measure and a clean evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper proposes a compression-plus-KG pipeline for clinical summarization and the empirical effort is broad, but the formal heart of the method has a load-bearing flaw. Equation 6 defines token importance as an average of attention rows. With standard softmax attention, every row of an attention matrix sums to 1, so I_i is identical for every token. Top-k selection then has no content. Unless the authors mean pre-softmax logits (they never say so), CPTF as written cannot be what produces the reported improvements. That is not a minor gap; it is the central mechanism.\n\nWhat the paper does well: the problem is real, the two benchmarks are public, the ablation design is sensible, the appendix material is unusually thorough, and the repo is public. Combining token pruning with patient-specific KG context for generative summarization is a new application even though each ingredient is known. The writing is clear and the authors engage seriously with prior work.\n\nThe soft spots are consequential. Section 4.1 fixes 200 tokens at temperature 0.7, but Table 3's ConTextual row matches Appendix D's best configuration (300 tokens, temp 0.1). Table 1 and Table 3 report different BERT-F1 for the same named ConTextual on MIMIC-BHC (81.48 vs 81.37) without explanation, which tracks the configuration mismatch. Table 4's latency and throughput numbers are mutually inconsistent, since throughput should track the reciprocal of latency. No significance tests are reported. The KG is built from the same cohort as the input notes, so calling it \"domain knowledge\" overstates what is added; it is largely same-source augmentation. There is also a placeholder reference in the bibliography.\n\nIf the attention-based importance score is never well-defined, then the reported gains could come from length reduction alone, from KG context, or from hyperparameter search, and the paper's stated contribution is unestablished. These are fixable: define a salience measure that is not constant by construction, rerun all comparisons under one matched configuration, report significance, and correct the efficiency metrics.\n\nRecommendation: send it to peer review, because the idea is salvageable and the empirical scope deserves referee time, but it should come back with major revision. As it stands, I would not cite it.","headline":"The empirical package is promising but the formal definition of the main token-filtering mechanism is degenerate as written, so the paper's central contribution is not currently established.","tokens_in":20483,"tokens_out":2993,"would_cite":false,"duration_ms":30083,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that attention-based token filtering plus a domain knowledge graph makes a 1B-parameter LLM outperform larger baselines on clinical summarization.","keywords":["clinical text summarization","context-preserving token filtering","knowledge graph","retrieval-augmented generation","large language models","attention mechanisms","electronic health records","MIMIC-IV"],"falsifier":"Run ConTextual on a held-out set of notes with clinician-annotated essential tokens; if summaries generated from the gold tokens are not at least as good as those from CPTF's attention-selected tokens, the attention proxy is not doing the claimed work. A second decisive check is to compare full ConTextual with a variant that keeps the knowledge-graph context but skips CPTF: if the gap mostly vanishes, token filtering is not load-bearing.","tokens_in":19462,"feed_emoji":"🩺","tokens_out":7395,"duration_ms":65289,"temperature":0.7,"pith_summary":"ConTextual is a framework for summarizing long clinical notes with a small language model. Its central claim is that retaining only the tokens that the model's own attention layers rank as important, and then supplementing those tokens with facts from a domain-specific knowledge graph, produces summaries that are more coherent and clinically faithful than summarizing the full text. The paper tests this on two public benchmarks, brief hospital course notes and doctor-patient conversations with SOAP summaries, and reports that ConTextual beats all baselines, including larger models, on lexical, semantic, and LLM-judged quality metrics. If this holds, the practical payoff is that high-quality clinical summarization can be pursued by curating and enriching the input rather than scaling the model.","feed_headline":"Filtered notes plus medical knowledge graphs beat larger LLMs","feed_subtitle":"On MIMIC and SOAP benchmarks, ConTextual outscored all baselines on lexical and semantic metrics using a 1B model.","key_machinery":"The load-bearing mechanism is the Context-Preserving Token Filtering algorithm: token $i$ gets importance $I_i = \\sum_{l=1}^L w_l \\cdot \\frac{1}{n} \\sum_{j=1}^n \\bar{A}_l[i,j]$, where $\\bar{A}_l$ is the head-averaged attention matrix of layer $l$ and $w_l = \\alpha + (1-\\alpha)\\frac{l}{L}$, and the top $k = \\lfloor r n \\rfloor$ tokens are kept in their original order. The second mechanism is a domain knowledge graph over patients, problems, treatments, and tests, with HAS PROBLEM, UNDERWENT TEST, and WAS TREATED WITH edges, built from the same patient cohort. During generation, retrieval-augmented prompting concatenates the reduced text with entities linked to the patient's identifier, so the LLM sees both compressed source text and structured clinical relations.","core_discovery":"The authors claim that attention-based token filtering and structured knowledge retrieval are complementary rather than competing ways to improve clinical summarization. ConTextual computes a per-token importance score from layer-weighted attention, keeps the top fraction of tokens while preserving their original order, and concatenates the reduced text with patient-linked knowledge-graph context before generation. On two public benchmarks, the paper reports the highest BLEU-1, BLEU-2, ROUGE-L, and BERT-F1 among all baselines, for example MIMIC-BHC BLEU-1 12.63 versus 10.52 for the strongest instruction-tuned baseline, and BERT-F1 81.37 versus 80.71 for the strongest larger baseline. Their ablation shows that token filtering alone can reduce ROUGE-L on MIMIC-BHC, while adding the knowledge-graph context restores and exceeds baseline quality, which the authors present as evidence that the two components work together.","pith_inferences":["The paper does not validate its attention-importance proxy against clinician annotations; replacing CPTF's selection with gold clinician-labeled tokens would test whether attention is actually the source of the quality gain.","Because the KG is built from the same cohort used for evaluation, a natural extension is to rerun ConTextual with a public, out-of-domain KG; if the gains persist, the method generalizes beyond cohort-specific graph construction.","CPTF retains raw tokens and preserves their order, so it may discard multi-word clinical phrases such as 'ruptured AVM'; an extension that selects phrases or spans could retain more meaning at the same retention ratio.","The compression-plus-structured-context recipe is not inherently clinical, so it could be tested on other dense long-form domains such as legal opinions or scientific literature, where attention salience and entity relations both matter."],"forward_implications":["A 1B-parameter model with CPTF and KG context can outperform 7B-parameter and domain-adapted baselines on standard clinical summarization metrics, so input curation can substitute for model scale in this setting.","CPTF is model-agnostic and requires no fine-tuning, so the same token-filtering module can be layered onto other decoder-only LLMs to shorten prompts and raise throughput.","The ablation pattern implies that aggressive token pruning without structured medical context can hurt fidelity, so token filtering and KG augmentation should be treated as a pair rather than evaluated alone.","Few-shot prompting with structured clinical tags yields better ROUGE-L and BERT metrics than zero-shot or one-shot prompting within this framework, making prompt design a cheap source of additional gains."],"supporting_citations":[{"why":"Supplies the MIMIC-IV-Ext-BHC dataset and the brief hospital course summarization benchmark.","marker":"[Aali et al., 2024a]"},{"why":"Supplies the SOAP summary dataset from doctor-patient conversations.","marker":"[Neupane, 2024]"},{"why":"Is the base LLaMA 3.2 1B instruction-tuned model used for both CPTF and summary generation.","marker":"[AI, 2024]"},{"why":"Introduces attention-based token pruning, the prior work CPTF adapts from encoder-only to generative models.","marker":"[Goyal et al., 2020]"},{"why":"Provides Flan-T5, the strongest instruction-tuned baseline on lexical metrics that ConTextual must beat.","marker":"[Chung et al., 2022]"},{"why":"Provides Mistral-7B-Instruct, the strongest larger baseline on BERT-F1 that ConTextual must beat.","marker":"[Jiang et al., 2023]"},{"why":"Supplies the biomedical knowledge-graph background for structured context integration.","marker":"[Lu et al., 2025]"},{"why":"Defines BERTScore, the semantic metric behind the BERT-F1 comparisons.","marker":"[Zhang et al., 2019]"}],"fun_headline_variants":["Attention-filtered tokens and domain KGs outperform larger LLMs","Filtered notes plus medical KGs sharpen clinical summaries","Selective token retention and KGs improve clinical summary fidelity","Complementary boost: token filtering and KGs beat big LLMs","Compact model with token filtering and KGs wins clinical summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that LLaMA 3.2's attention weights rank tokens by clinical importance, a proxy the paper does not test against human clinical judgment.","fun_headline_variants_meta":{"raw":{"variants":["Attention-filtered tokens and domain KGs outperform larger LLMs","Filtered notes plus medical KGs sharpen clinical summaries","Selective token retention and KGs improve clinical summary fidelity","Complementary boost: token filtering and KGs beat big LLMs","Compact model with token filtering and KGs wins clinical summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1441,"prompt_tokens":939,"completion_tokens":502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":419}},"tokens_in":555,"tokens_out":502,"duration_ms":5307,"temperature":1.0,"reasoning_tokens":419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:04:37.706683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ConTextual on a held-out set of notes with clinician-annotated essential tokens; if summaries generated from the gold tokens are not at least as good as those from CPTF's attention-selected tokens, the attention proxy is not doing the claimed work. A second decisive check is to compare full ConTextual with a variant that keeps the knowledge-graph context but skips CPTF: if the gap mostly vanishes, token filtering is not load-bearing.","supporting_citations":[],"review_version":1}