Pith. sign in

REVIEW 4 major objections 6 minor 34 references

ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that attention-based token filtering plus a domain knowledge graph makes a 1B-parameter LLM outperform larger baselines on clinical summarization.

desk verdict The empirical package is promising but the formal definition of the main token-filtering mechanism is degenerate as written, so the paper's central contribution is not currently established. read the letter →

arxiv 2504.16394 v3 pith:GJRJW4LR submitted 2025-04-23 cs.CL cs.AI

Fahmida Liza Piya , Rahmatollah Beheshti This is my paper · ORCID
classification cs.CLcs.AI
keywords clinicaltextsummarizationcontext-preservingtokenfilteringknowledgegraphretrieval-augmentedgenerationlargelanguagemodelsattentionmechanismselectronichealthrecordsMIMIC-IV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ConTextual is a framework for summarizing long clinical notes with a small language model. Its central claim is that retaining only the tokens that the model's own attention layers rank as important, and then supplementing those tokens with facts from a domain-specific knowledge graph, produces summaries that are more coherent and clinically faithful than summarizing the full text. The paper tests this on two public benchmarks, brief hospital course notes and doctor-patient conversations with SOAP summaries, and reports that ConTextual beats all baselines, including larger models, on lexical, semantic, and LLM-judged quality metrics. If this holds, the practical payoff is that high-quality clinical summarization can be pursued by curating and enriching the input rather than scaling the model.

What carries the argument

The load-bearing mechanism is the Context-Preserving Token Filtering algorithm: token $i$ gets importance $I_i = \sum_{l=1}^L w_l \cdot \frac{1}{n} \sum_{j=1}^n \bar{A}_l[i,j]$, where $\bar{A}_l$ is the head-averaged attention matrix of layer $l$ and $w_l = \alpha + (1-\alpha)\frac{l}{L}$, and the top $k = \lfloor r n \rfloor$ tokens are kept in their original order. The second mechanism is a domain knowledge graph over patients, problems, treatments, and tests, with HAS PROBLEM, UNDERWENT TEST, and WAS TREATED WITH edges, built from the same patient cohort. During generation, retrieval-augmented prompting concatenates the reduced text with entities linked to the patient's identifier, so the LLM sees both compressed source text and structured clinical relations.

What would settle it

Run ConTextual on a held-out set of notes with clinician-annotated essential tokens; if summaries generated from the gold tokens are not at least as good as those from CPTF's attention-selected tokens, the attention proxy is not doing the claimed work. A second decisive check is to compare full ConTextual with a variant that keeps the knowledge-graph context but skips CPTF: if the gap mostly vanishes, token filtering is not load-bearing.

Watch

Extended reading notes

Core claim

The authors claim that attention-based token filtering and structured knowledge retrieval are complementary rather than competing ways to improve clinical summarization. ConTextual computes a per-token importance score from layer-weighted attention, keeps the top fraction of tokens while preserving their original order, and concatenates the reduced text with patient-linked knowledge-graph context before generation. On two public benchmarks, the paper reports the highest BLEU-1, BLEU-2, ROUGE-L, and BERT-F1 among all baselines, for example MIMIC-BHC BLEU-1 12.63 versus 10.52 for the strongest instruction-tuned baseline, and BERT-F1 81.37 versus 80.71 for the strongest larger baseline. Their ablation shows that token filtering alone can reduce ROUGE-L on MIMIC-BHC, while adding the knowledge-graph context restores and exceeds baseline quality, which the authors present as evidence that the two components work together.

Load-bearing premise

The method's gains rest on the assumption that LLaMA 3.2's attention weights rank tokens by clinical importance, a proxy the paper does not test against human clinical judgment.

Editorial extensions

If this is right

  • A 1B-parameter model with CPTF and KG context can outperform 7B-parameter and domain-adapted baselines on standard clinical summarization metrics, so input curation can substitute for model scale in this setting.
  • CPTF is model-agnostic and requires no fine-tuning, so the same token-filtering module can be layered onto other decoder-only LLMs to shorten prompts and raise throughput.
  • The ablation pattern implies that aggressive token pruning without structured medical context can hurt fidelity, so token filtering and KG augmentation should be treated as a pair rather than evaluated alone.
  • Few-shot prompting with structured clinical tags yields better ROUGE-L and BERT metrics than zero-shot or one-shot prompting within this framework, making prompt design a cheap source of additional gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not validate its attention-importance proxy against clinician annotations; replacing CPTF's selection with gold clinician-labeled tokens would test whether attention is actually the source of the quality gain.
  • Because the KG is built from the same cohort used for evaluation, a natural extension is to rerun ConTextual with a public, out-of-domain KG; if the gains persist, the method generalizes beyond cohort-specific graph construction.
  • CPTF retains raw tokens and preserves their order, so it may discard multi-word clinical phrases such as 'ruptured AVM'; an extension that selects phrases or spans could retain more meaning at the same retention ratio.
  • The compression-plus-structured-context recipe is not inherently clinical, so it could be tested on other dense long-form domains such as legal opinions or scientific literature, where attention salience and entity relations both matter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ConTextual, a framework for clinical text summarization that combines two components: Context-Preserving Token Filtering (CPTF), which selects a subset of input tokens using attention-derived importance scores, and a domain-specific knowledge graph (KG) whose entities are retrieved by patient identifier and concatenated with the reduced text before summary generation. The method is evaluated on the MIMIC-IV-Ext-BHC and SOAP summary datasets using LLaMA 3.2 1B, with comparisons to several baselines and ablations, and the authors claim consistent improvements in BLEU, ROUGE-L, BERTScore, LLM-as-a-judge scores, and efficiency.

Significance. If the reported gains were valid, ConTextual would offer a scalable, lightweight approach to clinical summarization: attention-based token filtering plus KG-augmented context could reduce computational cost while preserving fidelity, which is practically valuable for resource-constrained healthcare settings. The paper also ships a public code repository and evaluates on two public benchmark datasets, which is a strength. However, the central technical claims are currently undermined by a degeneracy in the token-importance definition, an inconsistent evaluation protocol, and a possible data-leakage channel through the knowledge graph, so the significance cannot be assessed until these issues are resolved.

major comments (4)
  1. [Section 3.1, Eq. (6) and Algorithm 1] The token importance score I_i is constant across all token positions as written. In standard transformer attention, each row of the attention matrix \bar A_l is a softmax distribution, so \sum_j \bar A_l[i,j] = 1 for every i, including under causal masking. Equation (6) then gives I_i = (1/n) \sum_l w_l, which is independent of i, and the top-k selection in Eq. (7) and Algorithm 1 is degenerate, choosing an arbitrary subset of size k. This means the claimed attention-based filtering cannot rank tokens at all, and the CPTF component's contribution to the reported results is unestablished. The paper needs to define a non-degenerate importance measure (for example, based on pre-softmax attention logits, attention gradients, or head-wise statistics) and rerun the ablation; the current Table 1 also shows that adding CPTF to LLaMA 3.2 lowers ROUGE-L on MIMIC-BHC from 8.85 to 7.08, which further indicates that the mechanism as described is not preserving the most relevant content.
  2. [Section 4.1 and Table 3 / Appendix D] The experimental protocol is inconsistent across tables. Section 4.1 states that all experiments use a fixed generation budget of 200 tokens and temperature 0.7, but the ConTextual results in Table 3 (MIMIC-BHC BLEU-1 12.63, BLEU-2 4.65, ROUGE-L 11.04; SOAP BLEU-1 11.55, BLEU-2 6.09, ROUGE-L 10.70) match the Appendix D configuration Token=300, Temp=0.1, not the stated 200/0.7. Meanwhile Table 1 reports ConTextual on MIMIC-BHC with BLEU-1 9.06, BLEU-2 3.35, ROUGE-L 9.98, and BERT-F1 81.48, which corresponds to a different configuration. The baselines are therefore not compared under the same settings as ConTextual, and the paper presents two different sets of numbers for the same named model. The authors must fix one protocol, report the configuration clearly, and rerun all baselines under identical conditions.
  3. [Section 3.2, Eq. (11), and Appendix B] The knowledge graph appears to leak information from the test cohort. The KG is constructed from clinical records of the same patients whose notes are being summarized, and the retrieval function \eta(d) uses the unique patient identifier in the input note to retrieve entities directly linked to that patient. As a result, the retrieved context C(d) can contain information derived from the very note being summarized, or from other notes of the same patient, that is not actually available in the input at inference time. This makes the reported gains in clinical fidelity and factual consistency difficult to interpret. The KG must be built exclusively from the training split, and the authors should confirm that no test-patient entities or relations are included in retrieval.
  4. [Table 4] The efficiency numbers are internally inconsistent. On MIMIC-BHC, LLaMA 3.2 has throughput 36.72 summaries/s and latency 3.61 s, while LLaMA 3.2 + CPTF has throughput 139.10 summaries/s and latency 12.38 s, and ConTextual has throughput 142.87 summaries/s and latency 14.29 s. Higher throughput with substantially higher latency is not explained by the text, which attributes the latency increase to KG retrieval but does not account for the throughput increase. The authors should specify the measurement protocol (batch size, hardware, warmup, whether latency is per summary or per batch) and provide a consistent interpretation of these metrics.
minor comments (6)
  1. [Abstract] The first sentence of the abstract contains the typo 'Contextual' instead of 'ConTextual'.
  2. [Section 3, paragraph after Eq. (3)] The text writes 'through CPFT' where the acronym should be 'CPTF' to match the method name.
  3. [Related Work, reference list] The citation 'Authors, 2024' is an incomplete placeholder with 'Journal Name, Volume(Number):Page Range, 2024' and should be filled in or removed.
  4. [Appendix B, Knowledge Graph Construction] The relationship counts do not add up: 6,760 + 5,469 + 3,214 = 15,443, but the text reports 11,443 total edges.
  5. [Figure 4 and Appendix C] The x-axis label in Figure 4 appears to be missing the symbol for \alpha, and the caption should explicitly state which metric each panel shows.
  6. [Table 7 and Appendix D] The few-shot row in Table 7 reports numbers identical to the ConTextual Token=300, Temp=0.1 row in Table 5; clarify whether the prompting-strategy comparison was run under the final configuration or a different one to avoid confusion.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. 6 defines token importance as a row sum of softmax attention, making I_i constant and the central CPTF top-k selection vacuous by construction.

  1. self definitional [Section 3.1, Eqs. (4)-(9); Algorithm 1, Steps 2-3 (Appendix A)]
    "The token importance score Ii is computed ... Ii = sum_{l=1}^{L} w_l * (1/n) * sum_{j=1}^{n} A_l[i,j] ... S <- argsort(-I)[:k] // Select indices of top-k important tokens"

    Under the standard transformer attention used in Eq. 4, each row of the head-averaged attention matrix A_l is a softmax distribution over the n keys, so sum_j A_l[i,j] = 1 for every token i. Substituting into Eq. 6 gives I_i = (1/n) sum_l w_l, a token-independent constant. Therefore Eq. 7's constrained optimization and Algorithm 1's argsort(-I)[:k] cannot distinguish any token from any other: every k-subset has the same objective value, and the 'top-k important tokens' are chosen by arbitrary tie-breaking rather than by the claimed context-preserving importance. The paper nowhere defines A_l as pre-softmax logits; it repeatedly says 'attention weights' and averages attention matrices. Hence the central CPTF selection mechanism reduces by construction to no content.

full rationale

The main circularity-like defect is in the CPTF scoring definition itself: Eq. (6) defines I_i as a row sum of attention probabilities, and because those probabilities are normalized, the score is uniform across all tokens. This is not an external benchmark concern but a direct consequence of the paper's own equations, so the claimed 'context-preserving token filtering' cannot rank tokens at all. The reported gains would have to come from KG augmentation, length reduction, prompt design, or experimental artifacts, not from the token-importance mechanism that is the paper's first contribution. The self-citations in the paper (Piya et al. 2024; Piya and Beheshti 2025) appear only as background support and are not load-bearing for the empirical claims, so no separate self-citation circularity is established. The KG is constructed from the same patient cohort and retrieved by patient identifier; that is same-source augmentation rather than a circular derivation, although it may raise separate data-leakage concerns. Because the central claimed mechanism is empty by construction but the overall framework still has independent empirical components, a score of 6 is appropriate: one core 'prediction' reduces by construction, while the paper is not wholly a self-citation loop.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The quantified claims rely on two hyperparameters, alpha and retention ratio r, the latter unreported, plus an unvalidated assumption that attention mass equals clinical importance. The knowledge graph, despite being called domain-specific knowledge, is built from the same patient cohort and therefore adds no independent medical facts; it is a structured re-encoding of the same input. No new physical or conceptual entities are introduced.

free parameters (2)
  • alpha = 0.5
    Layer-weighting hyperparameter in Eq. 5; chosen in Appendix C to balance lower and higher layer attention, justified by stability of ROUGE-L, not by theory.
  • retention ratio r = not reported
    Defined in Eq. 8 as the fraction of tokens kept by CPTF; the paper never states the value used in any experiment, making the main quality and efficiency trade-off unreproducible and a possible source of cherry-picking.
assumptions (4)
  • domain assumption Attention weights from an LLM reflect token-level importance for clinical summarization.
    CPTF defines token importance as layer-weighted mean attention in Eq. 6; no evidence links attention mass to clinical relevance.
  • ad hoc to paper The linear layer weighting scheme w_l = alpha + (1-alpha)*l/L improves importance estimates.
    Eq. 5 is a heuristic, not derived; alpha is tuned to 0.5.
  • domain assumption The patient-specific KG built from the same cohort provides complementary, accurate context via NER extraction.
    Appendix B: entities are extracted with samrawal/bert-base-uncased clinical-ner; retrieval C(d) uses patient ID overlap, so the graph is a structured view of the same notes, not an independent source, and extraction errors propagate.
  • domain assumption The evaluation datasets, gold summaries, and few-shot prompts are appropriate and fairly applied to all baselines.
    Section 4.1; baseline prompting and fine-tuning configurations are not described, and Table 3 appears to use ConTextual's best settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs." pith.science (2026). https://pith.science/paper/GJRJW4LR

@misc{pith2026250416394,
  author       = {Pith},
  title        = {Pith review of: ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJRJW4LR}},
  note         = {Machine review of arXiv:2504.16394}
}
read the original abstract

Unstructured clinical data can serve as a unique and rich source of information that can meaningfully inform clinical practice. Extracting the most pertinent context from such data is critical for exploiting its true potential toward optimal and timely decision-making in patient care. While prior research has explored various methods for clinical text summarization, most prior studies either process all input tokens uniformly or rely on heuristic-based filters, which can overlook nuanced clinical cues and fail to prioritize information critical for decision-making. In this study, we propose Contextual, a novel framework that integrates a Context-Preserving Token Filtering method with a Domain-Specific Knowledge Graph (KG) for contextual augmentation. By preserving context-specific important tokens and enriching them with structured knowledge, ConTextual improves both linguistic coherence and clinical fidelity. Our extensive empirical evaluations on two public benchmark datasets demonstrate that ConTextual consistently outperforms other baselines. Our proposed approach highlights the complementary role of token-level filtering and structured retrieval in enhancing both linguistic and clinical integrity, as well as offering a scalable solution for improving precision in clinical text generation.

Figures

Figures reproduced from arXiv: 2504.16394 by the authors.

Figure 1
Figure 1. Overview of the ConTextual Framework for clinical text summarization. . and Beheshti, 2025]. However, most prior approaches integrate KGs in a static fashion—treating the graph as a fixed resource—which means the knowledge base does not dynamically update or adapt to new data or task-specific needs, potentially limiting scalability and flexibility in fast-evolving clinical settings [Authors, 2024]. Recent work has b… view at source ↗
Figure 2
Figure 2. Overview of the Context Preserving Token Filtering algorithm. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the clinical knowledge graph constructed from unstructured EHR data. [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparing results with Weight factor for layer importance [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 9 canonical work pages

  1. [1]

    HealthGAT: Node Classifications in Electronic Health Records using Graph Attention Networks

    Fahmida Liza Piya, Mehak Gupta, and Rahmatollah Beheshti. Healthgat: Node classifications in electronic health records using graph attention networks. arXiv preprint arXiv:2403.18128,

    work page Pith review arXiv

  2. [4]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´e, Morgane Rivi`ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786,

    arXiv

  3. [6]

    MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

    11 Rishabh Bhardwaj, Kishan Patel, Simran Khanuja, Diptesh Kanojia Sharma, and Pushpak Bhat- tacharyya. Mediswift: Building fast and accurate biomedical language models with sparse pretraining. arXiv preprint arXiv:2403.00952,

    work page Pith review arXiv

  4. [7]

    Biomedlm: A domain-specific foundation model for biomedical natural language processing

    William Boag, Felix Tan, Ritwik Das, Brett K Beaulieu-Jones, and Andrew L Beam. Biomedlm: A domain-specific foundation model for biomedical natural language processing. arXiv preprint arXiv:2403.18421,

    arXiv

  5. [8]

    Improving llms for clinical ner via prompt engineering

    Y Hu, Q Chen, J Du, et al. Improving llms for clinical ner via prompt engineering. Journal of the American Medical Informatics Association, 31(9):1812–1820, 2024a. Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, et al. Longrecipe: Recipe for efficient long context generaliza- tion...

    arXiv

  6. [9]

    A comprehensive study on quantization techniques for large language models

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization techniques for large language models. In 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pages 224–231. IEEE,

    work page 2024

  7. [10]

    Enabling Scalable Evaluation of Bias Patterns in Medical LLMs

    Hamed Fayyaz, Raphael Poulain, and Rahmatollah Beheshti. Enabling scalable evaluation of bias patterns in medical llms. arXiv preprint arXiv:2410.14763,

    work page Pith review arXiv

  8. [11]

    Aligning (medical) llms for (counter- factual) fairness

    Raphael Poulain, Hamed Fayyaz, and Rahmatollah Beheshti. Aligning (medical) llms for (counter- factual) fairness. arXiv preprint arXiv:2408.12055,

    arXiv

Show all 34 references
  1. [12]

    Large language models can learn temporal reasoning

    Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853,

    arXiv

  2. [13]

    On context utilization in summarization with large language models

    12 Mathieu Ravaut, Aixin Sun, Nancy F Chen, and Shafiq Joty. On context utilization in summarization with large language models. arXiv preprint arXiv:2310.10570,

    arXiv

  3. [14]

    Ada-leval: Evaluating long-context llms with length-adaptable benchmarks

    Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. arXiv preprint arXiv:2404.06480,

    arXiv

  4. [15]

    Evaluating and mitigating limitations of large language models in clinical decision making

    Paul Hager, Friederike Jungmann, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Robbie Holland, Rickmer Braren, Marcus Makowski, Georgios Kaisis, et al. Evaluating and mitigating limitations of large language models in clinical decision making. medRxiv, pages 2024–01,

    2024

  5. [17]

    Optimal path for biomedical text summarization using pointer gpt

    Hyunkyung Han and Jaesik Choi. Optimal path for biomedical text summarization using pointer gpt. arXiv preprint arXiv:2404.08654,

    arXiv

  6. [19]

    Large language models are reasoning teachers

    Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071,

    arXiv

  7. [20]

    Teaching small language models to reason

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410,

    arXiv

  8. [21]

    Rho-1: Not all tokens are what you need

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965,

    arXiv

  9. [22]

    Sparser is faster and less is more: Efficient sparse attention for long-range transformers

    Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747,

    arXiv

  10. [23]

    Large language models in healthcare: A comprehensive benchmark

    13 Fenglin Liu, Hongjian Zhou, Yining Hua, Omid Rohanian, Lei Clifton, and David Clifton. Large language models in healthcare: A comprehensive benchmark. medRxiv, pages 2024–04, 2024b. Ciyuan Peng, Feng Xia, Mehdi Naseriparsa, and Francesco Osborne. Knowledge graphs: Opportuni...

    2024 arXiv

  11. [25]

    Vahan Arsenyan et al

    URL https://arxiv.org/abs/2501.11632. Vahan Arsenyan et al. Large language models for biomedical knowledge graph construction: Infor- mation extraction from emr notes. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, Bangkok, Thailand,

    arXiv

  12. [26]

    doi: 10.18653/v1/2024.bionlp-1.23

    Association for Computational Linguistics. doi: 10.18653/v1/2024.bionlp-1.23. URL https://aclanthology.org/2024.bionlp-1.23/. Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. A survey of knowledge enhanced pre-trained language models. IEEE Transactions on...

    2024 doi

  13. [27]

    A. Aali, D. Van Veen, Y . Arefeen, J. Hom, C. Bluethgen, E. P. Reis, S. Gatidis, N. Clifford, J. Daws, A. Tehrani, J. Kim, and A. Chaudhari. MIMIC-IV-Ext-BHC: Labeled Clinical Notes Dataset for Hospital Course Summarization. PhysioNet, 2024a. URL https://physionet.org/content/...

    arXiv

  14. [28]

    Iz Beltagy, Matthew E Peters, and Arman Cohan

    URL https://huggingface.co/datasets/ SubashNeupane/dataset_SOAP_summary. Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    2004 arXiv

  15. [30]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416,

    arXiv

  16. [31]

    BioGPT: generative pre-trained transformer for biomedical text generation and mining

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 09 2022b. ISSN 1477-4054. doi: 10.1093/bib/bbac409. URL https: //do...

    doi

  17. [34]

    From local to global: A graph rag approach to query-focused summarization

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130,

    arXiv

  18. [35]

    15 A Context-Preserving Token Filtering Algorithm We introduce an algorithm—Context-Preserving Token Filtering (CPTF)—designed to retain seman- tically important tokens from an input sequence while minimizing overall length. By leveraging internal attention dynamics from a mul...

    2024

  19. [2004]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675,

    1904 arXiv

  20. [2017]

    Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11972,

    Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Kar- ishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, et al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11972,

    arXiv

  21. [2020]

    Biobart: Pretraining and evaluation of a biomedical generative language model

    Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. Biobart: Pretraining and evaluation of a biomedical generative language model. arXiv preprint arXiv:2204.03905,

    arXiv

  22. [2021]

    Automatic summarization of doctor-patient encounter dialogues using large language model through prompt tuning

    Mengxian Lyu, Cheng Peng, Xiaohan Li, Patrick Balian, Jiang Bian, and Yonghui Wu. Automatic summarization of doctor-patient encounter dialogues using large language model through prompt tuning. arXiv preprint arXiv:2403.13089,

  23. [2022]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  24. [2023]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  25. [2024]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  26. [2025]

    Bioinstruct: Bionlp tasks instruction tuning for llama model

    Zhexin Wang, Vishakh Padmakumar Kumar, Tianyu Kang, Chang Xu, Tianle Cai, Xuanjing Ma, Yefeng Zheng, Zhengping Liu, and Meng Jiang. Bioinstruct: Bionlp tasks instruction tuning for llama model. arXiv preprint arXiv:2310.19975,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.