Pith. sign in

REVIEW 4 major objections 4 minor 39 references

heiDS at ArchEHR-QA 2025: From Fixed-k to Query-dependent-k for Retrieval Augmented Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Choosing the retrieval cut-off per question, rather than fixing a top-k, keeps clinical RAG answers factual and relevant while removing manual tuning.

desk verdict A competent shared-task writeup with two simple truncation heuristics, but the abstract's 'benefits' claim is contradicted by the paper's own tables and conclusion. read the letter →

arxiv 2506.19512 v1 pith:2I3FPXNA submitted 2025-06-24 cs.CL

Ashish Chouhan , Michael Gertz This is my paper · ORCID
classification cs.CL
keywords retrievalaugmentedgenerationquery-dependent-krankedlisttruncationclinicalquestionansweringevidenceattributionelectronichealthrecordsextremevaluetheoryArchEHR-QA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a retrieval-augmented question-answering pipeline for electronic health records does not need a globally fixed number k of evidence sentences: letting the retrieval score distribution decide where to cut the ranked list per query can produce answers that are as factual and relevant as any fixed top-k. The authors enter the ArchEHR-QA 2025 shared task with a system that retrieves clinically similar sentences, truncates the list with a query-dependent-k strategy, and has an LLM generate answers citing the retrieved sentences. They contribute two parameter-free cut-off heuristics, autocut* and elbow, alongside the existing surprise and autocut methods. On the 20-case development set, the best configuration—surprise truncation with LLaMA-3.3-70B and post-retrieval attribution—reaches strict precision 0.62, recall 0.26, and F1 0.37, with query-dependent-k methods showing comparable performance to fixed-k while avoiding manual k tuning. The practical motivation is that real clinical notes are too long for full-context prompting, and per-query cut-offs adapt to variable note lengths.

What carries the argument

The machinery that carries the argument is the query-dependent-k ranked list truncation (RLT) strategy: a mechanism that inspects the ranked similarity scores between the query and candidate clinical sentences and decides where to cut. Surprise uses extreme value theory, fitting a generalized Pareto distribution to the score tail and truncating at a threshold from Pickands (1975) and Bahri et al. (2023). Autocut cuts at the first discontinuity in the score decline. The paper's new variants, autocut*, which cuts where the drop from one score to the next is significant relative to previous drops, and elbow, which finds the elbow in the score curve as in clustering, make the selection fully automatic. These strategies are paired with a cosine-similarity semantic index over sentence embeddings and either post-retrieval attribution, where the LLM cites sentence ids inline, or post-generation attribution, where a weighted lexical, fuzzy, and semantic similarity mapping with a threshold connects answer sentences back to evidence.

What would settle it

Run the official ArchEHR-QA evaluation on the 100-case test set for each retrieval strategy with the same one-shot prompt and LLaMA-3.3-70B model. If the best fixed-k strategy (k = 10, 15, or 20) beats every query-dependent-k strategy on strict F1 and overall relevance by a margin larger than the development-set spread, the paper's comparable-performance claim collapses.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that per-query ranked list truncation works: replacing a fixed top-k with a truncation point derived from the shape of similarity scores yields comparable retrieval and final-answer quality without per-corpus tuning. In the submitted pipeline, the surprise method fits a generalized Pareto distribution to the retrieval scores and sets a threshold, so each patient question receives its own k. The two new heuristics, autocut* and elbow, do the same without any parameters. The paper reports that LLaMA-3.3-70B with post-retrieval attribution and surprise truncation scores strict F1 0.37 and overall relevance 0.35, equal to the best fixed-k pipelines on the development set, while post-generation attribution lags behind with best strict F1 0.27. It also acknowledges that the best pipeline does not beat the organizer baseline, which feeds all note sentences to the model, and explains the gap by prompt sensitivity and the recall cost of retrieval.

Load-bearing premise

The paper's conclusions rest on the assumption that 20 development case studies, used both to pick the best configuration and to produce the final numbers, are representative enough to show that per-query retrieval cut-offs work as well as fixed ones.

Editorial extensions

If this is right

  • A per-query retrieval cut-off removes the need to hand-tune k, so the same pipeline can be applied to case studies with widely different numbers of clinical sentences, from a few up to 54 in the development set and 74 in the test set.
  • The best operating point is high precision at the cost of recall: strict precision reaches 0.62 while recall is only 0.26, meaning false attributions are rare but relevant evidence is often missed.
  • The two new parameter-free heuristics, autocut* and elbow, are drop-in replacements for fixed-k at the retrieval stage, with elbow reaching retrieval F1 0.50 under both strict and lenient variants in the paper's tables.
  • Post-retrieval attribution outperforms post-generation attribution in this setup, with best strict F1 0.37 versus 0.27, suggesting that asking the LLM to cite retrieved sentences inline is more effective than attributing after generation.
  • In real-world settings where full clinical notes cannot fit into the LLM context, a retrieval step is unavoidable, and query-dependent-k methods make that step self-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether the elbow and autocut* cut-offs transfer to other retrieval corpora, such as legal or scientific documents, because they look only at the geometry of similarity scores; the paper demonstrates them only on 20 development cases.
  • The reported gap to the organizer baseline is attributed to prompt and context-size differences; an untested corollary is that combining the one-shot prompt with a larger per-query context budget could recover the recall loss and close that gap.
  • The new heuristics could serve as a cheap first-stage filter in front of a re-ranker, truncating the candidate list by elbow before re-ranking the survivors, which would avoid the cost of re-ranking all candidates.
  • Because the same 20-case development set was used both to select the best configuration and to compute the final scores, the comparable-performance result is not decisive on its own; stable conclusions require the 100-case test set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents heiDS, the authors' system for the ArchEHR-QA 2025 shared task. The pipeline embeds clinical note sentences with bge-large-en-v1.5, retrieves the top-k most similar sentences for a query formed from both the patient and clinician questions, and then prompts an LLM (LLaMA-3.3-70B or Mixtral-8x7B) to generate answers with inline attributions. The central methodological claim is that replacing a fixed retrieval cut-off k with a query-dependent-k strategy (the existing surprise and autocut methods, plus two newly proposed methods, autocut* and elbow) improves the factualness and relevance of generated answers compared to fixed-k retrieval. The paper reports retrieval-level and pipeline-level experiments on the 20-case development set, including ablations of query formulation, prompting style, token limit, and attribution parameters.

Significance. If the claimed benefit of query-dependent-k were supported, the paper would offer a practical contribution: avoiding manual tuning of k in clinical RAG pipelines while maintaining or improving answer quality. The paper has concrete strengths: the code is publicly available, the official ArchEHR-QA evaluation script is used, the ablations in Appendices D-F and G systematically examine query formulation, prompting, token limits, and attribution weights, and the authors are candid about several limitations, including LLM non-determinism and the lack of held-out evaluation in the paper. However, the reported numbers do not demonstrate the central benefit claim. On the 20-case dev set, surprise and elbow match or nearly match fixed-k=10 on strict F1 (0.37 vs. 0.37) and differ by only 0.01 on overall score (0.36 vs. 0.35). The paper's own Section 4 states that the query-dependent-k strategies 'showed comparable performance to fixed-kapproaches,' which directly undercuts the abstract's 'benefits' claim. The contribution is therefore best characterized as a careful system description with a negative or neutral result, and the claims need to be realigned with the evidence.

major comments (4)
  1. [Abstract and Section 4] The abstract claims that 'The experimental results show the benefits of our strategy in producing factual and relevant answers when compared to a fixed-k.' This is not supported by Table 2: surprise achieves strict F1=0.37 and overall=0.36, elbow achieves 0.37 and 0.35, fixed-k=10 achieves 0.37 and 0.35, and fixed-k=15 achieves 0.35 and 0.35. The largest difference is 0.01 on the overall score, and Section 4 itself concludes that query-dependent-k strategies 'showed comparable performance to fixed-kapproaches.' The abstract and conclusion should be revised to state comparable performance with the practical benefit of avoiding manual k tuning, rather than claiming empirical superiority.
  2. [Section 3, footnote 2 and Tables 1-2] All experiments described in Section 3 use only the 20-case development set, and the same development set was used to select the query formulation, prompting style, token limit, retrieval strategies, and attribution parameters. With n=20, differences of 0.01-0.02 in overall score are well within the expected sampling noise, and no error bars, confidence intervals, significance tests, or results on the held-out test set are reported. This selection-on-evaluation-set loop makes the reported advantage of surprise over fixed-k=10 (0.36 vs. 0.35 overall) not interpretable as evidence for the central claim. The authors should either report bootstrap confidence intervals, perform a significance test, add test-set results, or explicitly weaken the claim to 'comparable performance' without implying statistical superiority.
  3. [Table 1] The retrieval-level results do not support the benefit claim either. Fixed-k=10 has strict F1=0.50 and lenient F1=0.58; surprise has strict F1=0.42 and lenient F1=0.49; elbow has strict F1=0.50 and lenient F1=0.55; autocut and autocut* are substantially worse. Thus the query-dependent-k strategies do not outperform fixed-k=10 on retrieval F1, and the choice of surprise as the submitted pipeline appears to be based on criteria other than the retrieval metrics reported in Table 1. The paper should discuss this discrepancy explicitly, since it weakens the motivation for preferring query-dependent-k over fixed-k in this dataset.
  4. [Section 2.4] The two newly proposed methods, autocut* and elbow, are described only informally: autocut* 'inspects how much each similarity score decreases compared to the previous score' and elbow 'locates the elbow where the transition from high to low relevance occurs.' No equations, pseudocode, or precise decision rules are provided for the 'significant changes' threshold or the elbow-location criterion. Since these methods are introduced as contributions and their results are reported in Tables 1 and 2, the lack of algorithmic detail prevents reproducibility and makes it impossible to determine whether the reported behavior is tied to the actual method or to implementation choices. Please provide precise definitions, including any hyperparameters or stopping criteria.
minor comments (4)
  1. [Title] The title contains a spacing typo: 'From Fixed-kto Query-dependent-kfor Retrieval Augmented Generation' should read 'From Fixed-k to Query-dependent-k for Retrieval Augmented Generation.'
  2. [Table 2] In the row for autocut* with LLaMA, the model name is written as 'LLaMA-3.37B'; this should be 'LLaMA-3.3-70B' to match the rest of the table.
  3. [Appendix G] The text states that 'Table 5 summarizes the top 10 configurations by strict F1-score,' but Table 5 lists only five configurations. Please either include all ten rows or correct the description.
  4. [Section 2.2] The bullet list in Section 2.2 says the decisions are 'supported by experiments' and then lists choices, but the actual experiment results for query formulation, prompting, and token limit appear only in Appendices D, E, and F. Please add cross-references at the bullet items to make the support explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's dev-set tuning raises evaluation-validity concerns but its central claim does not reduce to its own inputs by construction.

full rationale

This is an empirical shared-task system paper rather than a derivation. The query-dependent-k strategies (surprise, autocut*, elbow) are score-distribution heuristics whose cut-off rules are not defined in terms of the reported factuality or relevance outcomes, so the central comparison is not self-definitional. I found no self-citation chains: the methods are credited to external sources (Bahri et al., Weaviate, and the standard elbow method), and no load-bearing result rests on prior work by Chouhan and Gertz. The closest concern is that the pipeline was configured on the development set—query formulation, one-shot prompting, 200-token limit, retrieval-strategy selection, and the Appendix G attribution weights/thresholds—and then evaluated on that same 20-case dev set; the manuscript itself states in the Section 2.1 footnote that 'All experiments described in Section 3 use the dev set.' This is a genuine selection-on-evaluation-set validity risk, but it is not circularity of the kind the guidelines require: final scores are measurements on the same set, not quantities forced equal by definition, and no fitted parameter is renamed as a prediction. The abstract's 'benefits' claim is also not supported by Table 2 (surprise strict F1 0.37 / overall 0.36 vs. fixed-k=10 strict F1 0.37 / overall 0.35) and is contradicted by the paper's own conclusion that the strategies 'showed comparable performance to fixed-k approaches'; an overstated or unsupported empirical claim is a correctness/evidence problem, not a circular step. No circular step meeting the quoted-reduction standard is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its free parameters are the pipeline hyperparameters tuned on the dev set, and its axioms are the domain assumptions about embeddings, annotations, and statistical methods that the RAG pipeline silently depends on.

free parameters (6)
  • max_tokens = 200
    Selected by comparing 100, 200, and 300 on the dev set (Appendix F), used in all final pipelines.
  • attribution_weights w1, w2, w3 = w1=0.0, w2=0.5, w3=0.5
    Grid-searched on the dev set for post-generation attribution (Appendix G).
  • attribution_threshold T = 0.5
    Chosen in the same grid search as the attribution weights (Appendix G).
  • fixed-k values = 3, 10, 15, 20, 54
    Hand-chosen cut-offs; k=15 used as a top fixed-k pipeline in the main comparison.
  • query formulation = patient+clinician question
    Selected by dev-set comparison (Appendix D).
  • prompting approach = one-shot
    Selected by dev-set comparison (Appendix E).
assumptions (5)
  • domain assumption BAAI/bge-large-en-v1.5 embeddings with cosine similarity capture clinically relevant sentence-query relatedness.
    Section 2.1 builds the FAISS index on this embedding model; retrieval quality, and thus all downstream results, depend on this choice.
  • domain assumption The ArchEHR-QA development set annotations (essential, supplementary, not-relevant) are an accurate gold standard for citation factuality.
    Section 3.1 uses these labels as ground truth; no inter-annotator agreement is reported in this paper.
  • domain assumption The surprise RLT method's generalized Pareto fit to retrieval scores is valid for ranked lists of up to 54 sentences.
    Section 2.3 applies surprise on per-patient ranked lists; the method's statistical assumptions come from Bahri et al. (2023) and are not re-validated here.
  • domain assumption LLM outputs at temperature 0.001 are stable enough that single runs are comparable across configurations.
    The Limitations section admits non-determinism, yet Tables 1-6 report single-run scores without repeated sampling.
  • standard math Standard extreme value theory (Pickands, 1975) applies as invoked in the surprise method.
    Section 2.3 relies on generalized Pareto distributions from extreme value theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of heiDS at ArchEHR-QA 2025: From Fixed-k to Query-dependent-k for Retrieval Augmented Generation." pith.science (2026). https://pith.science/paper/2I3FPXNA

@misc{pith2026250619512,
  author       = {Pith},
  title        = {Pith review of: heiDS at ArchEHR-QA 2025: From Fixed-k to Query-dependent-k for Retrieval Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2I3FPXNA}},
  note         = {Machine review of arXiv:2506.19512}
}
abstract

This paper presents the approach of our team called heiDS for the ArchEHR-QA 2025 shared task. A pipeline using a retrieval augmented generation (RAG) framework is designed to generate answers that are attributed to clinical evidence from the electronic health records (EHRs) of patients in response to patient-specific questions. We explored various components of a RAG framework, focusing on ranked list truncation (RLT) retrieval strategies and attribution approaches. Instead of using a fixed top-k RLT retrieval strategy, we employ a query-dependent-k retrieval strategy, including the existing surprise and autocut methods and two new methods proposed in this work, autocut* and elbow. The experimental results show the benefits of our strategy in producing factual and relevant answers when compared to a fixed-$k$.

Figures

Figures reproduced from arXiv: 2506.19512 by the authors.

Figure 1
Figure 1. Distribution of the number of sentences per [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Distribution of the sentence length in the test [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 16 canonical work pages

  1. [1]

    Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Kalai. 2024. https://aclanthology.org/2024.findings-eacl.62/ Do language models know when they`re hallucinating references? In Findings of the Association for Computational Linguistics: EACL 2024, pages 912--928, St. Julian ' s, Malta. Association for Computational Linguistics

    work page 2024

  2. [2]

    Dara Bahri, Che Zheng, Yi Tay, Donald Metzler, and Andrew Tomkins. 2023. https://doi.org/10.1145/3539618.3592066 Surprise: Result list truncation via extreme value theory . In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, page 2404–2408. Association for Computing Machinery

    arXiv 2023

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    2005

  4. [4]

    Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander M a dry. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/adbea136219b64db96a9941e4249a857-Paper-Conference.pdf ContextCite: Attributing Model Generation to Context . In Advances in Neural Information Processing Systems, volume 37, pages 95764--95807. Curran Associates, Inc

    work page 2024

  5. [5]

    Amin Dada, Osman Koras, Marie Bauer, Amanda Butler, Kaleb Smith, Jens Kleesiek, and Julian Friedrich. 2025. https://aclanthology.org/2025.cl4health-1.10/ M e D i S um QA : Patient-oriented question-answer generation from discharge letters . In Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health), pages 124--136, Albuquerq...

    work page 2025

  6. [6]

    Prithiviraj Damodaran. 2023. https://github.com/PrithivirajDamodaran/FlashRank FlashRank, Lightest and Fastest 2nd Stage Reranker for search pipelines

    work page 2023

  7. [7]

    Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.910 RARR : Researching and revising what language models say, using language models . In Proceedings of the 61st Annual Meeting of the Association for...

    doi:10.18653/v1/2023.acl-long.910 2023

  8. [8]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.398 Enabling large language models to generate text with citations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465--6488, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.398 2023

Show all 39 references
  1. [9]

    a yrinen, Kaija Saranto, and Pirkko Nyk \

    Kristiina H \"a yrinen, Kaija Saranto, and Pirkko Nyk \"a nen. 2008. https://pubmed.ncbi.nlm.nih.gov/17951106/ Definition, structure, content, use and impacts of electronic health records: a review of the research literature . International journal of medical informatics, 77(5...

    2008

  2. [10]

    Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.161 Training language models to generate text with citations via fine-grained rewards . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024 doi

  3. [11]

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, and 52 others. 2024 b . https://pr...

    2024

  4. [12]

    Jeff Johnson, Matthijs Douze, and Herv \'e J \'e gou. 2019. https://ieeexplore.ieee.org/document/8733051 Billion-scale similarity search with GPUs . IEEE Transactions on Big Data, 7(3):535--547

    2019

  5. [13]

    Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwanghyun Kim, Jeewon Yang, Seunghyun Won, and Edward Choi. 2024. https://nips.cc/virtual/2024/poster/97643 EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries . In Advances...

    2024

  6. [14]

    Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aiguo Wu, and Min Zhang. 2023. https://arxiv.org/abs/2311.03731 A Survey of Large Language Models Attribution . arXiv preprint arXiv:2311.03731

    2023 arXiv

  7. [15]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  8. [16]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics,...

    2024 doi

  9. [17]

    Siru Liu, Allison B McCoy, Aileen P Wright, Babatunde Carew, Julian Z Genkins, Sean S Huang, Josh F Peterson, Bryan Steitz, and Adam Wright. 2024 b . https://pubmed.ncbi.nlm.nih.gov/37503263/ Leveraging large language models for generating responses to patient messages—a subje...

    2024

  10. [18]

    Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. https://doi.org/10.18653/v1/2024.naacl-long.167 E xpert QA : Expert-curated questions and attributed answers . In Proceedings of the 2024 Conference of the North American Chapter of ...

    2024 doi

  11. [19]

    Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. 2024. https://dl.acm.org/doi/10.1145/3626772.3657864 Ranked List Truncation for Large Language Model-based Re-Ranking . In Proceedings of the 47th International ACM SIGIR Conference on Resea...

    2024

  12. [20]

    Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and 1 others. 2022. https://arxiv.org/abs/2203.11147 Teaching language models to support answers with verified qu...

    2022 arXiv

  13. [21]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. https://arxiv.org/abs/2112.09332 WebGPT: Browser-assisted question-answering with human feedback . arXiv...

    2021 arXiv

  14. [22]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  15. [23]

    Nilay Patel, Shivashankar Subramanian, Siddhant Garg, Pratyay Banerjee, and Amita Misra. 2024. https://doi.org/10.18653/v1/2024.naacl-long.216 Towards improved multi-source attribution for long-form answer generation . In Proceedings of the 2024 Conference of the North America...

    2024 doi

  16. [24]

    James Pickands. 1975. https://projecteuclid.org/journals/annals-of-statistics/volume-3/issue-1/Statistical-Inference-Using-Extreme-Order-Statistics/10.1214/aos/1176343003.full Statistical inference using extreme order statistics . The Annals of Statistics, 3(1):119--131

    1975

  17. [25]

    Pritika Ramu, Koustava Goswami, Apoorv Saxena, and Balaji Vasan Srinivasan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.985 Enhancing post-hoc attributions in long document comprehension via coarse grained answer decomposition . In Proceedings of the 2024 Conference on E...

    2024 doi

  18. [26]

    Furkan S ahinu c , Ilia Kuznetsov, Yufang Hou, and Iryna Gurevych. 2024. https://doi.org/10.18653/v1/2024.acl-long.265 Systematic task exploration with LLM s: A study in citation text generation . In Proceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024 doi

  19. [27]

    Abel Salinas and Fred Morstatter. 2024. https://doi.org/10.18653/v1/2024.findings-acl.275 The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance . In Findings of the Association for Computational Linguistics: ACL 2024...

    2024 doi

  20. [28]

    Chi, Nathanael Sch\" a rli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch\" a rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, volume 202...

    2023

  21. [29]

    William R Small, Batia Wiesenfeld, Beatrix Brandfield-Harvey, Zoe Jonassen, Soumik Mandal, Elizabeth R Stevens, Vincent J Major, Erin Lostraglio, Adam Szerencsy, Simon Jones, and 1 others. 2024. https://pubmed.ncbi.nlm.nih.gov/39012633/ Large Language Model--Based Responses to...

    2024

  22. [30]

    Sarvesh Soni and Dina Demner-Fushman. 2025 a . A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization . arXiv preprint

    2025

  23. [31]

    Sarvesh Soni and Dina Demner-Fushman. 2025 b . Overview of the ArchEHR-QA 2025 Shared Task on Grounded Question Answering from Electronic Health Records . In The 24th Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Vienna, Austria. Association for C...

    2025

  24. [32]

    Dietrich Trautmann, Natalia Ostapuk, Quentin Grail, Adrian Pol, Guglielmo Bonifazi, Shang Gao, and Martin Gajek. 2024. https://doi.org/10.18653/v1/2024.nllp-1.14 Measuring the groundedness of legal question-answering systems . In Proceedings of the Natural Legal Language Proce...

    2024 doi

  25. [33]

    Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. https://doi.org/10.1162/tacl_a_00107 Optimizing statistical machine translation for text simplification . Transactions of the Association for Computational Linguistics, 4:401--415

    2016 doi

  26. [34]

    Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. https://www.nature.com/articles/s41597-023-02487-3 Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation . Scientific data, 10(1):586

    2023

  27. [35]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023 doi

  28. [36]

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and 1 others. 2024. https://arxiv.org/abs/2409.02897 LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA . arXiv preprint arXiv:2409.02897

    2024 arXiv

  29. [37]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 BERTScore: Evaluating Text Generation with BERT . In Proceedings of the Eighth International Conference on Learning Representations ( ICLR '20) . OpenReview.net

    2020 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.