pith. sign in

arxiv: 2605.06594 · v1 · submitted 2026-05-07 · 💻 cs.CL

Automated Clinical Report Generation for Remote Cognitive Remediation: Comparing Knowledge-Engineered Templates and LLMs in Low-Resource Settings

Yongxin Zhou , Fabien Ringeval , Fran\c{c}ois Portet This is my paper

Pith reviewed 2026-05-08 09:58 UTC · model grok-4.3

classification 💻 cs.CL
keywords clinical report generationcognitive remediation therapytemplate-based systemsLLM evaluationremote rehabilitationnatural language generationlow-resource healthcare
0
0 comments X

The pith

Template-based systems generate more clinically reliable reports than GPT-4 for remote cognitive remediation sessions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper examines how to automatically create clinical reports from data collected during home-based cognitive remediation therapy when therapists are scarce and no standard reference reports exist. It directly pits a rule-based template system, built on explicit speech therapy knowledge and decision rules, against a zero-shot GPT-4 model, with both using the same pre-extracted structured variables from session interactions. Eight evaluators rated the outputs on nine criteria and found templates superior in fluidity, coherence, and results presentation, while GPT-4 produced shorter text. The comparison highlights a practical trade-off that affects how such tools can be adopted responsibly in low-resource healthcare. It also yields specific design recommendations and a methodology for future evaluations of clinical natural language generation.

Core claim

The study demonstrates that when generating reports for avatar-guided cognitive remediation in settings without reference reports, a knowledge-engineered template system achieves higher scores on clinical reliability measures including fluidity, coherence, and presentation of results compared to a zero-shot GPT-4 approach, which instead produces more concise outputs. Both systems rely on identical expert-validated structured variables, allowing a controlled factual comparison evaluated by eight speech therapists and students via a nine-criterion questionnaire. This establishes that domain knowledge encoding supports traceability and reliability in such applications.

What carries the argument

Side-by-side comparison of rule-based templates encoding domain knowledge versus zero-shot LLM generation, both driven by the same pre-extracted structured variables and assessed through expert questionnaire

Load-bearing premise

That the ratings from a nine-criterion questionnaire by eight evaluators accurately capture clinical reliability and utility when no reference reports are available for comparison.

What would settle it

A larger study with more evaluators or direct measures of clinical decision quality showing no consistent advantage for templates over GPT-4 reports would undermine the reliability trade-off.

Figures

Figures reproduced from arXiv: 2605.06594 by Fabien Ringeval, Fran\c{c}ois Portet, Yongxin Zhou.

Figure 1
Figure 1. Figure 1: Framework for summarizing remediation sessions in THERADIA. The system processes two data types: (1) view at source ↗
Figure 2
Figure 2. Figure 2: Screenshot of the video recorded during a remediation session between a participant and the avatar (operated view at source ↗
Figure 3
Figure 3. Figure 3: Template-based report generation system. The pipeline integrates multiple data sources (dialogue transcripts, view at source ↗
Figure 4
Figure 4. Figure 4: Example of report generated by the template-based system for participant view at source ↗
Figure 5
Figure 5. Figure 5: Example of report generated by GPT-4 for participant view at source ↗
Figure 6
Figure 6. Figure 6: Screenshot of Exo 1 Retrouvez votre chemin interface. Exo 2 Objets où êtes-vous ? Shown in view at source ↗
Figure 7
Figure 7. Figure 7: Screenshot of Exo 2 Objets où êtes-vous ? interface view at source ↗
Figure 8
Figure 8. Figure 8: Screenshot of Exo 3 Que_d’accros interface view at source ↗
Figure 9
Figure 9. Figure 9: Screenshot of Exo 4 Jeux de blasons interface. Shown in view at source ↗
Figure 10
Figure 10. Figure 10: Screenshot of Exo5 Mettez de l’ordre dans ces comptes interface. Exo 6 Garçon SVP (a) Instruction (b) Exercise view at source ↗
Figure 11
Figure 11. Figure 11: Screenshot of Exo 6 Garçon SVP interface. 23 view at source ↗
Figure 12
Figure 12. Figure 12: Screenshot of Exo 7 Menez l’enquête interface. Shown in view at source ↗
Figure 13
Figure 13. Figure 13: Screenshot of Exo 8 Tour Hanoï interface. of exercise difficulty across repetitions, designed to elicit emotions in participants without their awareness. In the non-induction subgroup (E), exercise difficulty remains constant throughout the session view at source ↗
Figure 14
Figure 14. Figure 14: Excerpt from the log file recorded while participant view at source ↗
read the original abstract

The growing demand for cognitive remediation therapy, combined with limited speech therapist availability, has accelerated the adoption of remote rehabilitation tools. These systems generate large volumes of interaction data that are difficult for clinicians to review efficiently. This paper investigates automated clinical report generation for avatar-guided, home-based cognitive remediation sessions in a low-resource setting with no reference reports. We present and compare two approaches: (1) a rule-based template system encoding speech therapy domain knowledge as explicit decision rules and validated templates, ensuring clinical reliability and traceability; and (2) a zero-shot LLM-based approach (GPT-4) aimed at more fluent and concise output. Both systems use identical pre-extracted, expert-validated structured variables, enabling a controlled factual comparison. Outputs were evaluated by eight speech therapists and final-year students using a nine-criterion questionnaire. Results reveal a clear trade-off between clinical reliability and linguistic quality. The template-based system scored higher on fluidity, coherence, and results presentation, while GPT-4 produced more concise output. Directional differences are consistent across evaluation dimensions, though no comparison reached statistical significance after correction, reflecting the scale constraints of expert clinical evaluation. Based on evaluator feedback, we derive eight design recommendations for clinical reporting systems in remote rehabilitation settings. More broadly, this work contributes a replicable methodology combining expert elicitation, taxonomy-driven generation, and multi-dimensional human evaluation for clinical NLG in low-resource settings, and illustrates how controlled comparisons can inform the responsible adoption of generative AI in healthcare.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper compares a rule-based template system encoding speech therapy knowledge against a zero-shot GPT-4 approach for automated clinical report generation from avatar-guided cognitive remediation sessions in low-resource settings with no reference reports. Both systems receive identical pre-extracted, expert-validated structured variables. Eight evaluators (speech therapists and students) rate outputs on a nine-criterion questionnaire, revealing directional patterns where templates score higher on fluidity, coherence, and results presentation while GPT-4 is more concise; no differences reach significance after correction. The work derives eight design recommendations and presents a replicable methodology combining expert elicitation, taxonomy-driven generation, and multi-dimensional human evaluation.

Significance. If the directional patterns hold under stronger evaluation, the work offers a practical, controlled comparison methodology for clinical NLG in healthcare domains with limited data and expert time. The explicit use of identical inputs, derivation of design recommendations from evaluator feedback, and focus on low-resource constraints provide actionable guidance for responsible AI adoption in remote rehabilitation, even if the current evidence base remains preliminary.

major comments (3)
  1. [Results / Abstract] Results section and abstract: The claim of a 'clear trade-off between clinical reliability and linguistic quality' is not supported by the reported statistics. The paper states that no comparison reaches significance after correction for multiple tests, and the evaluator sample is only eight; this renders the directional patterns too weak to ground a reliable trade-off conclusion or the subsequent design recommendations.
  2. [Evaluation / Methods] Evaluation methodology (described in abstract and methods): Without reference reports or objective accuracy metrics, the nine-criterion questionnaire can only capture surface properties (fluidity, conciseness) rather than clinical reliability, factual completeness, or utility against ground truth. The assumption that pre-extracted structured variables already encode everything needed for complete reports is stated but not tested against raw session data, weakening the 'controlled factual comparison' claim.
  3. [Discussion] Discussion of design recommendations: The eight recommendations are derived from feedback by the same small evaluator pool whose ratings show no significant differences; this circularity limits the strength of the recommendations as evidence-based guidance for clinical reporting systems.
minor comments (1)
  1. [Abstract / Methods] The abstract and methods could more explicitly state the exact nine criteria and how they map to 'clinical reliability' versus 'linguistic quality' to improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their detailed and constructive feedback. We agree that the small evaluator sample and lack of statistical significance require careful framing of our claims. We will revise the manuscript to tone down the language around the 'trade-off' and emphasize the preliminary nature of the results and recommendations. Below we address each major comment.

read point-by-point responses

  1. Referee: [Results / Abstract] Results section and abstract: The claim of a 'clear trade-off between clinical reliability and linguistic quality' is not supported by the reported statistics. The paper states that no comparison reaches significance after correction for multiple tests, and the evaluator sample is only eight; this renders the directional patterns too weak to ground a reliable trade-off conclusion or the subsequent design recommendations.

    Authors: We appreciate this point and acknowledge that with only eight evaluators and no significant differences after correction, the evidence for a trade-off is directional rather than statistically robust. The term 'clear trade-off' in the abstract overstates the findings. We will revise the abstract and results section to state that 'directional patterns suggest a potential trade-off between clinical reliability and linguistic quality, though these did not reach statistical significance after multiple comparison correction, consistent with the constraints of expert evaluation in low-resource settings.' This better reflects the data while preserving the observed trends. The design recommendations are presented as preliminary insights from this study. revision: partial

  2. Referee: [Evaluation / Methods] Evaluation methodology (described in abstract and methods): Without reference reports or objective accuracy metrics, the nine-criterion questionnaire can only capture surface properties (fluidity, conciseness) rather than clinical reliability, factual completeness, or utility against ground truth. The assumption that pre-extracted structured variables already encode everything needed for complete reports is stated but not tested against raw session data, weakening the 'controlled factual comparison' claim.

    Authors: The low-resource setting explicitly lacks reference reports, as noted in the title and abstract, making objective metrics infeasible in this study. The structured variables were extracted and validated by experts to control for factual content, allowing comparison of generation approaches on the same input. The questionnaire was designed with input from speech therapy experts and includes items targeting clinical aspects such as 'clinical reliability', 'coherence', and 'results presentation', rated by practicing therapists and students. While we agree this is subjective and surface-level to some extent, expert human judgment is the standard for evaluating clinical utility in such domains without ground truth. We did not test against raw session data because the focus was on report generation from structured variables; we will add clarification in the methods section that this assumption is based on expert validation of the variables. Future work could compare to raw data if feasible. revision: partial

  3. Referee: [Discussion] Discussion of design recommendations: The eight recommendations are derived from feedback by the same small evaluator pool whose ratings show no significant differences; this circularity limits the strength of the recommendations as evidence-based guidance for clinical reporting systems.

    Authors: The recommendations stem from both the quantitative ratings and the qualitative open-ended feedback provided by the evaluators. Although the sample is small, the feedback highlighted specific issues like the need for better handling of edge cases in templates and conciseness in LLMs. We recognize the potential circularity and will revise the discussion to present the recommendations as 'preliminary design insights derived from this initial evaluation' and recommend larger-scale validation studies to strengthen them. revision: partial

Circularity Check

0 steps flagged

No circularity: direct empirical comparison via human ratings with no derivations or self-referential reductions

full rationale

The paper is an empirical study that compares two report-generation systems (rule-based templates vs. zero-shot GPT-4) on identical pre-extracted variables, then collects human ratings from eight evaluators on a nine-criterion questionnaire. No equations, first-principles derivations, fitted parameters, or predictions appear in the described methodology or results. All claims rest on observed rating patterns rather than any quantity defined in terms of itself or reduced via self-citation to the authors' prior inputs. This matches the default case of non-circular empirical work; the evaluation design may have other limitations (small sample, lack of ground-truth references), but those do not constitute circularity under the specified criteria.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The study is an empirical comparison and introduces no mathematical models, free parameters, or new postulated entities; it relies on standard domain knowledge for template construction and off-the-shelf LLM usage.

pith-pipeline@v0.9.0 · 5581 in / 1241 out tokens · 74842 ms · 2026-05-08T09:58:38.157110+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [1]

    Ehud Reiter, Roma Robertson, and Liesl M

    ISBN 0521620368. Ehud Reiter, Roma Robertson, and Liesl M. Osman. Lessons from a failure: Generating tailored smoking cessation let- ters.Artificial Intelligence, 144(1):41–58, 2003. ISSN 0004-3702. doi:https://doi.org/10.1016/S0004-3702(02)00370-

    work page doi:10.1016/s0004-3702(02)00370- 2003

  2. [2]

    François Portet, Ehud Reiter, Albert Gatt, Jim Hunter, Somayajulu Sripada, Yvonne Freer, and Cindy Sykes

    URLhttps://www.sciencedirect.com/science/article/pii/S0004370202003703. François Portet, Ehud Reiter, Albert Gatt, Jim Hunter, Somayajulu Sripada, Yvonne Freer, and Cindy Sykes. Automatic generation of textual summaries from neonatal intensive care data.Artificial Intelligence, 173(7):789–816, 2009. ISSN 0004-3702. doi:https://doi.org/10.1016/j.artint.200...

    work page doi:10.1016/j.artint.2008.12.002 2009

  3. [3]

    URL https://www.sciencedirect.com/science/ article/pii/S2950162823000073

    doi:https://doi.org/10.1016/j.metrad.2023.100007. URL https://www.sciencedirect.com/science/ article/pii/S2950162823000073. Chunyu Liu, Yongpei Ma, Kavitha Kothur, Armin Nikpour, and Omid Kavehei. Biosignal copilot: Leveraging the power of llms in drafting reports for biomedical signals.medRxiv, 2023. doi:10.1101/2023.06.28.23291916. URL https://www.medrx...

    work page doi:10.1016/j.metrad.2023.100007 2023

  4. [4]

    doi:https://doi.org/10.1016/j.jbi.2026.104997

    ISSN 1532-0464. doi:https://doi.org/10.1016/j.jbi.2026.104997. URL https://www.sciencedirect.com/ science/article/pii/S1532046426000213. Rohit V oleti, Julie M. Liss, and Visar Berisha. A review of automated speech and language features for assessment of cognitive and thought disorders.IEEE Journal on Selected Topics in Signal Processing, 14(2):282–298, February

    work page doi:10.1016/j.jbi.2026.104997 2026

  5. [5]

    doi:10.1109/JSTSP.2019.2952087

    ISSN 1932-4553. doi:10.1109/JSTSP.2019.2952087. Benoît Crabbé and Marie Candito. Expériences d’analyse syntaxique statistique du français. In Frédéric Béchet and Jean-Francois Bonastre, editors,Actes de la 15ème conférence sur le Traitement Automatique des Langues Naturelles. Articles longs, pages 161–170, Avignon, France, June 2008. ATALA. URLhttps://acl...

    work page doi:10.1109/jstsp.2019.2952087 1932

  6. [6]

    Retrouvez votre chemin

    Exo 1“Retrouvez votre chemin” [Find your way]

  7. [7]

    Objets où êtes-vous ?

    Exo 2“Objets où êtes-vous ?” [Objects, Where are You?]

  8. [8]

    Que d’accros

    Exo 3“Que d’accros” [This Story is Full of Blanks]

  9. [9]

    Jeux de blasons

    Exo 4“Jeux de blasons” [Blazon game]

  10. [10]

    Mettez de l’ordre dans ces comptes

    Exo 5“Mettez de l’ordre dans ces comptes” [The Right Count]

  11. [11]

    Garçon SVP

    Exo 6“Garçon SVP” [Restaurant]

  12. [12]

    Menez l’enquête

    Exo 7“Menez l’enquête” [Carry out the investigation]

  13. [13]

    Tour Hanoï

    Exo 8“Tour Hanoï” [Towers of Hanoi] As described in Section 3, sessions for young and senior participants comprised eight exercises, each repeated once. For MCI participants, the number of exercises was reduced to four (Exo 1, 2, 3, and 7), each repeated once, to accommodate the difficulties of their involvement. Instructions were provided before each exe...

    2024