Pith. sign in

REVIEW 5 major objections 8 minor 62 references

Completing A Systematic Review in Hours instead of Months with Interactive AI Agents

T0 review · 5 major / 8 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A clinician steering an interactive AI agent can complete a systematic review in about 1.5 hours and reach 79.7 percent of human-written quality.

desk verdict Impressive screening and a genuinely useful system, but the '79.7% of human quality' headline overclaims—the rubric scores an extended abstract, not a systematic review. read the letter →

arxiv 2504.14822 v2 pith:AQPIQWD3 submitted 2025-04-21 cs.HC cs.CL

Rui Qiu , Shijie Chen , Yu Su , Po-Yin Yen , Han-Wei Shen This is my paper · ORCID
classification cs.HCcs.CL
keywords systematicreviewlargelanguagemodelsinteractiveAIagentshuman-in-the-loopvisualanalyticsevidencesynthesisrecordscreeningmulti-agentsystem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InsightAgent combines a semantic map of the literature, parallel LLM agents that read and synthesize each cluster, and a visual interface that lets a domain expert steer the agents in real time. The paper's central claim is that this combination lets a single clinician complete a systematic review in roughly 1.5 hours rather than months, reaching 79.7% of human-written quality. In user studies with 9 medical professionals and 15 published biomedical systematic reviews, letting the expert interact with the agent raised generated-review quality by 27.2% (p = 3.43e-7) and user satisfaction by 34.4% (p = 1.89e-6), and interactive screening reached 98.5% recall and 88.2% F1. If true, it would matter because record screening and evidence synthesis are the main bottlenecks of evidence-based medicine, and the result suggests human oversight rather than full automation is the key to usable AI-generated reviews.

What carries the argument

The load-bearing mechanism is the relevance-and-similarity map (RSS map) combined with multi-agent partition and live human steering. The RSS map places articles in a plane where relevance to the research question grows toward the center and semantically similar articles form nearby clusters; K-means then splits the corpus into about nine clusters, each assigned to an agent that reads from the center outward, selecting among the eight nearest neighbors at each step. New evidence is folded into prior memory through the merge $M_{k+1}=f(M_k,S_j)$, and a dependency tree records which article summaries support each interim synthesis. Users can drag agents to missed articles (path navigation), issue natural-language directives (chat navigation), or edit criteria and summarization requirements (instruction navigation); each intervention triggers a reflection pass that updates the reading strategy and reconciles conflicted memory. The provenance tree is what makes the output auditable, and the receptive-field rule is what keeps exploration systematic.

What would settle it

Run the same 15 reviews through a preregistered, blinded study in which independent raters evaluate system outputs without knowing which system produced them, and report per-item agreement between raters; if the 27.2% interaction gain and the 79.7% figure shrink to noise, the central claim collapses. Separately, compare each reconstructed PubMed search's retrieved set against the published PRISMA flow diagram to check corpus fidelity.

Watch

Extended reading notes

Core claim

The discovery is that a human-in-the-loop agent architecture, rather than a fully autonomous LLM pipeline, is what lifts generated systematic reviews toward usable quality. InsightAgent maps the corpus onto a radial relevance-and-similarity layout, partitions it into semantic clusters, and dispatches an agent per cluster; each agent reads from the center outward with a constant receptive field of eight neighbors, stores summaries in a local memory that is merged incrementally, and records every merge in a provenance tree. Expert users can intervene by dragging an agent's path, chatting natural-language directives, or editing inclusion and summarization parameters, after which the agent reflects and reconciles its memory. Across 15 reconstructed biomedical corpora, the interactive GPT-4o variant reached 98.5% recall and 88.2% F1 in screening, scored 79.7/100 on a rubric where human reviews anchor at 100, and improved on its autonomous counterpart by 27.2% in quality and 34.4% in satisfaction; the same result holds qualitatively, with experts reporting more comprehensive, traceable, and sometimes more detailed syntheses than the human-written reference.

Load-bearing premise

The central comparison assumes the 100-point expert rating scale really measures review quality even though the two raters were not blind to which system made each review and the paper never reports whether the raters agreed with each other; it also assumes the 15 rebuilt search results match the original reviews' screening decisions.

Editorial extensions

If this is right

  • A single domain expert, not a team, can produce a systematic review in about 1.5 hours with quality near human output, sharply lowering the months-long cost of evidence synthesis.
  • Human interaction is the decisive factor: disabling it drops generated-review quality by 27.2%, so fully autonomous LLM summarizers are the wrong target for this task.
  • Near-perfect recall (98.5%) in record screening means relevant studies are rarely missed when an expert can correct the agent's path.
  • Traceable provenance improves expert confidence: users reported that visualizing exploration paths and checking evidence trees made them trust the output.
  • The system's remaining gaps — abstract-only reading, no pooled statistics or evidence weighting — define the next step toward closing the 20-point gap to human-written quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same experts rated both conditions and were not blind to which output came from InsightAgent, part of the observed 27.2% gain could reflect expectations about interactivity; a blinded replication would test this.
  • If the effect is real, the interaction benefit may transfer to other high-stakes synthesis tasks — clinical guidelines, policy reviews, regulatory dossiers — wherever a domain expert can spot-check an agent's reading path.
  • The corpus reconstruction step deserves validation: comparing each reconstructed PubMed search's retrieved set against the original review's PRISMA flow diagram would test whether the 15 corpora are faithful test beds.
  • A natural next experiment is an ablation separating visualization-only, chat-only, and path-only conditions with a larger expert sample, to identify which interaction channel carries the quality gain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. This paper presents InsightAgent, a human-in-the-loop LLM-agent system for biomedical literature review. InsightAgent maps a retrieved corpus into a radial relevance-and-similarity layout (RSS map), partitions it with K-means, dispatches parallel agents to screen titles/abstracts and incrementally synthesize evidence, and exposes a provenance/dependency tree so users can verify claims. Users can intervene through path navigation (dragging an agent to an article), chat, and direct instruction, with the agent entering a reflection phase after each intervention. The evaluation covers 15 published systematic reviews and 9 medical experts, reporting that InsightAgent (GPT-4o) reaches 98.5% recall and 88.2% F1 in record screening; that interaction improves generated-review quality by 27.2% (p = 3.43e-7) and user satisfaction by 34.4% (p = 1.89e-6); and that the interactive system scores 79.7 out of 100 rubric points, characterized as 79.7% of human-written quality, in sessions averaging about 1.5 hours.

Significance. The system contribution is substantial: the multi-agent partitioning is compared against a single-agent variant (Appendix B); the provenance-tree mechanism directly addresses the untraceable-summary problem identified for ChatCite and AutoSurvey; code and data are publicly released; and the evaluation against 15 real published reviews with domain experts is far more grounded than typical LLM-summarization evaluations. The screening results, particularly the interactive 98.5% recall, are striking, and Appendix I's error taxonomy (quantitative evidence, evidence weighting, heterogeneity, faithfulness) is a useful analytic contribution. However, the significance as currently framed depends on the claim that the system completes a high-quality systematic review, and the measured outcome — an extended-abstract rubric score anchored on human abstracts — does not support that construct. With a re-scoped claim (abstract-level evidence synthesis) and additional measurement reporting (inter-rater reliability, corpus-reconstruction validation), the paper would be a strong contribution to human-centered AI for evidence synthesis.

major comments (5)
  1. [Section 4.1; Appendix E; Appendix H; Limitations] The abstract's central claim that a clinician can 'complete a high-quality systematic review' in about 1.5 hours at 79.7% of human-written quality is not supported by the paper's own measurement instrument. Appendix E states that the rubric is tailored to 'agent-based summarization of a biomedical systematic review in extended-abstract format'; Category 2 item 2.1 ('Agent-Based Summarization Description') is definitionally inapplicable to a human-written review; and the human reference used in the worked example (Appendix H, Table 14) is the published abstract of Tan-Lim and Esteban-Ipac, not the completed systematic review. The Limitations section concedes that only titles and abstracts are read, that the system cannot extract or synthesize numerical statistics or effect sizes, cannot perform meta-analysis, and does not weight evidence by study design — each a core component of a high-quality healthcare systematic review. The 79.7% figure is therefore a quality score for extended abstracts against an abstract-level anchor, and neither the abstract's nor Section 5's 'complete a high-quality systematic review' claim follows from it. The authors should either re-frame the headline claims to abstract-level evidence synthesis support, or validate the rubric against full systematic reviews and report scores for the human references on the same rubric.
  2. [Section 4.1; Section 4.2.3; Table 3] The statistical claims rest on an unreported measurement foundation. Each review is 'independently scored by two experts,' but the paper reports no inter-rater reliability (Cohen's kappa or per-item agreement) and no rater calibration; the paired t-test p-values (3.43e-7 for quality, 1.89e-6 for satisfaction) cannot be assessed without knowing the agreement structure or whether the paired observations are per-review, per-rater, or per-session. The same nine users who operated the systems also rated the outputs, with no stated blinding to system identity or to the auto/interactive condition, and no reported counterbalancing of presentation order. Additionally, Table 3 reports interface-usability Likert items (e.g., drag-and-drop, citation tracing) for the InsightAgentauto condition, in which the interactive interface is not in use; it is unclear what participants were rating in that condition. Please report inter-rater statistics, the blinding and counterbalancing protocol (or explicitly acknowledge its absence), and a precise description of the autonomous-condition questionnaire.
  3. [Section 4.1; Appendix D; Table 6] The reconstructed corpora are not validated against the published search strategies. Appendix D states that each corpus was replicated from PubMed, but the 15 reviews include Cochrane reviews (e.g., Hyun et al., Wu et al., Sharrad et al.) whose published searches cover multiple databases and trial registries; a PubMed-only reconstruction will likely omit records those reviews screened. The paper does not report what fraction of each published review's included studies is present in the reconstructed corpus, and Table 6 shows inclusion rates as low as 0.37% (Wu et al.), so even a small reconstruction gap can cap the achievable recall. If the ground-truth included set is not fully present, the reported 98.5% interactive recall is an artifact of corpus construction rather than a measure of screening ability; if the corpus contains extra irrelevant records, reported precision reflects reconstruction noise. Please report, per review, the coverage of the published included set and a sensitivity analysis of the screening metrics to reconstruction completeness.
  4. [Section 4.1; Conclusion; Abstract] The '1.5 hours versus months' comparison conflates session length with the duration of the claimed workflow. Section 4.1 reports that participants finished a session in 1.5 hours while 'no time limit' was set, and the session produces an extended abstract from a pre-retrieved title/abstract corpus. The months-long human pipeline being compared against includes full-text screening, dual independent review, risk-of-bias assessment, and quantitative synthesis, all of which the Limitations state the system does not perform. The time comparison should be scoped to the synthesis stage on a provided corpus, and the measured time should be decomposed into corpus preparation, screening, interaction, and report generation, with the comparison made against the corresponding stages of the manual process.
  5. [Section 4.2.4, Figure 4; Table 5] The units of analysis in the interaction and error analyses are not derivable from the stated study design. Section 4.2.4 says 50 interim-synthesis pairs were sampled per interaction type (150 pairs total), but the study involves 9 users and 15 reviews, and it is unclear where the 150 pairs come from, who scored them with the rubric, and how the before/after scores in Figure 4 relate to the two-expert rubric used in Table 2. Likewise, Table 5 reports R=59 for InsightAgent (GPT-4o) and R=30 for InsightAgent (Llama-3.3), while the stated design (15 reviews × 2 raters) yields 30 reports per condition; the provenance of R=59 should be explained so that the error rates in Appendix I are auditable.
minor comments (8)
  1. [Section 4.2.1 (text near Table 1)] The sentence reporting precision as '(62.4%/41.9% vs 20.4%)' uses 41.9%, which matches no entry in Table 1; the GPT-4o autonomous precision is 51.9%.
  2. [Section 3.2] The sentence 'We report the details of our interface design in appendix 6' should reference Appendix C instead.
  3. [Section 3.3] The sentence 'We provide an example final synthesis template in Appendix A' is inaccurate: Appendix A contains the four operational prompts (Retrieve, Read, Synthesize, Reflect), and the final-synthesis template appears only inside the Synthesize prompt; please correct the cross-reference.
  4. [Figure 3 caption] The caption reads 'LLamA 3.3 70B' and should read 'Llama 3.3 70B'.
  5. [Appendix E] The sentence 'To make is suitable for our evaluation setting' contains a typo; it should read 'To make it suitable.'
  6. [Table 15 vs. Table 14] The InsightAgent example report states that 11 studies met the inclusion criteria while the human review included 9; this discrepancy should be discussed as an inclusion-fidelity issue rather than left to the evaluator's comment in Table 19.
  7. [Introduction; Section 4.2.2] The claim of a '47% (F1 points)' improvement in article identification is misleading: the ratio 88.2/60.0 is a 47% relative increase in F1, not 47 F1 points (the absolute gain is 28.2 points); please state the metric unambiguously.
  8. [Section 4.2.2] The statement that InsightAgent is 'the first system to be practically useful for domain experts' is a qualitative judgment not established by the reported experiments; please remove it or support it with the scoped evidence.

Circularity Check

1 steps flagged · score 6.0 of 10

The 79.7%-of-human-quality headline is normalized by an assumed 100-point human anchor on a rubric tailored to agent-generated extended abstracts; screening and interaction gains remain independent.

  1. self definitional [Abstract; Section 4.1 (Systematic Review Quality); Appendix E (Evaluation Rubric); Appendix H Table 14]
    "The rubric is collaboratively defined by domain experts, with peer-reviewed human systematic reviews as the ground truth (100 points). ... To make is suitable for our evaluation setting, we tailor it to agent-based summarization of a biomedical systematic review in extended-abstract format."

    The headline 'reaching 79.7% of human-written quality' is computed as the system's rubric score divided by 100, where 100 is assigned to human reviews by fiat rather than measured. Appendix E explicitly tailors the rubric to agent-based extended abstracts (e.g., rewarding 'Agent-Based Summarization Description'), and Appendix H uses the original published abstract as the human reference, not a full systematic review scored on the same rubric. Therefore the denominator is an assumed constant, and the 79.7/100 ratio is the system's own rubric score normalized by a definitional maximum, not an observed comparison to human-written quality. The 27.2% interaction improvement and screening F1 are independent empirical comparisons and do not share this flaw.

full rationale

InsightAgent's record-screening results (Table 1) are compared against the published included-study sets of 15 systematic reviews, and the 27.2% interaction gain is a paired within-system user study; neither is a fitted parameter or a renamed input. The circularity is confined to the '79.7% of human-written quality' claim. Section 4.1 sets human reviews as 'ground truth (100 points)' without scoring them on the same instrument, while Appendix E states the rubric was tailored to agent-based extended-abstract summarization. Thus the human-relative quality ratio is constructed by definition rather than derived from a side-by-side measurement. The self-citations to the authors' RSS map (Qiu et al., 2024) are used for the spatial layout and are not load-bearing for the central quality comparison. Overall, partial circularity: one headline quantity reduces to its own assumed anchor, while the other main results remain empirically grounded.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claims depend on a small number of hand-chosen system parameters (cluster count heuristic, 8-neighbor receptive field, top-100 baseline cutoff) and four domain assumptions that are explicitly stated or clearly embedded in the workflow (layout as reading strategy, abstract-only screening, rubric validity, corpus reconstruction). No new physical entities are postulated. The most consequential unverified assumptions are the validity of the rubric without inter-rater reliability data and the fidelity of the reconstructed corpora.

free parameters (3)
  • K, number of semantic clusters = average of 9 clusters
    Chosen by the Elbow method on each corpus, a data-dependent heuristic that affects how many parallel agents run and therefore screening precision; the paper's own Appendix B shows single-agent (K=1) performs substantially worse.
  • Receptive field size of 8 neighbors = 8
    Hard-coded design choice in Stage 2; the paper states 'we standardize this receptive field to always include eight nearest neighboring articles' without an ablation justifying 8 over other values.
  • Top-100 retrieval cutoff for baselines = 100
    Applied to BM25 and AutoSurvey in Table 1 and justified by the fact that all 15 target reviews include fewer than 100 studies; this is a fair but hand-chosen baseline constraint.
assumptions (4)
  • domain assumption The RSS map's relevance-preserving spatial layout is a valid proxy for reading order such that starting at the center and moving outward improves screening accuracy.
    Invoked in Stage 1 and Stage 2 of Section 3; the agents' entire navigation strategy assumes that layout relevance correlates with inclusion relevance, which is plausible but not independently validated.
  • domain assumption Title and abstract text suffice to judge inclusion for systematic-review screening.
    Stated in Section 4.1; the paper acknowledges in Limitations that full-text omission may omit critical details. This is a known and contested assumption in real systematic reviews.
  • ad hoc to paper A 100-point rubric produced with domain experts and grounded in PRISMA is a valid measure of systematic-review quality.
    Invoked in Section 4.1 and Appendix E. The rubric is tailored to 'agent-based summarization in extended-abstract format', so it is partly specific to this task, and no inter-rater reliability evidence is provided.
  • domain assumption Published search strategies can be faithfully reconstructed from the 15 chosen papers to reproduce their corpora.
    Assumed in Section 4.1 and Appendix D; corpus sizes and included counts are listed, but no overlap statistic against the original PRISMA flow diagrams is reported.
invented entities (1)
  • Provenance/dependency tree as an evidence-tracking structure
    purpose: Makes each claim in the generated review traceable to article-level summaries or intermediate syntheses, replacing the untraceable citations of baselines.
    This is a software design element, not a physical or mathematical entity. It has no independent falsifiable handle outside the system, and its effectiveness is only measured through the author-designed rubric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Completing A Systematic Review in Hours instead of Months with Interactive AI Agents." pith.science (2026). https://pith.science/paper/AQPIQWD3

@misc{pith2026250414822,
  author       = {Pith},
  title        = {Pith review of: Completing A Systematic Review in Hours instead of Months with Interactive AI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AQPIQWD3}},
  note         = {Machine review of arXiv:2504.14822}
}
read the original abstract

Systematic reviews (SRs) are vital for evidence-based practice in high stakes disciplines, such as healthcare, but are often impeded by intensive labors and lengthy processes that can take months to complete. Due to the high demand for domain expertise, existing automatic summarization methods fail to accurately identify relevant studies and generate high-quality summaries. To that end, we introduce InsightAgent, a human-centered interactive AI agent powered by large language models that revolutionize this workflow. InsightAgent partitions a large literature corpus based on semantics and employs a multi-agent design for more focused processing of literature, leading to significant improvement in the quality of generated SRs. InsightAgent also provides intuitive visualizations of the corpus and agent trajectories, allowing users to effortlessly monitor the actions of the agent and provide real-time feedback based on their expertise. Our user studies with 9 medical professionals demonstrate that the visualization and interaction mechanisms can effectively improve the quality of synthesized SRs by 27.2%, reaching 79.7% of human-written quality. At the same time, user satisfaction is improved by 34.4%. With InsightAgent, it only takes a clinician about 1.5 hours, rather than months, to complete a high-quality systematic review.

Figures

Figures reproduced from arXiv: 2504.14822 by the authors.

Figure 1
Figure 1. Overview of the InsightAgent workflow. In Stage 1, the corpus is mapped into semantic clusters. In Stage 2, multiple agents concurrently read and synthesize evidence under real-time user guidance for each cluster. Finally, in Stage 3, findings of all agents are integrated into a complete systematic review. user oversight using visualization and the incorpo￾ration of user expertise via interactions. As illus￾trated i… view at source ↗
Figure 2
Figure 2. Detailed comparison of summarization quality of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Relationship between the number of human [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Impact of interaction types on summary quality improvement across different evaluation perspectives. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The number of interactions users conducted on [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: A screenshot of the InsightAgent interface while conducting a systematic review. The central Canvas is an infinite- scrollable space for creating multiple Environments (E1, E2, E3) and attaching any number of agents or collaboration panels. This design allows users to …
Figure 7
Figure 7. Figure 7: An integrated Synthesis View in InsightAgent in an early stage of review, featuring two agents collaborating on the same systematic review. The Hierarchy panel (left) merges each agent’s memory structure into a single visualization, color-coded to indicate which concep…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 60 canonical work pages

  1. [1]

    Use of Quantitative or Specific Evidence

    Insufficient Statistical Analysis & Quantita- tive Evidence This error occurs when a summary omits concrete numerical synthesis (e.g., pooled effect sizes or incidence rates) that human review- ers would compute across studies. Detection is triggered if either the rubric subscore on item 3.3 (“Use of Quantitative or Specific Evidence”) is≤ 2, or an expert...

    work page

  2. [2]

    Coverage & Representativeness

    Limited Evidence Weighting This error arises when the summary treats all studies equally, without accounting for design or quality differences (e.g., RCT vs. observational). We flag a report when rubric subscores on item 2.3 (“Coverage & Representativeness”) or 2.5 (“Risk of Bias”) are ≤ 2, or an expert comment highlights uniform weighting. InsightAgent v...

    work page

  3. [3]

    Applied Sciences, 12(15):7515

    Autoelbow: An automatic elbow detection method for estimating the number of clusters in a dataset. Applied Sciences, 12(15):7515. Emre Oral, Ria Chawla, Michel Wijkstra, Narges Mah- yar, and Evanthia Dimara. 2023. From information to choice: A critical inquiry into visualization tools for decision making. IEEE Transactions on Visual- ization and Computer ...

    work page 2023

  4. [4]

    IEEE Transactions on Visualization and Computer Graphics, 30(2):1533–1548

    Vadis: A visual analytics pipeline for dynamic document representation and information seeking. IEEE Transactions on Visualization and Computer Graphics, 30(2):1533–1548. Rui Qiu, Yamei Tu, Yu-Shuen Wang, Po-Yin Yen, and Han-Wei Shen. 2022. Docflow: A visual analytics system for question-based document retrieval and categorization. IEEE Transactions on Vi...

    work page 2022

  5. [5]

    European Respiratory Review, 33(171)

    The prevalence of pulmonary hypertension in post-tuberculosis and active tuberculosis populations: a systematic review and meta-analysis. European Respiratory Review, 33(171). Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, et al. 2024. Autosur- vey: Large language models can automatica...

    arXiv 2024

  6. [6]

    Nutrients, 15(1):148

    Effects of caffeine intake on endurance run- ning performance and time to exhaustion: A system- atic review and meta-analysis. Nutrients, 15(1):148. Chuncheng Wu, Ping Zhao, Ping Xu, Chaomin Wan, Surjit Singh, Shoban Babu Varthya, and Shuang- Hong Luo. 2024. Evening versus morning dosing regimen drug therapy for hypertension. Cochrane Database of Systemat...

    work page 2024

  7. [7]

    Hierarchical Map – Agent’s Synthesis Process • How clear was the agent’s synthesis process as displayed in the Hierarchical Map? Scale: (1 = Not clear at all, 5 = Very clear) Table 12: Detailed Questionnaire Detailed Questionnaires after User Study Category 4: Ability to Guide or Correct Agents These questions measure how effectively participants can dete...

    work page

  8. [8]

    excess of irrelevant evidence

    (Table 14). During the user study, partic- ipants leveraged 15 path navigation, 24 chat-based interactions, and 1 instruction navigation to refine the generated summary. Comparison with Human-Curated Summary. Table 15, 16, 17 present the detailed evaluations based on our evaluation rubric (Appendix E). The evaluation highlights that InsightAgent-produced ...

    work page

Show all 62 references
  1. [9]

    ( more conclusive result should be made ) The drug has a generally acceptable safety profile, though concerns about nephrotoxicity necessitate careful monitoring

    Conclusion and Future Directions In summary, remdesivir has demonstrated some benefits in reducing hospitalization duration, but its impact on mortality remains limited. ( more conclusive result should be made ) The drug has a generally acceptable safety profile, though concer...

    2024

  2. [11]

    Variability / Heterogeneity

    Omitted Heterogeneity Discussion A report is flagged if it fails to acknowledge conflicting or subgroup findings (rubric item 3.4 “Variability / Heterogeneity”≤ 2 or expert notes missing sub- group analysis). Baseline systems omit heterogene- ity in 87% (ChatCite) and 67% (Aut...

  3. [12]

    Faithfulness to Source Material

    Hallucinations or Faithfulness Errors This error covers any invented or misrepresented facts. We flag a report if rubric item 3.5 (“Faithfulness to Source Material”) is ≤ 2 or an expert identi- fies a discrepancy. Hallucinations occur in 53% of ChatCite and 40% of AutoSurvey s...

  4. [13]

    If none are relevant, clearly state why

    Identify the most relevant existing summary or synthesis to merge with this new paper. If none are relevant, clearly state why

  5. [14]

    If it is an existing synthesis, add the new paper’s insights to that synthesis

    If a relevant item is another paper summary, combine them into a new intermediate synthesis. If it is an existing synthesis, add the new paper’s insights to that synthesis. If no match is found, explicitly explain that outcome

  6. [15]

    Incorporate the user’s summarization requirement{summarization_requirement} into the final integrated summary if it applies to the content

  7. [16]

    • Study Design: Describe how the included summaries were chosen

    Structure the final synthesized summary (plain text inside each section) with the following sections, and cite sources using<citation>citation_number</citation>: ====================== • Introduction: Introduce the research question{query} and its broader context, explaining i...

  8. [17]

    identified_relevant_summaries

    Only use information provided in these summaries or syntheses; do not introduce details beyond the given text. Retain any citations from the identified synthesis if you are updating it. After deciding which summaries or syntheses to include, return a single JSON object followi...

    2024

  9. [18]

    Radial Chart – Search & Navigation • How easy was it to locate the target article using the search function? Scale: (1 = Very difficult, 5 = Very easy)

  10. [19]

    Hierarchical Map – Exploration • How easy was it to explore and understand the synthesis process? Scale: (1 = Very difficult, 5 = Very easy)

  11. [20]

    Hierarchical Map – Tracing Citations • How easy was it to trace the citation path for the summary? Scale: (1 = Very difficult, 5 = Very easy)

  12. [21]

    Hierarchical Map – Locating Specific Evidence • How easy was it to locate specific evidence within the Hierarchical Map? Scale: (1 = Very difficult, 5 = Very easy) Category 2: Confidence in Recommendations These questions measure the participant’s trust in the system’s outputs...

  13. [22]

    Radial Chart – Agent’s Progress • How confident are you in your understanding of the agent’s progress? Scale: (1 = Not confident at all, 5 = Very confident)

  14. [23]

    Hierarchical Map – Source Identification • How confident are you that you identified all the original sources correctly? Scale: (1 = Not confident at all, 5 = Very confident)

  15. [24]

    Hierarchical Map – Agent Incorporating Feedback • How confident are you that the agent will incorporate your feedback into future summaries? Scale: (1 = Not confident at all, 5 = Very confident)

  16. [25]

    Hierarchical Map – Synthesis Accuracy • How confident are you that the agent is synthesizing the correct information? Scale: Scale: (1 = Not confident at all, 5 = Very confident) Category 3: Visualization-Aided Understanding These items assess how well the visual interfaces (R...

  17. [26]

    Radial Chart – Cluster Clarity • How clear were the cluster boundaries on the Radial Chart? Scale: (1 = Not clear at all, 5 = Very clear)

  18. [27]

    Radial Chart – Topic Identification • How easy was it to understand the main topics of each cluster? Scale: (1 = Very difficult, 5 = Very easy)

  19. [28]

    Radial Chart – Node Information • How clear was the information provided when hovering over the article node? Scale: (1 = Not clear at all, 5 = Very clear)

  20. [29]

    Radial Chart – Topic Coverage • How comprehensive was the Radial Chart in displaying topic coverage? Scale: (1 = Not comprehensive at all, 5 = Very comprehensive)

  21. [30]

    Radial Chart – Missing Areas • How easy was it to identify missing areas in the agent’s coverage? Scale: (1 = Very difficult, 5 = Very easy)

  22. [31]

    Hierarchical Map – Synthesis Structure • How clear was the hierarchical structure of the evidence synthesis? Scale: (1 = Not clear at all, 5 = Very clear)

  23. [33]

    Radial Chart – Path Correction • How easy was it to identify incorrect paths on the Radial Chart? Scale: (1 = Very difficult, 5 = Very easy)

  24. [34]

    Radial Chart – Drag-and-Drop • How intuitive was the drag-and-drop functionality for adjusting paths? Scale: (1 = Not intuitive at all, 5 = Very intuitive)

  25. [35]

    Hierarchical Map – Missing Citations • How easy was it to identify missing citations or incorrect paths? Scale: (1 = Very difficult, 5 = Very easy)

  26. [36]

    Hierarchical Map – Feedback on Summaries • How intuitive was the feedback process within the Hierarchical Map? Scale: (1 = Not intuitive at all, 5 = Very intuitive) Category 5: Overall Satisfaction These questions capture general satisfaction and user impressions of how well t...

  27. [37]

    Overall System Trustworthiness • To what extent do you trust the generated systematic review? Scale: (1 = Not confident at all, 5 = Very confident) Table 13: Detailed Questionnaire CONT. Table 14: Report generated byHuman (original report)on Systematic review for Remdesivir be...

    2024

  28. [38]

    Among the antiviral treatments explored, remdesivir emerged as a promising candidate due to its mechanism of action as an RNA-dependent RNA polymerase (RdRp) inhibitor

    Introduction The COVID-19 pandemic has driven an urgent search for effective treatment options to mitigate the severe health outcomes associated with SARS-CoV-2 infection. Among the antiviral treatments explored, remdesivir emerged as a promising candidate due to its mechanism...

  29. [39]

    Mechanism of Action of Remdesivir (Irrelevant section) Remdesivir functions by inhibiting the viral RNA polymerase enzyme, which is essential for SARS-CoV-2 replication

  30. [40]

    The WHO Solidarity Trial, one of the largest global studies, found no significant mortality benefit associated with remdesivir use

    Clinical Efficacy of Remdesivir Several studies have evaluated remdesivir’s role in reducing COVID-19 mortality. The WHO Solidarity Trial, one of the largest global studies, found no significant mortality benefit associated with remdesivir use. However, other randomized contro...

  31. [41]

    Safety and Adverse Events The safety profile of remdesivir has been a topic of ongoing investigation, with nephrotoxicity emerging as a significant concern. Causal discovery models analyzing electronic health records suggest that remdesivir may contribute to acute kidney injur...

  32. [42]

    While molnupiravir and Paxlovid offer alternative mechanisms for viral inhibition, their clinical effectiveness varies based on patient demographics and disease severity

    Comparative Effectiveness and Combination Therapies When compared to other antiviral options, remdesivir has demonstrated mixed efficacy. While molnupiravir and Paxlovid offer alternative mechanisms for viral inhibition, their clinical effectiveness varies based on patient dem...

  33. [43]

    Disparities in Treatment and Access (Irrelevant section) Access to remdesivir treatment has not been uniform across different racial and socioeconomic groups

  34. [44]

    Economic and Cost-effectiveness Analysis (Irrelevant section) The cost-effectiveness of remdesivir has been a subject of debate among healthcare policymakers

  35. [45]

    Conflict- ing findings across clinical trials can be attributed to variability in study designs, differences in patient selection criteria, and inconsistencies in outcome measures

    Limitations of Current Evidence Despite extensive research, there are several limitations in the current evidence base regarding remdesivir’s efficacy. Conflict- ing findings across clinical trials can be attributed to variability in study designs, differences in patient selec...

  36. [47]

    • The scope is clear, defining key outcomes: mortality, clinical improvement, mechanical ventilation, and adverse events

    Introduction & Research Context (15 Points) (a) Clarity of Research Question (5/5) • The extended abstract explicitly states the research question: evaluating the efficacy and safety of remdesivir in treating COVID-19 patients. • The scope is clear, defining key outcomes: mort...

  37. [48]

    It also highlights human involvement in modifying agent actions, ensuring higher accuracy and contextual relevance

    Summarization Approach & Method (25 Points) (a) Agent-Based/Automated Summarization Description (5/5) • The abstract effectively demonstrates InsightAgent’s capabilities in selecting and synthesizing relevant studies. It also highlights human involvement in modifying agent act...

  38. [49]

    • Users reported that InsightAgent-generated summaries contained more details than the original human reports while maintaining essential findings

    Comprehensiveness & Accuracy of Summaries (30 Points) (a) Relevance & Completeness (10/10) • All major findings are well-captured: mortality impact, clinical improvement, need for mechanical ventilation, and adverse events. • Users reported that InsightAgent-generated summarie...

  39. [50]

    • One missing point: it could have elaborated more on why remdesivir fails in ventilated patients

    Discussion, Implications & Future Directions (20 Points) (a) Interpretation of Findings (9/10) • InsightAgent provides a good interpretation of when remdesivir works best (non-ventilated patients, early administration). • One missing point: it could have elaborated more on why...

  40. [51]

    • “One thing that surprised us was how InsightAgent made it easy to explore multiple perspectives—something that usually takes weeks of manual reading and synthesizing

    Writing Quality, Fluency & Referencing (10 Points) (a) Readability & Structure (5/5) • Clear and structured writing enhances readability. (b) Referencing & Supporting Evidence (5/5) • References are well-integrated, ensuring credibility. Final Score: 90/100 User Feedback & Dis...

  41. [52]

    • The discussion covers too many broad topics (e.g., healthcare disparities, economic costs) that dilute the focus

    Introduction & Research Context (15 Points) (a) Clarity of Research Question (3/5) • The report mentions remdesivir’s efficacy and safety, but the research question is not explicitly stated. • The discussion covers too many broad topics (e.g., healthcare disparities, economic ...

  42. [53]

    • Unlike InsightAgent, there is no evidence that human users had control over the summarization process

    Summarization Approach & Method (25 Points) (a) Agent-Based/Automated Summarization Description (2/5) • The methodology is not described, leaving the source of included studies unclear. • Unlike InsightAgent, there is no evidence that human users had control over the summariza...

  43. [54]

    reduces hospitalization

    Comprehensiveness & Accuracy of Summaries (30 Points) (a) Relevance & Completeness (5/10) • The report is comprehensive but includes many irrelevant details, such as healthcare equity, economic analysis, and alternative antivirals. • Deduction (-5 points): While complete, much...

  44. [55]

    • While the report mentions remdesivir’s role in different treatment settings, it does not explain why outcomes vary

    Discussion, Implications & Future Directions (20 Points) (a) Interpretation of Findings (6/10) • The interpretation lacks depth and fails to highlight key takeaways for different patient populations. • While the report mentions remdesivir’s role in different treatment settings...

  45. [56]

    There’s no way to verify that the studies included were high quality. The report assumes all findings are equally valid, which is a red flag

    Writing Quality, Fluency & Referencing (10 Points) (a) Readability & Structure (3/5) • The writing is clear but overly verbose, with some sections appearing redundant. • Deduction (-2 points): Information overload reduces readability. (b) Referencing & Supporting Evidence (3/5...

  46. [57]

    • The focus on remdesivir’s effectiveness and safety is evident, but the scope is unclear—it does not specify key outcomes or the target patient population

    Introduction & Research Context (15 Points) (a) Clarity of Research Question (2/5) • The report does not explicitly state a research question but rather presents findings in an unstructured format. • The focus on remdesivir’s effectiveness and safety is evident, but the scope ...

  47. [58]

    • Unlike InsightAgent, which allowed for human refinement and structured summarization, this system appears to have blindly aggregated text

    Summarization Approach & Method (25 Points) (a) Agent-Based/Automated Summarization Description (1/5) • There is no mention of how studies were selected, summarized, or synthesized. • Unlike InsightAgent, which allowed for human refinement and structured summarization, this sy...

  48. [59]

    There is no mention of how studies were selected, summarized, or synthesized

    Summarization Approach & Method (25 Points) (a) Agent-Based/Automated Summarization Description (1/5) i. There is no mention of how studies were selected, summarized, or synthesized. Unlike InsightAgent, which allowed for human refinement and structured summarization, this sys...

  49. [60]

    The report includes key study findings but does not structure them properly, making it difficult to determine which results are most relevant

    Comprehensiveness & Accuracy of Summaries (30 Points) (a) Relevance & Completeness (5/10) i. The report includes key study findings but does not structure them properly, making it difficult to determine which results are most relevant. Critical aspects like study variability a...

  50. [61]

    The report does not provide clear interpretations of the results

    Discussion, Implications & Future Directions (20 Points) (a) Interpretation of Findings (4/10) i. The report does not provide clear interpretations of the results. Findings are listed, but there is little attempt to synthesize the information into meaningful conclusions. ii. D...

  51. [62]

    It’s hard to extract key takeaways because everything is crammed into a single block of text

    Writing Quality, Fluency & Referencing (10 Points) (a) Readability & Structure (1/5) i. The entire report is written as one long paragraph, making it difficult to extract key findings. Deduction (-4 points): The poor structure severely reduces readability. Final Score: 42/100 ...

  52. [2022]

    Multimedia Tools and Applications, 81(11):14803–14830

    Visual analytics for technology and innovation management: An interaction approach for strategic decision making. Multimedia Tools and Applications, 81(11):14803–14830. Adeiza James Onumanyi, Daisy Nkele Molokomme, Sherrin John Isaac, and Adnan M Abu-Mahfouz

  53. [2023]

    line-of-action

    Efficacy of single-pill combination in uncon- trolled essential hypertension: A systematic review and network meta-analysis. Clinical Cardiology, 46(8):886–898. A Prompt Template we present the complete prompt in InsightAgent for retrieve (Prompt 1), read (Prompt 2), synthesis...

    2021

  54. [2024]

    arXiv preprint arXiv:2403.02574

    Chatcite: Llm agent with human workflow guidance for comparative literature summary. arXiv preprint arXiv:2403.02574. Shaheen Majid, Schubert Foo, Brendan Luyt, Xue Zhang, Yin-Leng Theng, Yun-Ke Chang, and Intan A Mokhtar. 2011. Adopting evidence-based practice in clinical dec...

    2011 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.