Pith. sign in

REVIEW 4 major objections 6 minor 69 references

LLM Augmentations to support Analytical Reasoning over Multiple Documents

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs can organize evidence but cannot connect the dots into convincing arguments.

desk verdict A useful negative result and a genuinely new architecture, but the evaluation of 'analytical creativity' rests on unvalidated ground truth and self-referential judging. read the letter →

arxiv 2411.16116 v1 pith:VREDI6DK submitted 2024-11-25 cs.CL cs.AI

Raquib Bin Yousuf , Nicholas Defelice , Mandar Sharma , Shengzhe Xu , Naren Ramakrishnan This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsintelligenceanalysisdynamicevidencetreesanalyticalreasoningretrieval-augmentedgenerationnarrativemarshallingLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Taking three intelligence-analysis dossiers in which analysts must connect scattered reports into a hidden plot, this paper tests whether large language models can do that analysis alone and when augmented with a memory module called dynamic evidence trees (DETs). Across five models and many prompt and temperature settings, the authors find that LLMs reliably summarize and group evidence but rarely produce the speculative, implicit narrative that connects the dots. The DET augmentation (plus data condensation and retrieval) raises the quality of the generated narratives by organizing evidence into evolving investigation threads, but it does not close the creativity gap: the models still miss plots such as a ship carrying dangerous cargo. The paper's conclusion is that current LLMs can serve as evidence marshalling and report-drafting modules for analysts, but not as autonomous analytical reasoners. This matters because, if correct, it tells practitioners where to place LLMs in an intelligence pipeline and where human judgment remains irreplaceable.

What carries the argument

The central mechanism is the Dynamic Evidence Tree (DET), a memory graph whose nodes are 'DOTs'—quadruples of (information, children, parents, source document) that can be either evidential dots (from reports) or hypothesis dots (synthesized from other dots). An LLM repeatedly retrieves similar DOTs by embedding search, filters them, and merges the new dot into the lowest-common-parent hypothesis branch, growing a tree that tracks each investigation thread. Two supporting mechanisms are data condensation (an LLM zero-shot compresses each report into a single clean evidential dot, which the paper shows is important for retrieval) and an LLM-based reranker on top of vector search. Together they let the architecture ingest documents sequentially, outrun the context window, and compare narrative output against manually isolated 'implicit ground truth' in which document-level dots have been removed.

What would settle it

Run the augmented pipeline on the Crescent dataset's Holland Queen subplot with a sufficient number of seeds, and count how often the final narrative explicitly states that the ship might be carrying dangerous cargo; if that fraction is high (e.g., above 50%), the paper's claim that LLMs cannot make such speculative connections would be falsified.

Watch

Extended reading notes

Core claim

The paper claims that, on the Crescent, Atlantic Storm, and Manpad datasets, LLMs as-is are inadequate for intelligence analysis, and that this inadequacy is specifically a failure of analytical creativity rather than of summarization or memory. The authors build a three-step augmentation: dynamic evidence trees (DETs) as a growing memory graph of evidential and hypothesis dots, LLM-based condensation of each report into a single information dot, and a retrieval pipeline of embedding search plus LLM filtering. With this augmentation, narrative scores on ROUGE, METEOR, and GPT-4 ratings improve, especially on the largest dataset, because the pipeline keeps investigation threads organized. But on the central task—inferring the implicit plot that connects documents—the models still fail, for example describing the Holland Queen reports without speculating that the ship might carry dangerous cargo. From this the paper concludes that LLMs should be viewed as organizers and narrative drafters within an analyst's workflow, not as autonomous 'dot-connectors'.

Load-bearing premise

The whole study rests on the assumption that the manually-built 'implicit ground truth' (solution charts stripped of document-level dots) and the ROUGE/METEOR and GPT-4 scores against it actually measure analytical creativity; if they don't, the conclusion that LLMs lack analytical creativity collapses.

Editorial extensions

If this is right

  • Current LLMs can be usefully deployed as modules for evidence marshalling and narrative drafting in intelligence workflows, but human analysts must remain responsible for hypothesis generation.
  • Adding a memory structure like DETs plus data condensation is enough to materially improve narrative organization on large dossiers, and removing the condensation step degrades performance.
  • Model scale alone is not the lever: GPT-4 showed no consistent gain over GPT-3.5 on these tasks, so reasoning augmentations such as chain-of-thought are the more direct route.
  • Traditional lexical metrics (ROUGE, METEOR, BERTScore) can give misleadingly high scores across distinct datasets due to shared vocabulary; evaluations of such reasoning should rely more on human or LLM-judge ratings of relevance, coverage, and thoughtfulness.
  • Temperatures around 0.5-1.0 are optimal for narrative generation; beyond that, randomness does not produce more creative reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to run the same DET pipeline with a deliberately 'imaginative' second-pass model that is prompted to generate hypotheses from the DET leaves; the paper's own small case study suggests this may recover some speculative connections, but that would go beyond the paper's claims.
  • The paper's reliance on GPT-4 as both judge and evaluated model is a confound; a natural next experiment is to have human intelligence analysts rate the augmented narratives, which could either validate or overturn the creativity-gap conclusion.
  • If the failure is truly about speculative reasoning, then fine-tuning or few-shot prompting with examples of implicit plot discovery (rather than more retrieval or memory) would be the targeted fix; the paper's results imply this is where the bottleneck lies.
  • The position-sensitivity finding (the order and placement of entities matters) suggests that even with a memory tree, the final generation step still compresses everything into a single context window; an architecture that reads the DET branches sequentially might fare better.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether large language models (LLMs) can support intelligence analysis, a task requiring both evidence marshaling and speculative, imaginative reasoning. The authors propose a three-part augmentation: Dynamic Evidence Trees (DETs) as a memory structure, LLM-based data condensation, and an LLM-driven retrieval pipeline. They evaluate GPT-3.5, GPT-4, Llama-2, Mistral-7B, and Gemma-2 on three intelligence datasets (Crescent, Atlantic Storm, Manpad) using document classification F1, ROUGE/METEOR narrative similarity against a manually derived "implicit ground truth," and GPT-4 Likert ratings. The central claims are that (1) LLMs as-is are inadequate for intelligence analysis, and (2) DET augmentation improves evidence organization and narrative quality on larger datasets but does not improve speculative reasoning. The paper also reports on the pitfalls of traditional lexical/embedding metrics (BERTScore, ROUGE, METEOR) in this domain, showing that they are inflated and non-discriminative.

Significance. If validated, the paper's negative result—that current LLMs lack the analytical creativity needed for intelligence analysis while being useful as evidence organizers—would be a valuable data point for the NLP and human-AI collaboration communities. The paper is among the first to apply LLMs to a multi-document intelligence analysis task with a structured memory augmentation, and it tests multiple model families and datasets. The public codebase and detailed qualitative case studies are also strengths. However, the central claim rests on an unvalidated and unreleased implicit ground truth and on automatic metrics that the paper itself demonstrates are inflated, and the LLM-as-judge evaluation uses a model that is also an evaluated system. These measurement issues mean that the headline result is not yet established with the rigor needed for a strong publication, though the direction is promising and the proposed architecture is clearly described.

major comments (4)
  1. [Section III-A, Table IV] The central negative claim that LLMs lack analytical creativity is evaluated exclusively against an "implicit ground truth" that the authors construct manually by removing document-level dots from solution charts. This target is subjective, no inter-annotator agreement is reported, and the ground truth is not released. Table IV simultaneously shows that the lexical and embedding metrics used in Table IIIa (ROUGE, METEOR, BERTScore) are inflated and non-discriminative in this domain, with BERTScore near 0.8 even for different datasets and random text. Since the same metrics are used to support the claim that DET augmentation improves narratives, the quantitative evidence for both the negative result and the positive augmentation claim is called into question. A human evaluation with multiple annotators, or a validated automatic measure, is needed to support the paper's main conclusions.
  2. [Section III-C and Table IIIb] GPT-4 is used as the automatic judge of narrative quality (Table IIIb) while GPT-4 is also one of the evaluated models (Table IIIc). This creates a potential bias: the judge may favor its own outputs, and there is no calibration against human analyst judgments. The paper cites prior work on LLM-as-a-judge, but in a setting as specialized as intelligence analysis, the unvalidated use of a model as both judge and evaluated system is a load-bearing weakness for the claim that augmentation improves narrative quality.
  3. [Section IV-A and Table IIIa] The claim in Figure 7 that "DET (regular) and DET (person-based) show improved performance across metrics and datasets" is not consistent with the numbers in Table IIIa. For example, on Manpad, Basic Prompt achieves METEOR 0.33 while DET (regular) achieves 0.27; on Crescent, Basic Prompt has R1 0.22 and RL 0.16 versus DET (regular) R1 0.18 and RL 0.13. The normalized plots in Figure 7 may be aggregating over temperatures, but no error bars or significance tests are provided. The augmentation benefit is therefore not demonstrated for all datasets and metrics, and the claims need to be qualified or supported with statistical testing.
  4. [Section IV-B] The temperature and context-length sweep for the augmented pipeline is performed on the Crescent test set, and the optimal settings are then reported and presumably used for other datasets. This introduces selection bias: the reported results are conditioned on parameters chosen to maximize performance on one dataset. The paper should either report results for all tested settings, use a separate validation set for each dataset, or explicitly treat the operating point as a free parameter and discuss its effect on the reported outcomes.
minor comments (6)
  1. [Abstract and Introduction] The abstract states that LLMs are "still inadequate to support intelligence analysts" but the introduction's contribution 3 says "augmentations help orchestrate and improves narratives on large datasets"; the wording should be unified to avoid overstating the negative result before presenting the evidence.
  2. [Section II-A, Eq. (2)] The notation in Equations (1)-(2) is confusing: the token probability P(hdi,j | hdi,1, ..., hdi,j−1) is standard, but the preceding discussion relates it to evidential and hypothesis dots without formally connecting the autoregressive factorization to the dot-merging process. Clarify the role of Equation (2) in the proposed model.
  3. [Section III-C] The GPT-4 Likert ratings in Table IIIb are reported as single scores for relevance, coverage, and thoughtfulness for each dataset, but the appendix or methods do not specify the number of rating runs, the prompt used, or the variance across runs. Reporting a distribution or inter-rater consistency would help.
  4. [Section IV-C] The qualitative case study for the Crescent dataset is informative, but it focuses on one subplot (the Holland Queen ship). Including a qualitative analysis of a failure on another dataset (e.g., Atlantic Storm) would strengthen the claim that the limitation is general.
  5. [Section V and Table IIIa] The recommendation "Larger is not necessarily better" is based on Table IIIc, but the table shows performance differences among models that are within the range of metric noise reported in Table IV. The claim should be tempered or supported with significance testing.
  6. [Limitations paragraph] The limitations paragraph only acknowledges prompt wording variability; it does not address the validity of the implicit ground truth or the LLM-as-judge issue. A fuller statement of evaluation limitations would improve transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central empirical claims are benchmark evaluations against an external manually constructed implicit ground truth, not reductions to the paper's own inputs.

full rationale

The paper contains no fitted-parameter-to-prediction reduction or equation-level circularity. The central claim (LLMs lack analytical creativity; DETs improve orchestration) is an empirical evaluation of LLM outputs against a manually isolated 'implicit ground truth' (Section III-A), which is an external benchmark, not a quantity defined by the models being tested. DET construction is an algorithmic pipeline (Algorithms 1-2) that does not fit any parameter from the evaluation target. Self-citations ([7], [24], [55]) are background/baseline references and are not load-bearing for the conclusion. The GPT-4-as-judge design and the paper's own demonstration that lexical/embedding metrics are inflated (Table IV) raise measurement-validity questions, but they are not instances where a predicted quantity is equal to an input by construction; they concern whether the benchmark measures analytical reasoning, not circular reasoning. Hence no circularity step is identified.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central empirical claims rest on a few domain assumptions about ground truth, condensation, and LLM-based evaluation. There is one explicit hyperparameter choice fitted to the test data. No new physical or mathematical entities are introduced.

free parameters (1)
  • Operating point (temperature, word limit) for augmented pipeline = 0.7/100 and 0.5/150
    Selected by sweeping on the Crescent evaluation set; these settings are used in the reported augmented narrative comparisons.
assumptions (4)
  • domain assumption Each report condenses to exactly one evidential dot in zero-shot extraction
    Section II-E states "All the reports in our experiments yielded a single dot each." This simplifies DET construction but may discard information and is an empirical claim about the datasets rather than a guaranteed property.
  • domain assumption Manually derived implicit ground truth from solution charts is the correct evaluation target
    Section III-A: the authors isolate implicit information "by manually going through the chart and removing the document level dots." If this derivation is subjective, all narrative scores inherit that subjectivity.
  • domain assumption GPT-4 Likert ratings are valid proxies for relevance, coverage, and thoughtfulness
    Section III-C relies on LLM-as-judge without human calibration in this paper; this assumption is load-bearing for the qualitative claim that LLMs lack analytical creativity.
  • domain assumption Reports arrive in chronological order and sequential processing matches analyst workflow
    Section II-A and II-G: the pipeline processes reports sequentially and arbitrary order is allowed within a day. This assumption affects how evidence trees are built.
invented entities (1)
  • Dynamic Evidence Tree (DET)
    purpose: Memory module to track and merge evidential and hypothesis dots across an investigation
    A software construct introduced in Section II-D; its benefit is demonstrated only within this paper's experiments and it is not benchmarked against simpler memory baselines such as flat vector stores or standard RAG.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Augmentations to support Analytical Reasoning over Multiple Documents." pith.science (2026). https://pith.science/paper/VREDI6DK

@misc{pith2026241116116,
  author       = {Pith},
  title        = {Pith review of: LLM Augmentations to support Analytical Reasoning over Multiple Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VREDI6DK}},
  note         = {Machine review of arXiv:2411.16116}
}
read the original abstract

Building on their demonstrated ability to perform a variety of tasks, we investigate the application of large language models (LLMs) to enhance in-depth analytical reasoning within the context of intelligence analysis. Intelligence analysts typically work with massive dossiers to draw connections between seemingly unrelated entities, and uncover adversaries' plans and motives. We explore if and how LLMs can be helpful to analysts for this task and develop an architecture to augment the capabilities of an LLM with a memory module called dynamic evidence trees (DETs) to develop and track multiple investigation threads. Through extensive experiments on multiple datasets, we highlight how LLMs, as-is, are still inadequate to support intelligence analysts and offer recommendations to improve LLMs for such intricate reasoning applications.

Figures

Figures reproduced from arXiv: 2411.16116 by the authors.

Figure 1
Figure 1. Three steps to intelligence analysis (IA). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Dynamic Evidence Trees D. First augmentation: Dynamic Evidence Trees (DETs) As an improvement to the basic LLM, we augment it with dynamic evidence trees (DETs). This is the main memory [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Data condensation and dot extraction. E. Second augmentation: Data Condensation We utilize the language modeling capabilities of LLMs to digest the set of reports into usable information dots before generating reports. These condensed information dots can be merged with other information dots to create hypotheses. We empirically test various system prompts to break down the report in a zero-shot fashion such that a … view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: All augmentations together with retrieve and merge [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Intra and inter class distances and similarity for relevant [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Normalized metrics from the narrative for different methods with different temperatures [DET [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Crescent dataset solution and narratives from LLMs with ground truth [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Small use case for imaginative reasoning: (Left) speculative in nature, combining two people based on the similarity [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Effect of temperature, different prompt style, allow [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 40 canonical work pages

  1. [1]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  2. [2]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    work page 2022

  3. [3]

    Sparks of artificial general intelligence: Early exper- iments with gpt-4,

    S. Bubeck et al., “Sparks of artificial general intelligence: Early exper- iments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023

    arXiv 2023

  4. [4]

    Large Language Models Humanize Technology

    P. Kumar, “Large language models humanize technology,” arXiv preprint arXiv:2305.05576, 2023

    work page Pith review arXiv 2023

  5. [5]

    Challenges and applications of large language models,

    J. Kaddour et al. , “Challenges and applications of large language models,” arXiv preprint arXiv:2307.10169 , 2023

    arXiv 2023

  6. [6]

    The role of large language models in medical education: applications and implications,

    C. W. Safranek, A. E. Sidamon-Eristoff, A. Gilson, and D. Chartash, “The role of large language models in medical education: applications and implications,” p. e50945, 2023

    work page 2023

  7. [7]

    Neural methods for data-to-text generation,

    M. Sharma, A. K. Gogineni, and N. Ramakrishnan, “Neural methods for data-to-text generation,” ACM Transactions on Intelligent Systems and Technology, 2024

    work page 2024

  8. [8]

    Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study,

    Y .-a. Kang and J. Stasko, “Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study,” in 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), 2011, pp. 21–30

    work page 2011

Show all 69 references
  1. [9]

    Exploring the evolution of sensemaking strategies in immersive space to think,

    K. Davidson, L. Lisle, K. Whitley, D. A. Bowman, and C. North, “Exploring the evolution of sensemaking strategies in immersive space to think,” IEEE transactions on visualization and computer graphics , 2022

    2022

  2. [10]

    Llama: Open and efficient foundation language models,

    H. Touvron et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  3. [11]

    Llama 2: Open foundation and fine-tuned chat models,

    ——, “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  4. [12]

    Training language models to follow instructions with human feedback,

    L. Ouyang et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  5. [13]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao et al., “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  6. [14]

    Graph of thoughts: Solving elaborate problems with large language models,

    M. Besta et al. , “Graph of thoughts: Solving elaborate problems with large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 17 682–17 690

    2024

  7. [15]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models,

    M. Levy, A. Jacoby, and Y . Goldberg, “Same task, more tokens: the impact of input length on the reasoning performance of large language models,” arXiv preprint arXiv:2402.14848 , 2024

    2024 arXiv

  8. [16]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park et al. , “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , 2023, pp. 1–22

    2023

  9. [17]

    Mteb: Massive text embedding benchmark,

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “Mteb: Massive text embedding benchmark,” arXiv preprint arXiv:2210.07316 , 2022

    2022 arXiv

  10. [18]

    Embedding-based retrieval in facebook search,

    J.-T. Huang et al. , “Embedding-based retrieval in facebook search,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020, pp. 2553–2561

    2020

  11. [19]

    One embedder, any task: Instruction-finetuned text embeddings,

    H. Su et al. , “One embedder, any task: Instruction-finetuned text embeddings,” 2023

    2023

  12. [20]

    Mistral 7b,

    A. Q. Jiang et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  13. [21]

    Gemma: Open models based on gemini research and technology,

    G. Team et al. , “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295 , 2024

    2024 arXiv

  14. [22]

    Where do i start? algorithmic strategies to guide intelligence analysts,

    H. Wu et al. , “Where do i start? algorithmic strategies to guide intelligence analysts,” in Proceedings of the ACM SIGKDD Workshop on Intelligence and Security Informatics , 2012, pp. 1–8

    2012

  15. [23]

    Principles and tools for collaborative entity-based intelligence analysis,

    E. A. Bier, S. K. Card, and J. W. Bodnar, “Principles and tools for collaborative entity-based intelligence analysis,” IEEE transactions on visualization and computer graphics , vol. 16, no. 2, pp. 178–191, 2009

    2009

  16. [24]

    Storytelling in entity networks to support intelligence analysts,

    M. S. Hossain, P. Butler, A. P. Boedihardjo, and N. Ramakrishnan, “Storytelling in entity networks to support intelligence analysts,” in Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining , 2012, pp. 1375–1383

    2012

  17. [25]

    Entity workspace: An evidence file that aids memory, inference, and reading,

    E. A. Bier, E. W. Ishak, and E. Chi, “Entity workspace: An evidence file that aids memory, inference, and reading,” in Intelligence and Security Informatics, S. Mehrotra, D. D. Zeng, H. Chen, B. Thuraisingham, and F.-Y . Wang, Eds., 2006, pp. 466–472

    2006

  18. [26]

    Combining computational analyses and interactive visualization for document exploration and sensemaking in jigsaw,

    C. G ¨org et al. , “Combining computational analyses and interactive visualization for document exploration and sensemaking in jigsaw,” IEEE transactions on Visualization and Computer Graphics , vol. 19, no. 10, pp. 1646–1663, 2012

    2012

  19. [27]

    Jigsaw: supporting investiga- tive analysis through interactive visualization,

    J. Stasko, C. Gorg, Z. Liu, and K. Singhal, “Jigsaw: supporting investiga- tive analysis through interactive visualization,” in2007 IEEE Symposium on Visual Analytics Science and Technology. IEEE, 2007, pp. 131–138

    2007

  20. [28]

    Modern hierarchical, agglomerative clustering algorithms,

    D. M ¨ullner, “Modern hierarchical, agglomerative clustering algorithms,” 2011

    2011

  21. [29]

    Birch: an efficient data clustering method for very large databases,

    T. Zhang, R. Ramakrishnan, and M. Livny, “Birch: an efficient data clustering method for very large databases,”ACM sigmod record, vol. 25, no. 2, pp. 103–114, 1996

    1996

  22. [30]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  23. [31]

    Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments,” Proceedings of ACL-WMT, pp. 65–72, 2004

    2004

  24. [32]

    Gpt-4 technical report,

    J. Achiam et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  25. [33]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems , vol. 36, pp. 46 595–46 623, 2023

    2023

  26. [34]

    Gptscore: Evaluate as you desire,

    J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” arXiv preprint arXiv:2302.04166 , 2023

    2023 arXiv

  27. [35]

    Generative judge for evaluating alignment,

    J. Li et al., “Generative judge for evaluating alignment,” arXiv preprint arXiv:2310.05470, 2023

    2023 arXiv

  28. [36]

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers,

    H. Huang, Y . Qu, J. Liu, M. Yang, and T. Zhao, “An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers,” arXiv preprint arXiv:2403.02839 , 2024

    2024 arXiv

  29. [37]

    Is gpt-4 a reliable rater? evaluating consistency in gpt-4’s text ratings,

    V . Hackl, A. E. M ¨uller, M. Granitzer, and M. Sailer, “Is gpt-4 a reliable rater? evaluating consistency in gpt-4’s text ratings,” in Frontiers in Education, vol. 8. Frontiers Media SA, 2023, p. 1272229

    2023

  30. [38]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang et al. , “Self-instruct: Aligning language models with self- generated instructions,” arXiv preprint arXiv:2212.10560 , 2022

    2022 arXiv

  31. [39]

    Lmflow: An extensible toolkit for finetuning and inference of large foundation models,

    S. Diao et al. , “Lmflow: An extensible toolkit for finetuning and inference of large foundation models,” arXiv preprint arXiv:2306.12420, 2023

    2023 arXiv

  32. [40]

    Automatic story generation: Challenges and attempts,

    A. Alabdulkarim, S. Li, and X. Peng, “Automatic story generation: Challenges and attempts,” arXiv preprint arXiv:2102.12634 , 2021

    2021 arXiv

  33. [41]

    gkamradt/LLMTest needleinahaystack,

    gkamradt, “gkamradt/LLMTest needleinahaystack,” Jul. 2024, original- date: 2023-11-11T00:50:02Z. [Online]. Available: https://github.com/ gkamradt/LLMTest NeedleInAHaystack

    2024

  34. [42]

    Lost in the middle: How language models use long contexts,

    N. F. Liu et al. , “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024

    2024

  35. [43]

    Learning to Reason with LLMs

    “Learning to Reason with LLMs.” [Online]. Available: https: //openai.com/index/learning-to-reason-with-llms/

  36. [44]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  37. [45]

    G-eval: Nlg evaluation using gpt-4 with better human alignment,

    Y . Liu et al. , “G-eval: Nlg evaluation using gpt-4 with better human alignment,” arXiv preprint arXiv:2303.16634 , 2023

    2023 arXiv

  38. [46]

    Crescent Train New York, Atlanta, New Orleans | Amtrak

    “Crescent Train New York, Atlanta, New Orleans | Amtrak.” [Online]. Available: https://www.amtrak.com/crescent-train

  39. [47]

    The flan collection: Designing data and methods for effective instruction tuning,

    S. Longpre et al., “The flan collection: Designing data and methods for effective instruction tuning,” in International Conference on Machine Learning. PMLR, 2023, pp. 22 631–22 648

    2023

  40. [48]

    Promptsource: An integrated development envi- ronment and repository for natural language prompts,

    S. H. Bach et al. , “Promptsource: An integrated development envi- ronment and repository for natural language prompts,” arXiv preprint arXiv:2202.01279, 2022

    2022 arXiv

  41. [49]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang et al. , “V oyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291 , 2023

    2023 arXiv

  42. [50]

    Mathematical discoveries from program search with large language models,

    B. Romera-Paredes et al. , “Mathematical discoveries from program search with large language models,” Nature, vol. 625, no. 7995, pp. 468–475, 2024

    2024

  43. [51]

    Pirolli and S

    P. Pirolli and S. Card, The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis , Jan. 2005

    2005

  44. [52]

    A Data–Frame Theory of Sensemaking,

    G. Klein, J. K. Phillips, E. L. Rall, and D. A. Peluso, “A Data–Frame Theory of Sensemaking,” in Expertise Out of Context . Psychology Press, 2007, num Pages: 43

    2007

  45. [53]

    Model-guided information discovery for in- telligence analysis,

    R. Alonso and H. Li, “Model-guided information discovery for in- telligence analysis,” in Proceedings of the 14th ACM international conference on Information and knowledge management , 2005, pp. 269– 270

    2005

  46. [54]

    A multi-agent system of evidential reasoning for intelligence analyses,

    E. Lindahl, S. O’Hara, and Q. Zhu, “A multi-agent system of evidential reasoning for intelligence analyses,” in Proceedings of the 6th interna- tional joint conference on Autonomous agents and multiagent systems , 2007, pp. 1–6

    2007

  47. [55]

    Interactive storytelling over document collections,

    D. Maiti, M. R. Islam, and N. Ramakrishnan, “Interactive storytelling over document collections,” arXiv preprint arXiv:1602.06566 , 2016

    2016 arXiv

  48. [56]

    The human is the loop: new directions for visual analytics,

    A. Endert et al. , “The human is the loop: new directions for visual analytics,” Journal of intelligent information systems , vol. 43, pp. 411– 435, 2014

    2014

  49. [57]

    Helping intelligence analysts make connections,

    M. S. Hossain, C. Andrews, N. Ramakrishnan, and C. North, “Helping intelligence analysts make connections,” in Workshops at the Twenty- Fifth AAAI Conference on Artificial Intelligence , 2011

    2011

  50. [58]

    Spaces to think: A comparison of small, large, and immersive displays for the sensemaking process,

    L. Lisle et al. , “Spaces to think: A comparison of small, large, and immersive displays for the sensemaking process,” in 2023 IEEE Inter- national Symposium on Mixed and Augmented Reality (ISMAR) . IEEE, 2023, pp. 1084–1093

    2023

  51. [59]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

    2020

  52. [60]

    Training language models with memory augmentation,

    Z. Zhong, T. Lei, and D. Chen, “Training language models with memory augmentation,” arXiv preprint arXiv:2205.12674 , 2022

    2022 arXiv

  53. [61]

    Self-knowledge guided retrieval augmentation for large language models,

    Y . Wang, P. Li, M. Sun, and Y . Liu, “Self-knowledge guided retrieval augmentation for large language models,” 2023

    2023

  54. [62]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao et al. , “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , 2023

    2023 arXiv

  55. [63]

    Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion,

    K. Shuster et al., “Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion,”arXiv preprint arXiv:2203.13224, 2022

    2022 arXiv

  56. [64]

    Chameleon: Plug-and-play compositional reasoning with large language models,

    P. Lu et al. , “Chameleon: Plug-and-play compositional reasoning with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  57. [65]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick et al., “Toolformer: Language models can teach themselves to use tools,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  58. [66]

    Graph-toolformer: To empower llms with graph rea- soning ability via prompt augmented by chatgpt,

    J. Zhang, “Graph-toolformer: To empower llms with graph rea- soning ability via prompt augmented by chatgpt,” arXiv preprint arXiv:2304.11116, 2023

    2023 arXiv

  59. [67]

    Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning,

    E. Karpas et al., “Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning,” arXiv preprint arXiv:2205.00445 , 2022

    2022 arXiv

  60. [68]

    Talm: Tool augmented language models,

    A. Parisi, Y . Zhao, and N. Fiedel, “Talm: Tool augmented language models,” arXiv preprint arXiv:2205.12255 , 2022

    2022 arXiv

  61. [69]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin et al. , “Toolllm: Facilitating large language models to master 16000+ real-world apis,” arXiv preprint arXiv:2307.16789 , 2023. APPENDIX Fig. 10: Effect of temperature, different prompt style, allow- able context length on LLM responses

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.