Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking

Hai Toan Nguyen; Tien Dat Nguyen; Viet Ha Nguyen

REVIEW 4 major objections 6 minor 23 references

Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing fixed-size text chunks with supervised segmentation plus graph clustering consistently improves retrieval-augmented question answering, because retrieval then matches either a local segment or a thematic…

desk verdict A clean, bottom-up segmentation-plus-clustering chunking pipeline with consistent but modest gains on three RAG benchmarks; the core idea is worth a serious look, but the evaluation's unverified token budgets and missing code keep it from being citable yet. read the letter →

arxiv 2507.09935 v1 pith:KSGSMS6A submitted 2025-07-14 cs.CL cs.AI

Hai Toan Nguyen , Tien Dat Nguyen , Viet Ha Nguyen This is my paper

classification cs.CLcs.AI

keywords retrieval-augmentedgenerationtextsegmentationsemanticchunkinggraphclusteringmaximalcliquesmulti-vectorretrievallong-documentquestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that retrieval-augmented generation improves when documents are chunked by first segmenting them with a supervised model and then grouping related segments into clusters, rather than splitting text by fixed token counts or sentence-level similarity. It argues that such structure-aware chunks carry both local context and broader thematic context, and that retrieving with separate embeddings for segments and their cluster increases the chance of matching a query. On NarrativeQA, QuALITY, and QASPER, the proposed segmentation-clustering method reports higher scores than fixed-size and semantic chunking at matched token budgets. If true, this is a practical result: better answer quality on long documents without changing the reader model or the retriever, only how the index is organized.

What carries the argument

The framework's load-bearing mechanism is a bottom-up segmentation-clustering pipeline. A supervised bidirectional-LSTM segmentation model labels each sentence as either continuing a segment or ending one; the resulting segments become nodes in a relatedness graph, with edges where pairwise embedding similarity exceeds a threshold $\tau = \mu + k\sigma$ (with $\mu$ the mean similarity, $\sigma$ the standard deviation, and $k$ tuned to the target chunk size). Maximal cliques are detected, adjacent clique-sharing segments are merged into initial clusters, adjacent clusters sharing a clique are merged, and leftover single-sentence clusters are attached to the nearest neighbor. Each cluster gets a mean-pooled embedding, and retrieval scores a chunk by the maximum cosine similarity between the query and any of its segment embeddings or the cluster embedding. That max-over-multiple-vectors step is what lets a query match either local detail or broader context.

What would settle it

Compute and publish the actual token-length distribution of clusters produced by $\tau = \mu + k\sigma$ for k = 1.2, 0.7, and 0.4; if the mean cluster sizes do not approximate 512, 1024, and 2048 tokens, or if the spread is large enough to overlap the fixed-size baselines, the token-matched comparison is not actually token-matched. A second check: replace the retrained segmentation model with a model that reaches the original pk score of 20; if retrieval scores do not move, the paper's attributed mechanism of coherent boundaries is not doing the work.

Watch

Extended reading notes

Core claim

The central claim is that a bottom-up pipeline of supervised text segmentation followed by unsupervised graph-based clustering produces retrieval-augmented generation chunks that are more semantically coherent than fixed-size or semantic chunks, and that dual-level retrieval using both segment and cluster embeddings outperforms single-vector retrieval. Concretely, the paper reports that the 1024-token segment-plus-cluster configuration reaches ROUGE-L 26.54 and METEOR 30.26 on NarrativeQA and F1 24.67 on QASPER, and that the 512-token segment-plus-cluster configuration reaches 63.77 accuracy on QuALITY, each beating the corresponding fixed-size baseline. The improvement is attributed to clusters that can group non-adjacent but related passages, preserving higher-level themes that arbitrary boundaries would fragment.

Load-bearing premise

The result rests on the unverified premise that the clustering threshold k-values (1.2, 0.7, 0.4) produce clusters whose token sizes actually match the 512/1024/2048-token fixed-size baselines, and that the retrained segmentation model, despite being worse at detecting section boundaries, still yields boundaries that aid retrieval.

Editorial extensions

If this is right

At a fixed token budget, replacing fixed-size chunks with segment-cluster chunks lifts QA metrics on all three datasets, with the largest relative gains on NarrativeQA and QASPER.
Retrieval that combines segment-level and cluster-level vectors beats cluster-only retrieval, implying the two levels carry complementary information.
The 1024-token configuration is the best operating point; 2048-token chunks show diminishing returns, consistent with coherence loss as chunks grow.
Because clusters can join non-adjacent segments, the method can surface evidence scattered across a document that adjacent-window chunking misses.
The framework is drop-in for RAG: it changes only the indexing and retrieval representation, not the reader LLM or the vector store.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because the paper does not report the token-length distribution of its clusters, part of the reported gain could come from variable chunk sizes rather than semantic grouping; measuring and matching cluster-size distributions would separate these effects.
Beyond the paper: the retrained segmentation model's pk of 35, versus 20 for the model it was based on, makes a sensitivity analysis the obvious next experiment—if a stronger segmenter does not improve retrieval, segmentation quality is not load-bearing.
Beyond the paper: applying the same bottom-up clustering recursively to the clusters themselves would yield a true multi-level index, a path the paper names as future work but does not test.
Beyond the paper: the method's ability to cluster non-adjacent related segments suggests it could transfer to other scattered-evidence retrieval tasks, such as fact verification or summarization, though the paper does not evaluate those.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a readable, honest empirical paper that combines supervised text segmentation with graph-based clustering for RAG chunking, and it reports consistent 1–3 point gains over fixed-size and semantic chunking on NarrativeQA, QuALITY, and QASPER. What's actually new is the specific pipeline: Koshorek et al. segmentation, a GraphRAG-style clique-based clustering, and max-over-embeddings retrieval with both segment-level and cluster-level vectors. Each component is established, but this combination is not in the cited literature, so the result is a genuine new data point in the crowded chunking space.

The paper does several things well. It is transparent about the weak segmentation model, reporting a pk of 35 versus the original 20 on WIKI-50 and explicitly saying the focus is retrieval integration, not segmentation quality. The evaluation covers three long-document benchmarks with multiple metrics, and the method wins in almost every configuration, including cluster-only against base chunking. The writing is clear and the figures, especially the worked example in Table 1, help explain the algorithm.

The soft spots are real but fixable. First, the token-budget fairness is unverified. Chunk sizes for the proposed method are not fixed; they emerge from segmentation and the threshold tau = mu + k*sigma. The k values 1.2, 0.7, 0.4 are asserted to make average chunk sizes 512, 1024, and 2048 tokens, but no realized chunk-size distribution is reported. If the clusters run larger than nominal, retrieving 8, 4, or 2 chunks silently gives the method a larger retrieval budget than the fixed-size baselines. Second, the 256-token condition retrieves 20 chunks, which is about 5120 tokens, not 4096, so that row is not token-matched either. Third, there are no error bars, significance tests, multiple seeds, released code, or sensitivity analysis for k or segmentation quality. Fourth, the paper overclaims when it says the method groups 'related segments, even if they are not adjacent'; the merge procedure only merges adjacent clusters, so final clusters are contiguous spans, and non-adjacent clique edges do not create non-contiguous groups. The reference list also has a misspelled GraphRAG citation and a placeholder DOI in reference [20].

None of this sinks the central claim. The direction is plausible, and the wins are consistent, but the magnitude of the gains could partly reflect uncontrolled token counts. The authors' own acknowledgment of the poor pk score strengthens the need for a sensitivity analysis. I would send this to peer review with a request for major revision: report actual chunk-size distributions, run multiple seeds and significance tests, release code, and analyze sensitivity to k and segmentation quality. If those come back clean, it becomes a citable empirical result. As is, I would not cite it yet.

Referee Report

4 major / 6 minor

Summary. The paper proposes a RAG chunking framework that first applies a supervised text segmentation model (Koshorek et al.) to split documents into coherent segments, then uses graph-based clustering (adapted from GraphRAG) to group adjacent, semantically related segments into larger clusters. At retrieval time, each chunk is represented by multiple segment embeddings plus one cluster embedding, and the top-k chunks are selected by cosine similarity. The authors evaluate the method on NarrativeQA, QuALITY, and QASPER at average chunk sizes of 512, 1024, and 2048 tokens, comparing against fixed-size chunking and a semantic chunking baseline. They report consistent improvements, e.g., a 26.54 ROUGE-L on NarrativeQA at 1024 tokens versus 23.86 for the fixed-size baseline, and an F1 of 24.67 on QASPER versus 22.07 for the baseline at the same chunk size.

Significance. If the empirical claim holds, the paper makes a useful contribution by showing that structure-aware, bottom-up segmentation and clustering can improve RAG retrieval over fixed-size chunking on long-document QA benchmarks. The idea of combining segment-level and cluster-level embeddings is plausible and the evaluation spans three diverse datasets. The paper also adapts existing components (a supervised segmentation model and a graph-clustering algorithm) in a straightforward way, which aids reproducibility in principle. However, the current manuscript does not release code, data, or trained checkpoints, and the empirical evidence lacks statistical rigor. The central comparison also depends on an unverified token-budget alignment, so the significance of the reported gains cannot be fully assessed without additional experiments.

major comments (4)

[§4.2, §4.3, Tables 2–3] The fairness of the comparison rests on the claim that all methods are evaluated under approximately the same token budget, but the actual token counts for the proposed clusters are never reported. Section 4.2 states that 8, 4, and 2 chunks are retrieved for 512, 1024, and 2048-token chunks, respectively, to keep the total near 4096 tokens. However, the proposed method's chunks are not fixed-size; they are determined by the segmentation model and the clustering threshold tau = mu + k*sigma. The paper asserts that k = 1.2, 0.7, 0.4 aligns the average chunk size with 512, 1024, and 2048 tokens, but it provides no realized chunk-size distribution for any dataset or k value. If the clusters are larger than the nominal sizes, the method silently receives a larger retrieval budget than the fixed-size baselines, and the consistent gains in Tables 2 and 3 could be a token-count artifact rather than evidence of better chunk boundaries. Please report the mean/median/quantiles of the realized cluster token counts per dataset, and rerun the fixed-size baselines under exactly matched token budgets.
[§4.3, Tables 2–3] The segmentation model used in the framework is acknowledged to be substantially weaker than the original model (pk = 35 vs. 20 on WIKI-50), yet there is no sensitivity analysis tying segmentation quality to retrieval performance. The central mechanism of the paper is that supervised text segmentation creates coherent chunks; if the segmentation boundaries are noisy, the cluster sizes and content become less predictable, which interacts with the token-budget issue raised above. The paper needs at least one experiment varying k (e.g., 0.4, 1.2, 2.0) and ideally one using the original higher-quality segmentation checkpoint, to show that the reported gains are not an artifact of tuning k to the target chunk-size grid.
[§3.2, Table 1] The merge-clusters rule and the illustrative example in Table 1 are inconsistent. The rule states that adjacent clusters are merged if there is at least one clique containing a segment from each. In the example, clique {2,4,7} contains segment 4 from initial cluster {3,4,5} and segment 7 from initial cluster {6,7}, so clusters {3,4,5} and {6,7} should be merged, and after the first merge the combined cluster {1,2,3,4,5} should also merge with {6,7} via the same clique. The table instead reports {1,2,3,4,5} and {6,7} as the final state. This makes the clustering algorithm ambiguous and harms reproducibility; please correct either the rule or the example.
[§4.1–§4.4] No error bars, confidence intervals, significance tests, or multiple-seed runs are reported for any of the tables. The observed improvements are often small in absolute terms (e.g., 21.95 vs. 20.33 F1 on QASPER at 512 tokens; 63.77 vs. 60.23 accuracy on QuALITY at 512 tokens), and with a single run per condition one cannot distinguish a systematic advantage from evaluation noise. Since the paper's central claim is an empirical one, the authors should provide variance estimates across at least three to five independent runs of the full pipeline (or, if the evaluation is deterministic, state so explicitly and justify the absence of variance).

minor comments (6)

[§3.2, §4.2] The symbol k is used both for the clustering sensitivity parameter in Eq. (3) and for the number of retrieved chunks in Tables 2 and 3 (top-k). This notation collision makes the text harder to follow; please rename one of the two.
[§4.2, Tables 2–3] The semantic chunking baseline is evaluated only at the 256-token average chunk size, while the proposed method is evaluated at 512, 1024, and 2048 tokens. This leaves the comparison against semantic chunking incomplete at the larger sizes; adding semantic chunking at all sizes would strengthen the claim that the proposed method beats 'traditional chunking techniques' generally.
[§4.2] The paper does not report the prompt template, decoding parameters, or any other inference details for the GPT-4o-mini reader, nor the FAISS index configuration. These details are needed for reproducibility of the reported numbers.
[References] Reference [20] ('Rahul: A Guide to Chunking Strategies...') carries the placeholder DOI 10.5555/12345678, which is not a valid DOI; this reference needs to be corrected or replaced.
[Throughout] There are several typos and formatting issues: 'Glavias et al.' in §3.2 should presumably be 'Edge et al.'; 'is capable' in the Introduction should be 'are capable'; 'Fig. 3 demonstrated' should be 'Fig. 3 demonstrates'; and 'T able 1' appears in the text. A careful proofread is recommended.
[§4.3] The description of the training data ('100,000 documents from the Wiki727k dataset') is vague; please clarify the exact dataset split, the number of sentences, and whether the evaluation set WIKI-50 is the same as in the original paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reported gains are empirical and not forced by construction.

full rationale

The paper is an empirical benchmarking study; there is no derivation chain in which a claimed prediction is equivalent to an input. The only tunable parameter k in tau = mu + k*sigma (Eq. 3) is set in Section 4.3 to align nominal average chunk sizes (512/1024/2048 tokens) and is not optimized against the retrieval metrics, so the superiority claim in Section 4.4 is not statistically forced by construction. The retrieval budget procedure (Section 4.2) retrieves a proportional number of chunks to approximate 4096 tokens, but the paper does not report realized cluster size distributions, which creates an unverified fairness risk; this is a correctness/confound concern, not circularity, because no equation reduces the outcome to the parameter choice. The segmentation model (Koshorek et al. [14]) and clustering procedure (Edge et al. [6]) are external prior work, and the paper contains no load-bearing self-citations, uniqueness theorems, or renaming of a known result as a new derivation. The evaluation against fixed-size and semantic baselines is self-contained, so an honest non-finding is warranted.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method depends on one directly tuned parameter (k) that controls cluster granularity and is chosen to match the baseline chunk-size grid. The other assumptions are domain-level premises about the value of structure-aware chunking. No new physical or conceptual entities are introduced.

free parameters (2)

k (clustering threshold sensitivity) = 1.2 (512 tokens), 0.7 (1024), 0.4 (2048)
Parameter in tau = mu + k*sigma controlling the graph edge threshold. Chosen per target chunk size to align with baseline sizes, with no sensitivity analysis.
Top-k retrieval counts per chunk size = 20 for 256 tokens, 8 for 512, 4 for 1024, 2 for 2048
Set to equalize approximate total token count, but 20 times 256 gives 5120 tokens, not the stated roughly 4096, creating an inconsistency in the comparison.

assumptions (4)

domain assumption Supervised text segmentation produces semantically coherent units that help retrieval.
Core premise of the method, introduced in Section 3.2 without independent validation on the target corpora.
domain assumption Clustering adjacent segments by similarity threshold preserves sequential structure and improves chunk coherence.
The graph-clustering step in Section 3.2 assumes this improves over flat chunking; no ablation isolates its contribution.
domain assumption The retrained segmentation model with pk = 35 is adequate for the RAG pipeline.
Section 4.3 acknowledges the model underperforms the original (pk 35 vs 20) but asserts the framework still benefits; no analysis of segmentation error impact is given.
domain assumption GPT-4o-mini as reader fairly reflects retrieval quality.
The reader model and prompts are not specified; answers may mask or amplify retrieval differences.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking." pith.science (2026). https://pith.science/paper/KSGSMS6A

@misc{pith2026250709935,
  author       = {Pith},
  title        = {Pith review of: Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSGSMS6A}},
  note         = {Machine review of arXiv:2507.09935}
}

read the original abstract

Retrieval-Augmented Generation (RAG) systems commonly use chunking strategies for retrieval, which enhance large language models (LLMs) by enabling them to access external knowledge, ensuring that the retrieved information is up-to-date and domain-specific. However, traditional methods often fail to create chunks that capture sufficient semantic meaning, as they do not account for the underlying textual structure. This paper proposes a novel framework that enhances RAG by integrating hierarchical text segmentation and clustering to generate more meaningful and semantically coherent chunks. During inference, the framework retrieves information by leveraging both segment-level and cluster-level vector representations, thereby increasing the likelihood of retrieving more precise and contextually relevant information. Evaluations on the NarrativeQA, QuALITY, and QASPER datasets indicate that the proposed method achieved improved results compared to traditional chunking techniques.

Figures

Figures reproduced from arXiv: 2507.09935 by the authors.

**Figure 1.** Overview of our framework, incorporating text segmentation during the indexing process. cos(q, Ci) = max(cos(q, Es1), . . . , cos(q, Esm), cos(q, Ec)) (2) where Es1, Es2, . . . , Esm are segment embeddings for chunk Ci , and Ec is the cluster embedding for chunk Ci . The system ranks the similarity scores and selects the top-k chunks for processing by the LLMs to generate the response. 3.2 Text Segmentation and Clu… view at source ↗

**Figure 2.** Illustration of the framework’s chunking strategy: The text is first segmented into coherent parts using a text segmentation model. These segments are then clustered based on semantic similarities and their sequential order [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** A biography of Alexander Hamilton is being predicted by a text segmentation model. The model predicts segment boundaries by labeling each sentence with a 1 or a 0, where 1 marks the end of a segment and 0 otherwise [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Retrieved chunks based on multiple chunking strategy for the question about the story The Olympic Gene Pool [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 7 canonical work pages

[20]

doi.org/10.5555/12345678

Rahul: A Guide to Chunking Strategies for Retrieval Augmented Generation (RAG). doi.org/10.5555/12345678

doi:10.5555/12345678
[1]

A., Löser, A.: SECTOR: A Neural Model for Coherent Topic Segmentation and Classification

Arnold, S., Schneider, R., Cudré-Mauroux, P., Gers, F. A., Löser, A.: SECTOR: A Neural Model for Coherent Topic Segmentation and Classification. Transactions of the Association for Computational Linguistics, 7:169–184 (2019). doi.org/10.1162/ tacl_a_00261

work page 2019
[2]

In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)

Barrow, J., Jain, R., Morariu, V., Manjunatha, V., Oard, D., Resnik, P.: A Joint Model for Document Segmentation and Segment Labeling. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 313–322 (2020). doi.org/10.18653/v1/2020.acl-main.29

work page doi:10.18653/v1/2020.acl-main.29 2020
[3]

In: Machine Learning 34, 177–210 (1999)

Beeferman, D., Berger, A., Lafferty, J.: Statistical Models for Text Segmentation. In: Machine Learning 34, 177–210 (1999). doi.org/10.1023/A:1007506220214

doi:10.1023/a:1007506220214 1999
[4]

In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading Wikipedia to Answer Open- Domain Questions. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870–1879 (2017). doi. org/10.48550/arXiv.1704.00051

arXiv doi:10.48550/arxiv.1704.00051 2017
[5]

A., Gardner, M.: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., Gardner, M.: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In: Proceedings of the 2021 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 4599–4610 (2021). doi.org/10.18653/v1/2021.naacl...

doi:10.18653/v1/2021.naacl-main.365 2021
[6]

doi.org/10.48550/arXiv.2404.16130

Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Larson, J.: From Local to Global: A Graph RAG Approach to Query-Focused Summariza- tion (2024). doi.org/10.48550/arXiv.2404.16130

arXiv doi:10.48550/arxiv.2404.16130 2024
[7]

doi.org/10.48550/arXiv.2312.10997

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., Wang, H.: Retrieval-augmented generation for large language models: A survey (2023). doi.org/10.48550/arXiv.2312.10997

arXiv doi:10.48550/arxiv.2312.10997 2023

Show all 23 references

[8]

A.: Multi-paragraph segmentation of expository text

Hearst, M. A.: Multi-paragraph segmentation of expository text. In: Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pp. 9-16 (1994). doi.org/10.3115/981732.981734

1994
[9]

In: IEEE Transactions on Neural Networks and Learning Systems (2021)

Ji,S.,Pan,S.,Cambria,E.,Marttinen,P.,Yu,P.S.:ASurveyonKnowledgeGraphs: Representation, Acquisition, and Applications. In: IEEE Transactions on Neural Networks and Learning Systems (2021). doi.org/10.1109/TNNLS.2021.3070843

2021
[10]

doi.org/10.48550/arXiv.2406.15319

Jiang, Z., Ma, X., Chen, W.: LongRAG: Enhancing Retrieval-Augmented Gener- ation with Long-context LLMs (2024). doi.org/10.48550/arXiv.2406.15319

2024 doi arXiv
[11]

doi.org/10.48550/arXiv.2409

Kalra, R., Wu, Z., Gulley, A., Hilliard, A., Guan, X., Koshiyama, A., Treleaven, P.: HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applications (2024). doi.org/10.48550/arXiv.2409. 09046

2024 doi
[12]

https://github.com/FullStackRetrieval- com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_ Of_Text_Splitting.ipynb

Kamradt, G.: 5 Levels Of Text Splitting. https://github.com/FullStackRetrieval- com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_ Of_Text_Splitting.ipynb. Last accessed 16 August 2024

2024
[13]

M., Melis, G., Grefenstette, E.: The NarrativeQA Reading Comprehension Challenge

Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., Grefenstette, E.: The NarrativeQA Reading Comprehension Challenge. In: Trans- actions of the Association for Computational Linguistics, 6, pp. 317–328 (2018). doi.org/10.1162/tacl_a_00023

2018 doi
[14]

In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), pp

Koshorek, O., Cohen, A., Mor, N., Rotman, M., Berant, J.: Text Segmentation as a Supervised Learning Task. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Pape...

2018 doi
[15]

https://python.langchain

LangChain: How to recursively split text by characters. https://python.langchain. com/docs/how_to/recursive_text_splitter/. Last accessed 16 September 2024

2024
[16]

In: Proceedings of the 34th Inter- national Conference on Neural Information Processing Systems (NIPS ’20)

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th Inter- national Conference on Neur...

2020
[17]

doi.org/10.48550/arXiv.2402

Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., Gao, J.: Large Language Models: A Survey (2024). doi.org/10.48550/arXiv.2402. 06196

2024 doi
[18]

doi.org/10.48550/arXiv.2303.08774

OpenAI: GPT-4 Technical Report (2023). doi.org/10.48550/arXiv.2303.08774

2023 doi arXiv
[19]

Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmaku- mar, V., Ma, J., Thompson, J., He, H., Bowman, S.: QuALITY: Question Answer- ing with Long Input Texts, Yes!

Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmaku- mar, V., Ma, J., Thompson, J., He, H., Bowman, S.: QuALITY: Question Answer- ing with Long Input Texts, Yes!. In: Proceedings of the 2022 Conference of the North American Chapter of the Association ...

2022 doi
[21]

In: Proceedings of ACL 2012 Student Research Workshop, pp

Riedl, M., Biemann, C.: TopicTiling: A Text Segmentation Algorithm based on LDA. In: Proceedings of ACL 2012 Student Research Workshop, pp. 37–42, Asso- ciation for Computational Linguistics (2012). doi.org/10.5555/23456789

2012 doi
[22]

K., Saha, S., Jain, V., Mondal, S., Chadha, A.: A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Ap- plications

Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., Chadha, A.: A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Ap- plications. Indian Institute of Technology Patna and Stanford University (2023). doi.org/10.48550/arXiv.2402.07927

2023 doi arXiv
[23]

D.: RAP- TOR: Recursive Abstractive Processing for Tree-Organized Retrieval

Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., Manning, C. D.: RAP- TOR: Recursive Abstractive Processing for Tree-Organized Retrieval. In: Pro- ceedings of the International Conference on Learning Representations (2024). doi.org/10.48550/arXiv.2401.18059

2024 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [20]

doi.org/10.5555/12345678

Rahul: A Guide to Chunking Strategies for Retrieval Augmented Generation (RAG). doi.org/10.5555/12345678

doi:10.5555/12345678

[2] [1]

A., Löser, A.: SECTOR: A Neural Model for Coherent Topic Segmentation and Classification

Arnold, S., Schneider, R., Cudré-Mauroux, P., Gers, F. A., Löser, A.: SECTOR: A Neural Model for Coherent Topic Segmentation and Classification. Transactions of the Association for Computational Linguistics, 7:169–184 (2019). doi.org/10.1162/ tacl_a_00261

work page 2019

[3] [2]

In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)

Barrow, J., Jain, R., Morariu, V., Manjunatha, V., Oard, D., Resnik, P.: A Joint Model for Document Segmentation and Segment Labeling. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 313–322 (2020). doi.org/10.18653/v1/2020.acl-main.29

work page doi:10.18653/v1/2020.acl-main.29 2020

[4] [3]

In: Machine Learning 34, 177–210 (1999)

Beeferman, D., Berger, A., Lafferty, J.: Statistical Models for Text Segmentation. In: Machine Learning 34, 177–210 (1999). doi.org/10.1023/A:1007506220214

doi:10.1023/a:1007506220214 1999

[5] [4]

In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading Wikipedia to Answer Open- Domain Questions. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870–1879 (2017). doi. org/10.48550/arXiv.1704.00051

arXiv doi:10.48550/arxiv.1704.00051 2017

[6] [5]

A., Gardner, M.: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., Gardner, M.: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In: Proceedings of the 2021 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 4599–4610 (2021). doi.org/10.18653/v1/2021.naacl...

doi:10.18653/v1/2021.naacl-main.365 2021

[7] [6]

doi.org/10.48550/arXiv.2404.16130

Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Larson, J.: From Local to Global: A Graph RAG Approach to Query-Focused Summariza- tion (2024). doi.org/10.48550/arXiv.2404.16130

arXiv doi:10.48550/arxiv.2404.16130 2024

[8] [7]

doi.org/10.48550/arXiv.2312.10997

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., Wang, H.: Retrieval-augmented generation for large language models: A survey (2023). doi.org/10.48550/arXiv.2312.10997

arXiv doi:10.48550/arxiv.2312.10997 2023