Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a sparse, PageRank-based retrieval algorithm called MixPR matches or beats long-context LLMs on long-document benchmarks while running entirely on CPU in seconds.

desk verdict Solid cheap retrieval method with overclaimed SOTA: the paper's own tables contradict its headline, but the core MixPR algorithm and CPU-speed results are worth refereeing. read the letter →

arxiv 2412.06078 v1 pith:5UXBQDDY submitted 2024-12-08 cs.IR cs.LG

Nicholas Alonso , Beren Millidge This is my paper · ORCID
classification cs.IRcs.LG
keywords mixture-of-PageRankspersonalizedPageRankretrieval-augmentedgenerationlong-contextbenchmarkssparseTF-IDFembeddingsgraph-basedretrievalmulti-hopglobal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the expensive step of feeding a million tokens to a long-context LLM can be replaced by a cheap retrieval step that loses little or no accuracy on many long-document tasks. Its MixPR retriever builds a graph over text chunks from sparse TF-IDF embeddings and runs personalized PageRank on it, mixing a query-focused expert for local tasks with a pure PageRank expert for whole-document tasks. Tested on 22 sub-tasks of BABILong, RULER, Hash-Hop, and infinite-bench, MixPR with GPT-4o as generator beats the base LLM and prior RAG baselines, and matches or exceeds specialized fine-tuned architectures on several benchmarks. Because the embeddings and graph operations are sparse, chunking, embedding, and retrieval take only seconds on a CPU even at a million tokens. If the claims hold, long-context processing becomes dramatically cheaper and feasible on-device.

What carries the argument

The central object is the mixture-of-PageRanks (MixPR) retriever, built on personalized PageRank (PPR), the steady-state distribution $\pi = (1-\alpha) A \pi + \alpha p$ over text chunks. The graph adjacency $A$ is the column-normalized inner product of sparse TF-IDF chunk embeddings, so edges encode keyword overlap between chunks; the personalization vector $p$ is a one-hot distribution over the last chunk(s), where the query is located; and $\alpha$ controls the teleport probability, i.e., how far random walks drift toward query-related nodes versus the graph's structural importance. The core twist is a mixture of two experts—PPR with $\alpha \approx 0.6$ for local, query-dependent retrieval and pure PageRank with $\alpha = 0$ for global, query-independent retrieval—chosen by an LLM router that reads the first and last two chunks of the input. Sparse matrix storage and multiplication let the whole pipeline run on CPU in seconds.

What would settle it

Take any long-context document, move the query to the beginning or the middle of the input instead of the end, and measure MixPR's retrieval recall. If performance collapses when the query is not the final chunk, the reported success depends on the benchmark formatting assumption rather than on a general ability to retrieve from long documents.

Watch

Extended reading notes

Core claim

The paper's central claim is that a graph-based retriever with no neural embedding step can reach state-of-the-art performance on long-context benchmarks at a fraction of the compute of full-context LLM inference. MixPR constructs an adjacency matrix $A = \text{normalize}(E^\top E)$ from L2-normalized TF-IDF chunk embeddings, sets the personalization vector $p$ to a one-hot distribution over the final chunk(s) where the query sits, and computes the personalized PageRank distribution $\pi = (1-\alpha) A \pi + \alpha p$. The teleport parameter $\alpha$ tunes how much the ranking follows query relatedness versus structural importance; tests show large $\alpha$ ($0.5$–$0.99$) is needed for local, query-dependent tasks and $\alpha = 0$ (pure PageRank) is best for global, query-independent tasks like summarization and word-frequency estimation. MixPR routes between these two experts using the LLM generator with a zero-shot prompt, and reports near-perfect classification on the 22 tested sub-tasks. On the BABILong, RULER, Hash-Hop, and English-summarization benchmarks, GPT-4o with MixPR is first on RULER, second on BABILong, and near the top on Hash-Hop, while the sparse implementation processes millions of tokens in seconds entirely on CPU.

Load-bearing premise

The method assumes the query is the last text in the input, because the personalized PageRank walk always starts from the final chunk; if the query appears at the beginning or in the middle, retrieval is biased toward the wrong region.

Editorial extensions

If this is right

  • GPT-4o with MixPR-RAG achieves the top RULER score and second place on BABILong, beating full-context long-context LLMs on these benchmarks.
  • The same retriever handles both single-hop QA and multi-hop reasoning, and matches a Hash-Hop model fine-tuned specifically on the task.
  • Because it runs entirely on CPU and processes millions of tokens in seconds, MixPR can run on-device and in parallel with an LLM on the GPU.
  • Removing irrelevant chunks via retrieval improves LLM accuracy over full-context processing at longer input lengths on many sub-tasks.
  • Chronological ordering of retrieved chunks is essential; MixPR inherits this and therefore outperforms rank-ordered nearest-neighbor baselines on multi-hop tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the query-at-the-end assumption is relaxed, the personalization vector could be derived from the query's own embedding rather than the final chunk, which would extend MixPR to documents where the query comes first or appears mid-text.
  • The $\alpha$-router is a zero-shot LLM classification; a lightweight trained classifier or a query-length heuristic could replace it without the LLM generator, making the retriever self-contained.
  • Because the graph edges are keyword overlap of TF-IDF, MixPR inherits TF-IDF's weakness on synonymy and paraphrase; replacing TF-IDF with a learned sparse embedding that preserves CPU speed could close that gap.
  • The two-expert mixture suggests a general principle: long-context retrieval should separate query-anchored and document-global signals rather than relying on one ranking function.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MixPR, a retrieval-augmented generation (RAG) method that replaces dense embeddings and nearest-neighbor search with sparse TF-IDF embeddings and a mixture of personalized PageRank (PPR) experts. The method dynamically switches between a query-biased PPR (alpha = 0.6) for local retrieval tasks and pure PageRank (alpha = 0) for global retrieval tasks, with the switch performed by an LLM-based classifier. The authors report that MixPR-RAG achieves state-of-the-art results on BABILong, RULER, and Hash-Hop, and is far cheaper than dense RAG and full-context LLM inference, embedding and retrieving millions of tokens on CPU in seconds.

Significance. If the central claims hold, the paper would make a useful practical contribution: a CPU-only, sparse retrieval pipeline that handles multi-hop and global long-context tasks better than standard nearest-neighbor RAG and at a fraction of the compute of long-context LLMs. The strengths of the work include the sparse-matrix implementation, the clear separation of local versus global retrieval tasks, the comparison against non-fine-tuned RAG baselines, and the compute-time measurements. However, the headline 'state-of-the-art' and 'outperforming specialized retrieval architectures' claims are contradicted by the paper's own tables, and several hyperparameters are tuned directly on the target benchmarks. The significance of the contribution is therefore real but more modest than the abstract and conclusion suggest.

major comments (4)
  1. [Abstract and Conclusion (Section 6)] The abstract states that MixPR 'achieves state-of-the-art results across a wide range of long-context benchmark tasks, outperforming both existing RAG methods, specialized retrieval architectures, and long-context LLMs,' and Section 6 repeats that MixPR 'achieves SOTA results on BABILong, HashHop, and RULER.' These claims are internally inconsistent with the paper's own Table 1: on BABILong, the fine-tuned specialized architecture ARMT averages 94.8 over all lengths, while GPT-4o+MixPR-RAG averages 80.4, a 14.4-point gap. Similarly, Table 3 shows LTM-Magic.dev (a fine-tuned model) reaching 100 at most lengths up to 16M, while GPT-4o-mini+MixPR-RAG scores 97. The claims should be tempered to 'state-of-the-art among non-fine-tuned RAG methods' or 'competitive with fine-tuned specialized architectures on several benchmarks.'
  2. [Section 4 and Figure 3; Appendix A.2 and Figure 10] The key hyperparameters are selected on the same benchmarks used for the final evaluation. The alpha values (0.6 for local, 0 for global) are chosen based on Figure 3, which tests PPR recall on RULER and infinite-bench tasks; k=100 is chosen based on Figure 10, which tests MixPR on RULER and BABILong; and the adjacency sparsification threshold of 0.27 (Appendix A.2) is not justified beyond the specific datasets. This tuning protocol means the reported 'dynamic alpha' result is partly fitted to the target tasks. The paper should report how robust the results are to alpha, k, and the sparsification threshold, or evaluate on held-out tasks not used in any sweep, and disclose the full tuning procedure.
  3. [Section 4, Personalization Vector] The personalization vector p is defined as a one-hot vector over the last chunk(s) of the input, with the justification 'In all the datasets we studied, the last sentence or two consist of all or most of the query' (Section 4). This is an assumption about benchmark formatting rather than a property of long-context documents in general: in many real applications, the query appears at the start or in an arbitrary position. Under that placement, the PageRank walk would start from the wrong region and retrieval would be biased away from the relevant content. The paper should either generalize the personalization mechanism to locate the query anywhere in the input, or explicitly state this limitation and provide experiments with non-query-at-end inputs.
  4. [Table 2 and Section 5, Comparisons to SOTA] The RULER comparison in Table 2 is not strictly apples-to-apples: the footnote states that GPT-4o was tested on a subset of RULER (75 questions for most tasks and 300 for QA) due to high costs, while the leaderboard numbers for the other models are presumably full evaluations. With the variance acknowledged in the text ('there is very little variance ... on all tasks except QA, which varies more significantly'), the claim that GPT-4o+MixPR-RAG is 'first' on RULER needs confidence intervals or a full 500-question evaluation before it can be taken as a SOTA result. This issue affects the central 'SOTA on RULER' claim.
minor comments (5)
  1. [Abstract] The word 'efficent' should be 'efficient'.
  2. [Section 4] The author name 'V odrahelli' appears with an odd space; it should be 'Vodrahalli' (as spelled in the references).
  3. [Section 5, Datasets] The sentence 'In total, we test on 22 tasks with 14 of these being single-hop local retrieval tasks, 8 multi-hop retrieval tasks, and 3 global retrieval tasks' sums to 25, not 22. Please reconcile the task counts or the categorization.
  4. [Appendix A.2] The adjacency sparsification ('removing all values below .27') is described in the appendix but used in the main experiments; this details belongs in Section 4 where Equation (3) is defined, so that the main text is self-contained.
  5. [Figure 6] The compute-time comparison would be more informative if the figure or text stated whether the reported MixPR time includes the PPR power iterations (max 18) and the router LLM call, and if the software versions (scikit-learn, SciPy, FAISS) were given for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MixPR is an empirical retrieval system; hyperparameters are tuned rather than derived, and self-citations are not load-bearing.

full rationale

The paper does not claim a formal derivation of MixPR from first principles; it presents an engineering system validated on benchmarks. The PPR equation π = (1−α)Aπ + αp is standard and is not used to predict the benchmark results by construction. The personalization vector p is set to the last chunk(s) because queries appear at the end of the tested inputs; this is an explicit benchmark-format assumption, not a circular reduction. The dynamic-α router is motivated by sweeps in Figure 3, and the choices α = 0.6 for local tasks and α = 0 for global tasks are reported as tuned hyperparameters, not as fitted quantities renamed as predictions; the headline comparisons span many models and tasks and do not reduce to these settings by any equation. The self-citation [2] (Alonso and Millidge) is used to motivate graph-based retrieval and chronological ordering, but the paper re-derives these conclusions empirically in Figure 4 and also cites independent work [42] for chronological ordering, so the citation is not load-bearing. The abstract's SOTA claim is internally contradicted by Table 1 (ARMT beats GPT-4o+MixPR on BABILong), but that is a correctness and consistency concern, not circularity. No circular step meets the required standard of exhibiting a specific reduction by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on several domain assumptions about benchmark formatting and on hand-selected hyperparameters; no new physical or ontological entities are introduced.

free parameters (6)
  • alpha (PPR personalization weight) = .6 for local tasks; 0 for global tasks
    Selected after testing alpha values on the same benchmark-style tasks used for final evaluation (Figure 3).
  • adjacency sparsification threshold = 0.27
    Values below 0.27 are removed from A = E^T E to create a sparse graph (Appendix A.2).
  • chunk size N = 32 words
    Set for all experiments; chunking method depends on N and affects retrieval.
  • top-k chunks = 100 default; swept in K-analysis
    k=100 chosen as a sweet spot on RULER and BABILong; other values appear in appendix Figure 10.
  • personalization heuristic = one-hot on last chunk; 0.5/0.5 on last two when last chunk is short
    Hand-designed rule for p; assumes the query lives at the end of the input.
  • PPR max iterations = 18
    Implementation detail in Appendix A.2; truncates the power iteration used for retrieval.
assumptions (5)
  • domain assumption The column-normalized cosine-similarity matrix A = normalize(E^T E) yields a Markov chain whose stationary distribution is a useful measure of chunk importance.
    This is the core modeling assumption; TF-IDF word overlap is used as a proxy for semantic relation, with no neural verification.
  • domain assumption The query is contained in the final chunk or chunks of the input.
    Stated in Section 4: 'In all the datasets we studied, the last sentence or two consist of all or most of the query.' This makes the one-hot personalization vector point at the query.
  • domain assumption Every long-context query can be classified as either local or global from the first and last two sentences.
    The LLM router uses only these sentences; Figure 11 reports near-perfect accuracy on the 38 tested queries.
  • standard math Power iteration of the PPR recurrence converges within 18 iterations for these graphs.
    Implementation detail in Appendix A.2; the networkx-based iteration is truncated at 18 steps.
  • domain assumption TF-IDF embeddings are sufficient to build edges for retrieval on these benchmarks.
    The paper deliberately avoids neural embedders to stay on CPU; this trades away semantic sensitivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG." pith.science (2026). https://pith.science/paper/5UXBQDDY

@misc{pith2026241206078,
  author       = {Pith},
  title        = {Pith review of: Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5UXBQDDY}},
  note         = {Machine review of arXiv:2412.06078}
}
read the original abstract

Recent advances have extended the context window of frontier LLMs dramatically, from a few thousand tokens up to millions, enabling entire books and codebases to fit into context. However, the compute costs of inferencing long-context LLMs are massive and often prohibitive in practice. RAG offers an efficient and effective alternative: retrieve and process only the subset of the context most important for the current task. Although promising, recent work applying RAG to long-context tasks has two core limitations: 1) there has been little focus on making the RAG pipeline compute efficient, and 2) such works only test on simple QA tasks, and their performance on more challenging tasks is unclear. To address this, we develop an algorithm based on PageRank, a graph-based retrieval algorithm, which we call mixture-of-PageRanks (MixPR). MixPR uses a mixture of PageRank-based graph-retrieval algorithms implemented using sparse matrices for efficent, cheap retrieval that can deal with a variety of complex tasks. Our MixPR retriever achieves state-of-the-art results across a wide range of long-context benchmark tasks, outperforming both existing RAG methods, specialized retrieval architectures, and long-context LLMs despite being far more compute efficient. Due to using sparse embeddings, our retriever is extremely compute efficient, capable of embedding and retrieving millions of tokens within a few seconds and runs entirely on CPU.

Figures

Figures reproduced from arXiv: 2412.06078 by the authors.

Figure 1
Figure 1. Comparison of gpt-4o-mini and RAG systems (@k=100) that use gpt-4o-mini as the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. RAG Methods for Long-Context Tasks. Previous works [2, 42] have shown that chronologically ordering text chunks, rather than rank ordering, is necessary for many long-context tasks. Standard nearest-neighbor RAG (top) retrieve items most similar to the query (query-relatedness is depicted with red embeddings). Our PageRank-based retrievers represent relations between text chunks using a similarity matrix that can be… view at source ↗
Figure 3
Figure 3. PPR Retriever Alpha Test. (Left) The recall accuracy of a PPR retriever at k=100, under various alpha settings, for a set of query-dependent, local retrieval tasks. (Right) The recall performance of the same PPR retrieval at k=100, for query-independent, global retrieval tasks. answering (QA)) require retrieving items that have higher query-relatedness, while global retrieval tasks (such as summarization) require re… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance on multi-hop retrieval tasks. Results from benchmarks on the subset of tasks that involve multi-hop retrieval: BABILong question types 2 and 3, Hash-Hop with 2-6 hash links, and the variable tracing task from RULER. All RAG models tested with k=100. The non…
Figure 5
Figure 5. Figure 5: Performance on global retrieval tasks. (Left) Performance of various RAG models with k=100 averaged across the cwe and fwe tasks from RULER. (Right) Performance of various RAG models on the Eng.Sum task from infinite-bench. Models test with k of 100, 200, 300, 400, 500…
Figure 6
Figure 6. Figure 6: RAG compute times in compute constrained setting. Total time for RAG systems to chunk, embed, and retrieve from a long context. Retrieval systems are run on hardware available in high-end desktops: CPU with 16 cores and a single RTX 4090 GPU. We compare our MixPR retri…
Figure 7
Figure 7. Figure 7: Performance on one-hop retrieval tasks. Results on the subset of tasks that require one-hop retrieval: BABILong question types 1, 3 and 5, Hash-Hop with 1 hash link, and the needle￾in-the-haystack and QA tasks from RULER. All RAG models tested with k=100. All models pe…
Figure 8
Figure 8. Figure 8: BABILong Full Results GPT-4o-mini. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: BABILong Full Results Llama3.1-8B [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: K-Analysis. Test different K values for MixPR on RULER (all lengths) and BABILong (0k to 1M) using gpt-4o-mini as the LLM generator. For these tests, which consist mainly of local retrieval tasks, there is a sweet spot around k=100, after which noise/irrelevant text h…
Figure 11
Figure 11. Figure 11: Classification Accuracies. The MixPR RAG model uses the base LLM to classify queries as either requiring global or local retrieval using a prompting method. Here we show the classification accuracy across queries from every sub-task (22 total) from the benchmarks in t…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework

    cs.IR 2025-11 conditional novelty 6.0 of 10

    TeaRAG shows that hybrid chunk+triplet retrieval with Personalized PageRank and an iterative process-aware DPO reward keeps QA accuracy while cutting reasoning tokens by roughly 60%.

Reference graph

Works this paper leans on

45 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Understanding graph-based rag and multi-hop question an- swering

    Nick Alonso and Beren Millidge. Understanding graph-based rag and multi-hop question an- swering. https://www.zyphra.com/post/understanding-graph-based-rag-and-multi-hop-question- answering, 2024

    work page 2024

  3. [3]

    Model card and evaluations for claude models, 2023

    AI Anthropic. Model card and evaluations for claude models, 2023

    work page 2023

  4. [4]

    The anatomy of a large-scale hypertextual web search engine

    Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. Computer networks and ISDN systems, 30(1-7):107–117, 1998

    1998

  5. [5]

    An analysis of fusion functions for hybrid retrieval

    Sebastian Bruch, Siyu Gai, and Amir Ingber. An analysis of fusion functions for hybrid retrieval. ACM Transactions on Information Systems, 42(1):1–35, 2023. 10

    work page 2023

  6. [6]

    Recurrent memory transformer.Advances in Neural Information Processing Systems, 35:11079–11091, 2022

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer.Advances in Neural Information Processing Systems, 35:11079–11091, 2022

    2022

  7. [7]

    Adapting language models to compress contexts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint arXiv:2305.14788, 2023

    arXiv 2023

  8. [8]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024

    arXiv 2024

Show all 45 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    From local to global: A graph rag approach to query-focused summarization

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024

    2024 arXiv

  3. [11]

    Human-like episodic memory for infinite context llms

    Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context llms. arXiv preprint arXiv:2407.09450, 2024

    2024

  4. [12]

    Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp

    Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. arXiv preprint arXiv:2407.00402, 2024

    2024 arXiv

  5. [13]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  6. [14]

    Lightrag: Simple and fast retrieval-augmented generation

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. Lightrag: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779, 2024

    2024 arXiv

  7. [15]

    Hipporag: Neurobiologically inspired long-term memory for large language models

    Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. arXiv preprint arXiv:2405.14831, 2024

    2024 arXiv

  8. [16]

    Exploring network structure, dynamics, and function using networkx

    Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 2008

    2008

  9. [17]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  10. [18]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  11. [19]

    Needle in a haystack - pressure testing llms

    Kamradt. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/ LLMTest_NeedleInAHaystack/blob/main/README.md, 2023

    2023

  12. [20]

    One thou- sand and one pairs: A" novel" challenge for long-context language models

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thou- sand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024

    2024 arXiv

  13. [21]

    Extended mind transformers

    Phoebe Klett and Thomas Ahle. Extended mind transformers. arXiv preprint arXiv:2406.02332, 2024

    2024 arXiv

  14. [22]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. arXiv preprint arXiv:2406.10149, 2024

    2024 arXiv

  15. [23]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models.arXiv preprint arXiv:2402.14848, 2024

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models.arXiv preprint arXiv:2402.14848, 2024. 11

    2024 arXiv

  16. [24]

    Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024

    Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024

    2024

  17. [25]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  18. [26]

    Efficient algorithms for personalized pagerank

    Peter Lofgren. Efficient algorithms for personalized pagerank. Stanford University, 2015

    2015

  19. [27]

    https://magic.dev/blog/100m-token-context-windows, 2024

    Magic. https://magic.dev/blog/100m-token-context-windows, 2024

    2024

  20. [28]

    https://github.com/NVIDIA/RULER

    NVIDIA. https://github.com/NVIDIA/RULER

  21. [29]

    Scikit- learn: Machine learning in python

    Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830, 2011

    2011

  22. [30]

    Graph retrieval-augmented generation: A survey

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921, 2024

    2024 arXiv

  23. [31]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

    2023 arXiv

  24. [32]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019

    1911 arXiv

  25. [33]

    Babilong leaderboard

    RMT-Team. Babilong leaderboard. https://huggingface.co/spaces/RMT-team/babilong

  26. [34]

    Associative recurrent memory transformer

    Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer. arXiv preprint arXiv:2407.04841, 2024

    2024 arXiv

  27. [35]

    Term-weighting approaches in automatic text retrieval

    Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513–523, 1988

    1988

  28. [36]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  29. [37]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  30. [38]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, E...

    2020

  31. [39]

    Michelan- gelo: Long context evaluations beyond haystacks via latent structure queries

    Kiran V odrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelan- gelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:240...

    2024 arXiv

  32. [40]

    Towards ai-complete question answering: A set of prerequisite toy tasks

    Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015

    2015 arXiv

  33. [41]

    Infllm: Unveiling the intrinsic capacity of llms for under- standing extremely long sequences with training-free memory.arXiv preprint arXiv:2402.04617, 2024

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of llms for under- standing extremely long sequences with training-free memory.arXiv preprint arXiv:2402.04617, 2024. 12

    2024 arXiv

  34. [42]

    In defense of rag in the era of long-context language models

    Tan Yu, Anbang Xu, and Rama Akkiraju. In defense of rag in the era of long-context language models. arXiv preprint arXiv:2409.01666, 2024

    2024 arXiv

  35. [43]

    Long context compression with activation beacon

    Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon. arXiv preprint arXiv:2401.03462, 2024

    2024 arXiv

  36. [44]

    Infinitebench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. Infinitebench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling...

    2024

  37. [45]

    latent structure

    Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, and Jie Tang. Longrag: A dual-perspective retrieval-augmented generation paradigm for long-context question answering. arXiv preprint arXiv:2410.18050, 2024. A Appendix A.1 Extended Related Works Long...

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.