Pith. sign in

REVIEW 4 major objections 5 minor 104 references

MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a 237-sample subset of LongBench can rank LLMs on long-context understanding with 0.97 rank correlation while costing about 4.5% of the original.

desk verdict Useful benchmark artifact with a solid empirical core, but the paper hides a load-bearing detail (how cluster centers become text samples) and picks hyperparameters on the evaluation set. read the letter →

arxiv 2505.19959 v2 pith:K6VWCTVS submitted 2025-05-26 cs.CL

Zhongzhan Huang , Guoming Ling , Shanshan Zhong , Hefeng Wu , Liang Lin This is my paper
classification cs.CL
keywords long-contextunderstandingLLMevaluationbenchmarkcompressionLongBenchMiniitemresponsetheoryrankcorrelationdatapruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large long-context understanding benchmarks like LongBench are heavily redundant: most of their roughly 4,750 test items add little to how LLMs are ranked. To exploit that redundancy, it builds MiniLongBench, a 237-sample benchmark covering the same six task categories and 21 tasks, by embedding each long text, compressing to ten dimensions, learning sample representations from the pass/fail records of 20 LLMs, and keeping one representative per cluster. On more than 60 LLMs, MiniLongBench is reported to rank models nearly identically to LongBench (average Spearman correlation 0.97) while cutting evaluation cost to about 4.5% of the original. The paper also proposes a cheap scoring procedure: test a new model on the 237 samples, fine-tune a small logistic-regression vector on those results, and use it to estimate the full LongBench score. The authors note that building the benchmark is itself costly because it requires performance records from many LLMs, which are often not open-source.

What carries the argument

The mechanism is an item-response-style logistic model over LLM performance records. Each test sample is represented by a 10-dimensional vector $(e_j, \beta_j)$: $e_j$ is initialized from a text embedding plus PCA and then refined jointly with per-model ability vectors $\theta_i$, and $\beta_j$ is a learnable difficulty/bias term. The probability that model $i$ answers sample $j$ correctly is a logistic function of $e_j^\top \theta_i - \beta_j$. After fitting this model on $m=20$ LLMs' pass/fail outcomes across LongBench, the samples are clustered in that learned representation space with K-means, and the K cluster centers become the stored MiniLongBench items. For a new model, the same logistic form is used to connect its observed performance on the mini samples to an estimate of its score on all original LongBench samples. This representation learning is what distinguishes the compression from random sampling: it projects sparse long texts into a dense low-dimensional performance space so that clustering finds genuinely representative items.

What would settle it

Run the released MiniLongBench on a held-out set of 20 LLMs that were not used in training, compute Spearman correlation between their MiniLongBench-estimated and LongBench scores, and check whether the average Sp is materially below 0.97; additionally, inspect whether every stored cluster center maps to a real, scorable text item, since without such a mapping the evaluation procedure is undefined.

Watch

Extended reading notes

Core claim

The central claim is that a carefully pruned subset of LongBench preserves the benchmark's ranking behavior, so the expensive full benchmark is not needed to compare LLMs on long-context understanding. The authors show that randomly removing 95% of LongBench already produces subsets with Spearman correlations above 0.8, evidence of redundancy, and then replace random sampling with a deterministic compression. The resulting MiniLongBench contains 237 test samples; direct evaluation on these samples yields about 0.95 rank correlation with LongBench, and the paper's estimator, which fine-tunes a model-specific vector against the mini samples and averages the logistic predictions over all original LongBench samples, raises the average to 0.97. The authors report per-task correlations above 0.83 in all six categories and statistical significance with p < 0.001. The intended use is ranking, not exact score replication: the paper states that absolute scores can shift while rankings remain reliable.

Load-bearing premise

The load-bearing assumption is that each K-means cluster center in the learned ten-dimensional space can be turned back into a concrete long-context test item that the original scoring metric can evaluate; the paper never states how a vector center is mapped to such a text.

Editorial extensions

If this is right

  • Routine LLM development can use MiniLongBench instead of LongBench for long-context checkpoints, cutting a 15-30 hour evaluation to under an hour on the same hardware.
  • The same compression recipe can be applied to other long-context benchmarks, not just LongBench, whenever a set of model performance records is available.
  • Model rankings from MiniLongBench are stable enough (per-task Sp at least 0.83, overall 0.97) to support comparing models that were never seen during benchmark construction.
  • The stored representation and fine-tuning overhead are small enough (about 10 MB, roughly 0.03 seconds) that the estimated LongBench score can be recomputed on a laptop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's estimator predicts a model's score on every original LongBench item, the same 237 samples could be used to estimate performance on task subsets or on newly added items without running the full benchmark, as long as the logistic representation remains valid.
  • The method's dependence on thresholding continuous metrics into binary outcomes discards graded information; a graded extension of the logistic model might recover higher rank fidelity on summarization and synthetic tasks, where the paper itself reports the lowest correlations.
  • One open risk the authors do not address is whether the cluster centers correspond to naturally readable long texts: if a center is an average of dissimilar documents, scoring it with the original metric may be ill-defined. A testable fix is to select the nearest real LongBench sample to each center instead of using the center itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. MiniLongBench compresses the LongBench long-context benchmark from 4,750 samples to 237 samples. The construction pipeline first embeds texts with OpenAIEmbedding, reduces dimension with PCA, then fits a logistic regression model (Eq. 2) on the performance records of 20 LLMs to obtain sample representations (e_j, β_j). K-means is applied to these representations, and the cluster centers are declared to be the MiniLongBench test samples. The paper reports that, across more than 60 LLMs, MiniLongBench reproduces LongBench's model ranking with Spearman correlation 0.97 while reducing evaluation cost to 4.5%; a direct evaluation mode yields 0.95 (Appendix G). Ablations study the compression dimension d, compression ratio p, embedding choice, PCA, bias β_j, and the number and selection of training LLMs.

Significance. If the pipeline is reproducible, MiniLongBench would be a valuable low-cost benchmark for long-context evaluation. The paper's strengths include the large empirical matrix of 60+ LLMs, the release of code, data and a tutorial, and the explicit redundancy analysis via random sampling. The direct evaluation mode's Spearman correlation of 0.95 with LongBench is an impressive result in itself. However, the central claim depends on an unspecified step—decoding cluster-center vectors back into scoreable text items—and on hyperparameters selected using the same LLMs later used for validation. These issues prevent the claims from being fully assessable from the manuscript as written.

major comments (4)
  1. [Section 3, Algorithm 1, and Section 4.2] The step from cluster centers to usable benchmark items is undefined. Algorithm 1 returns Smini = (c1, ..., cK), where each c_k is a cluster center of the concatenated representations (e_j, β_j). Section 4.2 then says a new LLM ℓ0 is evaluated on 'all test samples cj from MiniLongBench Smini' with metric metr(ℓ0, cj), but a vector in R^11 is not a long-context input and cannot be scored by F1, Rouge-L, edit similarity, or exact match. The paper never states how a center is mapped to a concrete (context, question, answer) instance or to a newly constructed textual item. This is not a presentation detail: without this mapping, the evaluation procedure in Section 4.2, Eq. (4), and the released benchmark are undefined. Please specify the decoding rule (e.g., nearest original LongBench sample to each center, or synthesis from nearest neighbors) and justify that it preserves the task structure and metrics.
  2. [Section 5, Figs. 6, 7, and 9] Hyperparameters d=10, p=0.95, the PCA on/off decision, and the embedding choice are selected using Spearman correlations computed over the full set of LLMs, including the models later designated as 'Test' in Fig. 5. For example, Fig. 6 reports the effect of d with separate Train and Test Sp curves, and the paper chooses d=10 based on this plot. This is a selection-on-the-test-set procedure, so the reported held-out Sp values are optimistically biased as estimates of how MiniLongBench will perform on truly new models. Please either carry out hyperparameter selection using only the 20 training LLMs and reserve the remaining models purely for final evaluation, or present the selection procedure as part of the benchmark construction and temper the generalization claim accordingly.
  3. [Section 4.2, Eq. (4), Fig. 5] The headline Sp=0.97 is produced by a fine-tuned logistic-regression estimator that uses the original LongBench sample representations (e_j, β_j) and the binarized performance records of the 20 training LLMs. It is not the Spearman correlation of the raw scores on the 237 MiniLongBench items, which is 0.95 (Appendix G, Fig. 13). The abstract and Section 4.3 should state this distinction explicitly; the current wording 'MiniLongBench achieves ... 0.97' conflates the benchmark itself with the accompanying prediction procedure. If the intended claim is that the 237 items alone reproduce LongBench rankings, the correct number is 0.95.
  4. [Algorithm 1 and Table 1] There is an inconsistency between the stated compression rule and the reported per-task counts. Algorithm 1 sets K=(1−p)|SL| with p=0.95, which for LongBench's 4,750 samples gives approximately 237 centers; applied per task, this rule would yield about 10 of 200 samples for most tasks, while Table 1 reports counts such as 6/200 for NarrativeQA, 13/200 for HotpotQA, and 26/500 for LCC. Please clarify whether clustering is performed per task, globally, or with a different allocation, and explain how a global center belongs to the task labels in Table 1. This matters for reproducibility of the 237-sample benchmark.
minor comments (5)
  1. [Section 5, first paragraph] The phrase 'In Eq. (1) of Session 3' should read 'Section 3'.
  2. [Table 1 and Table 3] There are typos: 'Python/Jave' should be 'Python/Java', and 'Wizaed-Vicuna' should be 'Wizard-Vicuna'; the appendix tables also use 'Few-show' where 'Few-shot' is meant.
  3. [Section 3, Eq. (3)] The binarization threshold c is chosen by matching the sum of binarized scores to the sum of normalized scores; because this is a global rescaling, it may distort per-sample difficulty information used by the logistic regression. Please discuss why a per-sample or rank-preserving threshold is not needed.
  4. [Appendix F, Figs. 10 and 12] The ranking visualizations on 8 LLMs and 16 random trials are anecdotal; reporting the distribution of Sp across the random trials (for instance, with boxplots) would better support the claim of consistent ranking agreement.
  5. [Appendix B, Table 3] The distinction between 'T' and 'A' LLMs is unclear: some entries marked T are small or weak models (e.g., OLMo-1B) while the text says the 20 training LLMs were chosen for 'generally good performance across various aspects.' Please state the exact selection criterion or explain the discrepancy.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline 0.97 correlation is partly a fitted-prediction artifact, but direct evaluation (Sp=0.95) gives independent support; the center-to-text mapping is an omitted definition rather than circularity.

  1. fitted input called prediction [Section 4.2 (Eq. 4), relying on Eq. (2) fitted in Section 3]
    "To mitigate this, we can use MiniLongBench samples to estimate the performance (Polo et al., 2024; Pacchiardi et al., 2024) of the LLMs on LongBench, thereby reducing bias and achieving an improved Sp of up to 0.97."

    The 'improved Sp of up to 0.97' is obtained by Eq. (4), which averages [1+exp(-e_j^T theta + beta_j)]^{-1} over all LongBench items. Here (e_j, beta_j) are exactly the item parameters fitted in Section 3 by logistic regression on the 20 training LLMs' observed metr(l_i, s_j) over LongBench, and the 237 MiniLongBench centers are clusters of those fitted representations. Thus the predicted LongBench score is a function of a model trained on LongBench performance labels, so the headline correlation is partly a self-consistency check of the fitted IRT model rather than an independent property of the 237 texts. The held-out Test Sp=0.96 (Fig. 5) and direct-evaluation Sp=0.95 (Appendix G) provide independent evidence, so the circularity is partial.

full rationale

MiniLongBench's construction and evaluation are largely self-contained and benchmarked against LongBench on held-out LLMs; I found no load-bearing self-citation chain or imported uniqueness theorem. The only partial circularity is the headline 0.97: Eq. (4) predicts LongBench scores using (e_j, beta_j) that were fit in Section 3 to the 20 training LLMs' LongBench performance records, so the 0.97 is partly a self-consistency measure of that fitted IRT model. However, Fig. 5 reports Test Sp=0.96 on unseen LLMs, and Appendix G's direct evaluation (no Eq. 4) gives Sp=0.95, so the 237-sample benchmark has independent empirical content; the fitted-prediction step improves but does not create the correlation. The more serious issue, that cluster centers are defined as vectors while Section 4.2 evaluates metr(l0, c_j) on them as if they were texts, is an undefined-mapping and reproducibility gap rather than circularity, so it does not raise the circularity score.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on six fitted hyperparameters and four domain assumptions. No new physical or mathematical entities are introduced. The free parameters are selected using the same LLM evaluation data that later appears in the correlation analysis.

free parameters (6)
  • Compression dimension d = 10
    Selected in Section 5(1) based on Fig. 6, where lower d gives higher Spearman correlation on the LLM evaluation set.
  • Compression ratio p = 0.95
    Selected in Section 5(4) based on Fig. 7(a) as a balance between cost and Spearman correlation.
  • Binarization threshold c (Eq. 3) = not stated
    Searched in [0,1] to convert continuous metrics to binary labels; the exact value is not reported.
  • Initialization of beta_j = zero
    Chosen in Section 5(5) based on Fig. 7(c) as the best among zero, uniform, and normal initializations.
  • Number of training LLMs m = 20
    Chosen in Section 5(7) based on Fig. 9 as an acceptable trade-off; Sp increases with m.
  • Number of clusters K = (1-p)|SL|
    Determined by p and the size of each dataset; yields 237 total samples.
assumptions (4)
  • domain assumption LongBench scores are a valid ground-truth measure of long-context understanding.
    The entire evaluation treats LongBench as the reference standard (Section 2, Section 4).
  • domain assumption The logistic model in Eq. (2) captures the probability that an LLM answers a sample correctly.
    Adopted from Item Response Theory (Polo et al., 2024); the paper provides no empirical validation of this functional form.
  • domain assumption Text embeddings (OpenAIEmbedding) plus PCA capture the information needed to cluster samples by performance similarity.
    Section 3(1) and Section 5(2) assert this; the analysis shows only that removing PCA lowers Sp, not that the embeddings are sufficient.
  • standard math Spearman correlation is an appropriate measure of benchmark equivalence.
    Used throughout; the paper does not discuss alternative measures or their limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models." pith.science (2026). https://pith.science/paper/K6VWCTVS

@misc{pith2026250519959,
  author       = {Pith},
  title        = {Pith review of: MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6VWCTVS}},
  note         = {Machine review of arXiv:2505.19959}
}
read the original abstract

Long Context Understanding (LCU) is a critical area for exploration in current large language models (LLMs). However, due to the inherently lengthy nature of long-text data, existing LCU benchmarks for LLMs often result in prohibitively high evaluation costs, like testing time and inference expenses. Through extensive experimentation, we discover that existing LCU benchmarks exhibit significant redundancy, which means the inefficiency in evaluation. In this paper, we propose a concise data compression method tailored for long-text data with sparse information characteristics. By pruning the well-known LCU benchmark LongBench, we create MiniLongBench. This benchmark includes only 237 test samples across six major task categories and 21 distinct tasks. Through empirical analysis of over 60 LLMs, MiniLongBench achieves an average evaluation cost reduced to only 4.5% of the original while maintaining an average rank correlation coefficient of 0.97 with LongBench results. Therefore, our MiniLongBench, as a low-cost benchmark, holds great potential to substantially drive future research into the LCU capabilities of LLMs. See https://github.com/MilkThink-Lab/MiniLongBench for our code, data and tutorial.

Figures

Figures reproduced from arXiv: 2505.19959 by the authors.

Figure 1
Figure 1. The computational cost of LongBench and MiniLongBench. The proposed MiniLongBench effec￾tively reduces the computational cost of the LongBench, thereby achieving a low-cost LCU benchmark. of LLMs are still in their early stages, and their rapid development relies on recent proposals of LCU benchmarks (Shaham et al., 2022, 2023; An et al., 2023; Bai et al., 2024d). However, unlike nor￾mal LLM benchmarks (Li et al., 2… view at source ↗
Figure 2
Figure 2. The redundancy of LongBench. "Reduce 95%" means randomly removing 95% of the dataset with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The compression process of the LCU benchmark. "Emb." and "Per." respectively denote embedding and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The Length distribution for English and Chi [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The analysis of rank correlation (Sp) between [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The impact of compression dimension d on the construction of MiniLongBench. "r" is Pearson correlation coefficient. ferent tasks on the proposed MiniLongBench. For more detailed results, please refer to Appendix C. 5 Analysis In this Section, We conduct a more comprehe…
Figure 8
Figure 8. Figure 8: The impact of the selection of LLMs on the construction of MiniLongBench . [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The impact of the number of LLMs m on the construction of MiniLongBench . although the number of LLMs aids in representa￾tion learning, there is still considerable redundancy. Considering the computational cost, we take the acceptable m = 20 as default. (8) Is the aver…
Figure 10
Figure 10. Figure 10: The visualization of ranking. See more ranking examples in Appendix F. sampling to compress LongBench is insufficient. The primary reason is that while random sampling can probabilistically yield high Sp results, as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: The visualization of learned representation [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: The more examples of visualization of ranking by MiniLongBench and LongBench.. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: The analysis of rank correlation (Sp) between LongBench and MiniLongBench where the result of [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

104 extracted references · 21 canonical work pages

  1. [1]

    Herve Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433--459

    2010

  2. [2]

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018

    arXiv 2024

  3. [3]

    Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Onta \ n \'o n, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. 2023. Colt5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09752

    arXiv 2023

  4. [4]

    Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088

    arXiv 2023

  5. [5]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. https://doi.org/10.18653/v1/2024.acl-long.776 L -eval: Instituting standardized evaluation for long context language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    doi:10.18653/v1/2024.acl-long.776 2024

  6. [6]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Anthropic: Introducing claude 3.5 sonnet

    2024

  7. [7]

    Yushi Bai, Xin Lv, and et al. 2024 a . Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 3119--3137. Association for Computational Linguistics

    2024

  8. [8]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.74 L ong A lign: A recipe for long context alignment of large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376--1395, Miami, Florida, USA. Association...

    doi:10.18653/v1/2024.findings-emnlp.74 2024

Show all 104 references
  1. [9]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long contex...

    2024 doi

  2. [10]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. 2024 d . Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204

    2024 arXiv

  3. [11]

    Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 e . Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055

    2024 arXiv

  4. [12]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    2020 arXiv

  5. [13]

    Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, et al. 2024. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612

    2024 arXiv

  6. [14]

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079--11091

    2022

  7. [15]

    Li Cai, Kilchan Choi, Mark Hansen, and Lauren Harrell. 2016. Item response theory. Annual Review of Statistics and Its Application, 3(1):297--321

    2016

  8. [16]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    2023 arXiv

  9. [17]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509

    2019 arXiv

  10. [18]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978--2988

    2019

  11. [19]

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational ...

    2021

  12. [20]

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. 2023. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486

    2023 arXiv

  13. [21]

    Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language R...

    2024

  14. [22]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for C...

    2019

  15. [23]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  16. [24]

    Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa...

    2019

  17. [25]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232--5270

    2022

  18. [26]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128 K context. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learni...

    2024

  19. [27]

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2024. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing...

    2024

  20. [28]

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660

    2024

  21. [29]

    Team GLM, Aohan Zeng, Bin Xu, and et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  22. [30]

    Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. 2023. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662--59688

    2023

  23. [31]

    Greg Hamerly and Charles Elkan. 2003. Learning the k in k-means. Advances in neural information processing systems, 16

    2003

  24. [32]

    Wei He, Zhongzhan Huang, Mingfu Liang, Senwei Liang, and Haizhao Yang. 2021. Blending pruning criteria for convolutional neural networks. In Artificial Neural Networks and Machine Learning--ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Sl...

    2021

  25. [33]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625

    2020

  26. [34]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  27. [35]

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021 a . Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,...

    2021

  28. [36]

    Zhongzhan Huang, Senwei Liang, Mingfu Liang, Wei He, Haizhao Yang, and Liang Lin. 2022. The lottery ticket hypothesis for self-attention in convolutional neural network. arXiv preprint arXiv:2207.07858

    2022 arXiv

  29. [37]

    Zhongzhan Huang, Wenqi Shao, Xinjiang Wang, Liang Lin, and Ping Luo. 2021 b . Rethinking the pruning criteria for convolutional neural network. Advances in Neural Information Processing Systems, 34:16305--16318

    2021

  30. [38]

    Greg Kamradt. 2023. https://github.com/gkamradt/LLMTest_NeedleInAHaystack Needle in a haystack - pressure testing llms . https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  31. [39]

    Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11

    2024 arXiv

  32. [40]

    Alex Kipnis, Konstantinos Voudouris, Luca M Schulze Buschoff, and Eric Schulz. 2024. metabench--a sparse benchmark to measure general ability in large language models. arXiv preprint arXiv:2407.12844

    2024 arXiv

  33. [41]

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations

    2020

  34. [42]

    David G Kleinbaum, K Dietz, M Gail, Mitchel Klein, and Mitchell Klein. 2002. Logistic regression. Springer

    2002

  35. [43]

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2024. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint arXiv:2409.12941

    2024 arXiv

  36. [44]

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. arXiv preprint arXiv:2406.10149

    2024 arXiv

  37. [45]

    Philippe Laban, Alexander Richard Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context llms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885--9903

    2024

  38. [46]

    Shiye Lei and Dacheng Tao. 2023. A comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2023

  39. [47]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299--13308

    2024

  40. [48]

    Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang

    Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023 a . https://lmsys.org/blog/2023-06-29-longchat How long can open-source llms truly promise on context length?

    2023

  41. [49]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023 b . Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939

    2023 arXiv

  42. [50]

    Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics

    2002

  43. [51]

    Senwei Liang, Zhongzhan Huang, Mingfu Liang, and Haizhao Yang. 2020. Instance enhancement batch normalization: An adaptive regulator of batch noise. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4819--4827

    2020

  44. [52]

    Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. 2023. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. arXiv preprint arXiv:2304.13343

    2023 arXiv

  45. [53]

    Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. 2024. Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  46. [54]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091

    2023 arXiv

  47. [55]

    Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024. Longgenbench: Long-context generation benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 865--883

    2024

  48. [56]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv

    2019

  49. [57]

    Pedro Henrique Martins, Zita Marinho, and Andr \'e FT Martins. 2022. -former: Infinite memory transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5468--5485

    2022

  50. [58]

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. In The Thirty-eighth Annual ...

    2024

  51. [59]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Openai: Hello gpt-4o

    2024

  52. [60]

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349

    2023 arXiv

  53. [61]

    Lorenzo Pacchiardi, Lucy G Cheke, and Jos \'e Hern \'a ndez-Orallo. 2024. 100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563

    2024 arXiv

  54. [62]

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2022. Quality: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American ...

    2022

  55. [63]

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. In Forty-first International Conference on Machine Learning

    2024

  56. [64]

    Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations

    2022

  57. [65]

    Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191

    2024 arXiv

  58. [66]

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations

    2020

  59. [67]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  60. [68]

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53--68

    2021

  61. [69]

    Noveen Sachdeva and Julian McAuley. 2023. Data distillation: A survey. arXiv preprint arXiv:2301.04272

    2023 arXiv

  62. [70]

    Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196

    2023 arXiv

  63. [71]

    Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language...

    2022

  64. [72]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802

    2024 arXiv

  65. [73]

    Charles Spearman. 1961. The proof and measurement of association between two things

    1961

  66. [74]

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554

    2022 arXiv

  67. [75]

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. https://doi.org/10.1145/3530811 Efficient transformers: A survey . ACM Comput. Surv., 55(6)

    2022 doi

  68. [76]

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv \'e J \'e gou. 2021. Training data-efficient image transformers and distillation through attention. In International conference on machine learning, pages 10347--10357. PMLR

    2021

  69. [77]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539--554

    2022

  70. [78]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11)

    2008

  71. [79]

    Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. 2024. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:...

    2024 arXiv

  72. [80]

    Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel Bowman. 2022. Squality: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139--1156

    2022

  73. [81]

    Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. 2024. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. In Proceedings of the 2024 Conference on Empirical Methods in...

    2024

  74. [82]

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768

    2020 arXiv

  75. [83]

    Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. 2024. Longgenbench: Benchmarking long-form generation in long context llms. arXiv preprint arXiv:2409.02076

    2024 arXiv

  76. [84]

    Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. Memorizing transformers. In International Conference on Learning Representations

    2022

  77. [85]

    Jasper Xian, Tommaso Teofili, Ronak Pradeep, and Jimmy Lin. 2024. Vector search with openai embeddings: Lucene is all you need. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 1090--1093

    2024

  78. [86]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2024. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American ...

    2024

  79. [87]

    Yifei Yang, Zouying Cao, and Hai Zhao. 2024. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187

    2024 arXiv

  80. [88]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language ...

    2018

  81. [89]

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694

    2024 arXiv

  82. [90]

    Ruonan Yu, Songhua Liu, and Xinchao Wang. 2023. Dataset distillation: A comprehensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2023

  83. [91]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  84. [92]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. Glm-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations

    2023

  85. [93]

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. 2024 a . Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897

    2024 arXiv

  86. [94]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.814 B ench: Extending long context evaluation beyond 100 K tokens . In Proceedings of the 6...

    2024 doi

  87. [95]

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte...

    2021

  88. [96]

    Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Z itnik, and Pan Zhou. 2024 a . Moextend: Tuning new experts for modality and task extension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 80--91

    2024

  89. [97]

    Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. 2024 b . Let's think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In Proceedings of the IEEE/CVF Conference on Computer Vi...

    2024

  90. [98]

    Shanshan Zhong, Zhongzhan Huang, Weushao Wen, Jinghui Qin, and Liang Lin. 2023 a . Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models. In Proceedings of the 31st ACM International Conference on Multimedia, pages 567--578

    2023

  91. [99]

    Shanshan Zhong, Jinghui Qin, Zhongzhan Huang, and Daifeng Li. 2022. Cem: Machine-human chatting handoff via causal-enhance module. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3242--3253

    2022

  92. [100]

    Shanshan Zhong, Wushao Wen, Jinghui Qin, Qiangpu Chen, and Zhongzhan Huang. 2023 b . Lsas: Lightweight sub-attention strategy for alleviating attention bias problem. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 2051--2056. IEEE

    2023

  93. [101]

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. 2023. Recurrentgpt: Interactive generation of (arbitrarily) long text. arXiv preprint arXiv:2305.13304

    2023 arXiv

  94. [102]

    Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-short transformer: Efficient transformers for language and vision. Advances in neural information processing systems, 34:17723--17736

    2021

  95. [103]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  96. [104]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.