Pith. sign in

REVIEW 4 major objections 6 minor 213 references

Video-language models can't reliably judge physical laws; a new evidence-grounded dataset lifts accuracy from 51% to 81%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:39 UTC pith:POUTR42Y

load-bearing objection Useful benchmark, but the SFT improvement claim needs external validation before the headline is taken at face value. the 4 major comments →

arxiv 2608.02150 v1 pith:POUTR42Y submitted 2026-08-03 cs.CV cs.AI

PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

Zhongjie Ba , Shengwang Xu , Peng Cheng , Jinyang Zou , Ting Yu , Zhibo Wang , Zhan Qin This is my paper
classification cs.CV cs.AI
keywords physical law understandingvideo question answeringvideo-language modelsphysical consistencybenchmark datasetcounter-physical anomaliesevidence-grounded reasoningembodied AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PhyCheck is a video question-answering dataset built to test whether video-language models can genuinely judge when a scene obeys or violates a physical law. The authors argue that current models mostly pattern-match surface appearance: they score reasonably on videos where physics looks normal but drop sharply on videos where it is violated. The dataset pairs each video with a coarse yes/no question about a stated physical law and several fine-grained questions about the visual evidence for compliance or violation, plus a small pilot subset with hidden causal context. Fine-tuning a strong open video-language model on the fine-grained questions before the coarse-grained ones lifts accuracy from 50.93% to 81.00%, evidence that explicit evidence-grounded supervision can improve physical understanding. This matters for embodied AI and world models, which need to know not just what happened but whether it was physically possible.

Core claim

PhyCheck comprises 6,399 synthetic videos and 69,825 human-verified question-answer pairs, organized by six categories of physical law: mechanics, conservation, material properties, fluids and interface phenomena, optics, and thermal effects. Each video gets a coarse binary question - does the described phenomenon conform to the named law? - and a set of fine-grained binary questions that ask whether specific visual evidence is present (for example, an object moving without an external force). The paper's central claim is that this is the first video QA dataset to explicitly probe both compliance with and violation of physical laws. On it, current video-language models average around 60% acc

What carries the argument

The central object is PhyCheck's hierarchical annotation scheme. A coarse-grained question asks whether the observed phenomenon conforms to or violates a specified physical law, and the answer is inherited deterministically from the source benchmark's labels. Fine-grained questions, generated by a large multimodal model and verified by humans, name explicit visual details - spontaneous motion, unnatural acceleration, unexpected deformation - that support the verdict. These details form a reasoning chain from evidence to conclusion. The paper also uses two answer-polarity templates (consistency and violation wording) to reduce bias. The training recipe that produces the large gain is fine-gra

Load-bearing premise

The load-bearing premise is that the coarse physical ground truth copied from the source video-generation benchmark is correct; the human verification in this paper checks whether the fine-grained questions match the videos, not whether the inherited conforms/violates labels reflect actual physics.

What would settle it

Independently re-annotate a random sample of PhyCheck videos with physics-trained evaluators using trajectory tracking or simulation against the stated law, and measure agreement with the dataset's labels. Also retrain the fine-tuned model with the yes/no answer mapping flipped; if accuracy stays high, the gain comes from question-wording patterns rather than physical judgment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If PhyCheck's labels are reliable, then existing video-language models are not doing physics: their near-chance scores on violation videos show a strong bias toward accepting what they see.
  • Training on coarse-grained physical-consistency questions alone raises accuracy to about 67%, so a simple yes/no physical-consistency signal is a useful supervision signal.
  • Adding fine-grained evidence questions before the coarse questions raises accuracy to 81%, suggesting learning order matters for physical reasoning.
  • Failure rates vary by law category, so future models should be evaluated per-law rather than with a single aggregate score.
  • The context-sensitive pilot indicates that even accurate models may depend on surface appearance; tests that reveal hidden causes will be needed to certify physical understanding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims: the strong order effect (fine-grained before coarse) suggests a curriculum principle for physical reasoning that could transfer to other reasoning domains, not just video.
  • Because all videos come from generative models, a natural next test is whether fine-tuning on PhyCheck improves performance on real-world videos; the paper does not provide that transfer evidence.
  • The 50-pair context subset is a pilot; one could scale it into a full benchmark where the model must actively request missing causal information, rather than receive it as text.
  • The binary-format questions make the benchmark cheap to run but also expose it to a shortcut: a model could exploit wording patterns in the templates; answer-order ablations would reveal this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces PhyCheck, a video QA dataset for physical-law understanding built from 6,399 synthetic videos taken from VideoPhy2. It contains 69,825 binary question-answer pairs, organized into coarse-grained questions about physical-law conformity/violation and fine-grained evidence questions, plus a 50-pair context-sensitive pilot subset. The authors evaluate several Video-LLMs and report that fine-tuning Qwen2.5-VL on fine-then-coarse supervision raises accuracy from 50.93% to 81.00% on PhyCheck's test split. They also report a per-category failure analysis and a context-assisted probe.

Significance. PhyCheck targets a genuinely important gap: most physical-reasoning benchmarks are coarse-grained, and an evidence-grounded, violation-focused video QA dataset would be useful both for benchmarking and for model alignment. The paper's strengths include a large human-annotation effort, a reported raw inter-annotator agreement of 99.74% with Cohen's kappa 0.9959, binary yes/no questions that reduce some probing artifacts, and a public code release. If the reported supervised fine-tuning gains transfer beyond the PhyCheck training distribution, the dataset would be a meaningful contribution to physical understanding in video-LLMs. However, as it stands, the central improvement claim is supported only by in-distribution results on a synthetic, template-heavy benchmark.

major comments (4)
  1. [§4.3, Table 3; Abstract; §1.4] The central empirical claim—that PhyCheck supervision "substantially improves the understanding of physical-consistency"—is supported only by evaluation on PhyCheck's own test split. The videos are all synthetic (from seven text-to-video generators, §3.2) and the questions are templated or LLM-generated. In-domain fine-tuning and testing on the same distribution can improve accuracy by learning dataset-specific answer priors, question wording, or generator artifacts rather than physical law understanding. No out-of-distribution or external transfer evaluation is provided on PhysBench, PhysGame, MVP, or any real-world video benchmark. Please add such evaluations, and report results across multiple seeds; without them, the claim of improved physical understanding is unestablished.
  2. [§3.2] The dataset's physical ground truth is inherited directly from VideoPhy2: the paper "leverage[s] the physical rule annotations provided by VideoPhy2" and retains only the followed and violated samples. No independent audit of these labels is reported. The subsequent human verification checks whether the queried visual detail is present in the video, but it does not independently certify whether that detail constitutes a true physical-law violation. Since the fine-grained questions are generated by LLMs conditioned on the VideoPhy2 labels, the entire benchmark and downstream fine-tuning validity rests on the correctness of those external labels. Please provide an expert audit of a random sample (with agreement statistics) or another independent verification of the physical-law labels.
  3. [§4.3, Table 3] The order effect used to support the "hierarchical supervision design" is presented without variance or multiple seeds. Coarse-grained VQA alone gives 66.83%, but coarse-then-fine gives only 51.89%, while fine-then-coarse jumps to 81.00%. This non-monotonic, dramatic reversal is not explained and could indicate instability, catastrophic forgetting, or a fortuitous single run. The conclusion that "learning fine-grained physical consistency before coarse-grained physical reasoning provides a more effective learning trajectory" needs multiple seeds, error bars, and an analysis of why the coarse-then-fine order is worse than coarse alone.
  4. [Abstract; §1; §3.1] There are internal inconsistencies in the dataset description. The Abstract says "approximately 50k human-verified VQA pairs," while §3.1 reports 69,825 QA pairs. Also, §1 states the dataset combines "synthetic anomaly videos—built upon VideoPhy2—with curated real-world clips," but §3.1 says PhyCheck "comprises 6,399 synthetic videos" and the construction pipeline uses only VideoPhy2. These contradictions affect how readers interpret the dataset's composition and must be resolved.
minor comments (6)
  1. [§4.3, Table 3] The table uses checkmarks and circled numbers without a clear legend. Define precisely what "fine-then-coarse" and "coarse-then-fine" mean, including whether both stages use the same training set and for how many epochs.
  2. [§4.1] Implementation details are sparse: only "12 epochs" is given. Please report learning rate, batch size, optimizer, sequence length, and any frame sampling strategy.
  3. [§4.5] The context-sensitive subset has only 50 pairs, and the reported numbers (e.g., 0.96, 0.18, 0.98) lack confidence intervals or significance tests. The interpretation of the fine-tuned model's behavior as "epistemic caution" is speculative and should be softened.
  4. [§2.2] The claim that PhyCheck is "the first video QA dataset that explicitly probes both compliance with and violation of physical laws" should be qualified in light of prior work such as MVP and PhysGame, which already contain violation-oriented or shortcut-aware physical QA.
  5. [§3.2] The prompts given to Gemini-3 and GPT-5 are summarized but not included. Please provide the full prompt templates in an appendix so that the LLM-assisted annotation pipeline is reproducible.
  6. [Various] Minor naming/notation issues: "Qwen2.5-Finetune" is used inconsistently; Table 2's class-wise metrics for Conform/Violate are not defined rigorously; Table 4 subscripts denote video counts but no error bars are shown.

Circularity Check

0 steps flagged

No circularity: PhyCheck's ground truth is externally inherited from VideoPhy2 and human-verified; SFT gains are empirical in-domain results, not identity reductions.

full rationale

This is a benchmark-construction paper rather than a derivation. The load-bearing ground truth is imported from VideoPhy2, an external benchmark with no author overlap: the paper "leverage[s] the physical rule annotations provided by VideoPhy2" and retains followed/violated samples. Coarse answers are then deterministically assigned from those external labels ("The answers are deterministically assigned according to the original VideoPhy2 annotations"), and fine-grained questions are LLM-generated conditioned on the same labels but independently human-verified with 99.74% raw agreement. There is no fitted parameter later renamed as a prediction: Table 3 is a standard held-out train/test fine-tuning comparison, not an identity. The paper contains no load-bearing self-citation chain, and none of its references overlaps with the present authors. The lack of an external out-of-distribution transfer benchmark for the SFT claim is a real generalization limitation, but under the stated rules that is a correctness/validity concern, not circularity. The construction reduces to an external annotation source plus fresh human review, so the correct circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

Everything rests on the external VideoPhy2 labels and on human verification of generated questions. The paper's own contribution is the restructuring; the physical ground truth is inherited and filtered.

free parameters (3)
  • fine-tune epochs = 12
    Qwen2.5-VL fully fine-tuned for 12 epochs (§4.1); chosen by hand with no reported sweep or multiple seeds.
  • fine-then-coarse training order = fine-grained VQA followed by coarse-grained VQA
    Best configuration in Table 3 selected among four orders; used as evidence of superiority without variance reporting.
  • coarse-label filter = undetermined excluded
    Only 'followed' and 'violated' VideoPhy2 samples retained; 'undetermined' discarded (§3.2), changing the class balance and difficulty.
axioms (4)
  • domain assumption VideoPhy2's physical-rule annotations (followed/violated) are accurate ground truth
    In §3.2 the paper reuses VideoPhy2 rule labels and discards 'undetermined' samples; errors in those labels propagate into PhyCheck.
  • domain assumption Binary yes/no answers suffice to represent physical-law compliance
    All questions are binary (§3.1), collapsing graded physical plausibility into two classes.
  • domain assumption Human majority-vote verification of video evidence is correct
    Fine-grained questions are verified by two annotators with third-annotator resolution (§3.2); no adjudication by physical measurement.
  • domain assumption LLM-generated negative prompts interrogate genuine physical inconsistencies
    Gemini-3 and GPT-5 generate fine-grained details from coarse labels (§3.2); the paper provides no independent check that the generated evidence is physically causal rather than visually salient.

pith-pipeline@v1.3.0-daily-deepseek · 11668 in / 12598 out tokens · 96018 ms · 2026-08-04T13:39:56.884137+00:00 · methodology

0 comments
read the original abstract

Embodied intelligence and world models require video understanding systems to go beyond recognizing objects and actions and develop an understanding of physical regularities. However, despite their strong performance on general video understanding tasks, current video-language models still struggle to reliably determine whether an observed event conforms to specific physical laws. Existing benchmarks primarily assess the physical quality of generated videos, providing limited support for systematically evaluating and improving the physical-law understanding of Video Large Language Models (VideoLLMs). To address this gap, we introduce PhyCheck, a video question answering dataset organized at two complementary levels of granularity. The coarse-grained subset asks models to determine whether the phenomenon shown in a video conforms to or violates physical laws, while the fine-grained subset further examines whether models can capture physical details responsible for the violation or compliance. We use these subsets as structured supervision to improve physical understanding. In addition, the dataset contains a diagnostic subset with external causal context that reveal hidden factors affecting physical plausibility, assessing whether models can recalibrate their judgments accordingly. Experiments with Fine-tune Qwen2.5-VL show that training with the proposed data substantially improves the understanding of physical-consistency, while evaluations in the diagnostic subset reveal that current models still have difficulty incorporating additional causal conditions into their decisions. These findings highlight the gap between recognizing surface-level inconsistencies and understanding underlying physical mechanisms, and provide a foundation for evaluating and improving physical understanding in Video-LLMs.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

213 extracted references · 85 linked inside Pith

  1. [1]

    FirstName LastName , title =

  2. [2]

    FirstName Alpher , title =

  3. [3]

    Journal of Foo , volume = 13, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe , title =. Journal of Foo , volume = 13, number = 1, pages =

  4. [4]

    Journal of Foo , volume = 14, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe and FirstName Gamow , title =. Journal of Foo , volume = 14, number = 1, pages =

  5. [5]

    FirstName Alpher and FirstName Gamow , title =

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Representing long-range context for graph neural networks with global attention , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    arXiv preprint arXiv:2310.11829 , year=

    Towards graph foundation models: A survey and beyond , author=. arXiv preprint arXiv:2310.11829 , year=

    Pith/arXiv arXiv

  8. [8]

    Nature , volume=

    Foundation models for generalist medical artificial intelligence , author=. Nature , volume=. 2023 , publisher=

    2023

  9. [9]

    arXiv preprint arXiv:2407.01290 , year=

    Hypformer: Exploring Efficient Hyperbolic Transformer Fully in Hyperbolic Space , author=. arXiv preprint arXiv:2407.01290 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2401.12874 , year=

    From understanding to utilization: A survey on explainability for large language models , author=. arXiv preprint arXiv:2401.12874 , year=

    Pith/arXiv arXiv

  11. [11]

    Locret: Enhancing Eviction in Long-Context

    Yuxiang Huang and Binhang Yuan and Xu Han and Chaojun Xiao and Zhiyuan Liu , journal=. Locret: Enhancing Eviction in Long-Context. 2025 , note=

    2025

  12. [12]

    Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Boris Ginsburg , booktitle=

  13. [13]

    arXiv preprint arXiv:2412.15204 , year=

    LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks , author=. arXiv preprint arXiv:2412.15204 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2311.06786 , year=

    Explainability of Vision Transformers: A Comprehensive Review and New Perspectives , author=. arXiv preprint arXiv:2311.06786 , year=

    Pith/arXiv arXiv

  15. [15]

    SIAM Review , volume=

    Variance and covariance of distributions on graphs , author=. SIAM Review , volume=. 2022 , publisher=

    2022

  16. [16]

    International conference on machine learning , pages=

    Concept bottleneck models , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  17. [17]

    arXiv preprint arXiv:2412.10319 , year=

    Scbench: A kv cache-centric analysis of long-context methods , author=. arXiv preprint arXiv:2412.10319 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2504.13173 , year=

    It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization , author=. arXiv preprint arXiv:2504.13173 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2309.17453 , year=

    Efficient streaming language models with attention sinks , author=. arXiv preprint arXiv:2309.17453 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2308.16137 , year=

    Lm-infinite: Zero-shot extreme length generalization for large language models , author=. arXiv preprint arXiv:2308.16137 , year=

    Pith/arXiv arXiv

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    H2o: Heavy-hitter oracle for efficient generative inference of large language models , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Snapkv: Llm knows what you are looking for before generation , author=. Advances in Neural Information Processing Systems , volume=

  23. [23]

    Forty-first International Conference on Machine Learning , year=

    Cam: Cache merging for memory-efficient llms inference , author=. Forty-first International Conference on Machine Learning , year=

  24. [24]

    arXiv preprint arXiv:2406.13035 , year=

    D2o: Dynamic discriminative operations for efficient generative inference of large language models , author=. arXiv preprint arXiv:2406.13035 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2407.08454 , year=

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks , author=. arXiv preprint arXiv:2407.08454 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2406.06484 , year=

    Parallelizing linear transformers with the delta rule over sequence length , author=. arXiv preprint arXiv:2406.06484 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2407.04620 , year=

    Learning to (learn at test time): Rnns with expressive hidden states , author=. arXiv preprint arXiv:2407.04620 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2501.00663 , year=

    Titans: Learning to memorize at test time , author=. arXiv preprint arXiv:2501.00663 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2504.05646 , year=

    Lattice: Learning to efficiently compress the memory , author=. arXiv preprint arXiv:2504.05646 , year=

    arXiv

  30. [30]

    arXiv preprint arXiv:2410.13276 , year=

    Seerattention: Learning intrinsic sparse attention in your llms , author=. arXiv preprint arXiv:2410.13276 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2503.06692 , year=

    Inftythink: Breaking the length limits of long-context reasoning in large language models , author=. arXiv preprint arXiv:2503.06692 , year=

    arXiv

  32. [32]

    arXiv preprint arXiv:2505.24133 , year=

    R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration , author=. arXiv preprint arXiv:2505.24133 , year=

    arXiv

  33. [33]

    arXiv preprint arXiv:2503.00979 , year=

    Dialogue without limits: Constant-sized kv caches for extended responses in llms , author=. arXiv preprint arXiv:2503.00979 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2403.09636 , year=

    Dynamic memory compression: Retrofitting llms for accelerated inference , author=. arXiv preprint arXiv:2403.09636 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2404.15574 , year=

    Retrieval head mechanistically explains long-context factuality , author=. arXiv preprint arXiv:2404.15574 , year=

    Pith/arXiv arXiv

  36. [36]

    Advances in Neural Information Processing Systems , volume=

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention , author=. Advances in Neural Information Processing Systems , volume=

  37. [37]

    arXiv preprint arXiv:1905.09418 , year=

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned , author=. arXiv preprint arXiv:1905.09418 , year=

    Pith/arXiv arXiv 1905

  38. [38]

    arXiv preprint arXiv:2307.08691 , year=

    Flashattention-2: Faster attention with better parallelism and work partitioning , author=. arXiv preprint arXiv:2307.08691 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2412.05496 , year=

    Flex attention: A programming model for generating optimized attention kernels , author=. arXiv preprint arXiv:2412.05496 , year=

    Pith/arXiv arXiv

  40. [40]

    Annals of neurosciences , volume=

    [image] Memory: A Contribution to Experimental Psychology , author=. Annals of neurosciences , volume=

  41. [41]

    Let's (not) just put things in Context: Test-time Training for Long-context

    Rachit Bansal and Aston Zhang and Rishabh Tiwari and Lovish Madaan and Sai Surya Duvvuri and Fnu Devvrit and David Brandfonbrener and David Alvarez-Melis and Prajjwal Bhargava and Mihir Kale and Samy Jelassi , booktitle=. Let's (not) just put things in Context: Test-time Training for Long-context

  42. [42]

    Acta neurobiologiae experimentalis , volume=

    Two components of long-term memory , author=. Acta neurobiologiae experimentalis , volume=

  43. [43]

    arXiv preprint arXiv:2506.08889 , year=

    SeerAttention-R: Sparse Attention Adaptation for Long Reasoning , author=. arXiv preprint arXiv:2506.08889 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  45. [45]

    Open r1: A Fully Open Reproduction of DeepSeek-R1 , date =

  46. [46]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  47. [47]

    arXiv preprint arXiv:2501.05414 , year=

    Longproc: Benchmarking long-context language models on long procedural generation , author=. arXiv preprint arXiv:2501.05414 , year=

    arXiv

  48. [48]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  50. [50]

    The annals of mathematical statistics , volume=

    On information and sufficiency , author=. The annals of mathematical statistics , volume=. 1951 , publisher=

    1951

  51. [51]

    arXiv preprint arXiv:2503.08879 , year=

    LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference , author=. arXiv preprint arXiv:2503.08879 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2410.02660 , year=

    How to train long-context language models (effectively) , author=. arXiv preprint arXiv:2410.02660 , year=

    arXiv

  53. [53]

    arXiv preprint arXiv:2412.19442 , year=

    A survey on large language model acceleration based on kv cache management , author=. arXiv preprint arXiv:2412.19442 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2406.10774 , year=

    Quest: Query-aware sparsity for efficient long-context llm inference , author=. arXiv preprint arXiv:2406.10774 , year=

    Pith/arXiv arXiv

  55. [55]

    Advances in Neural Information Processing Systems , volume=

    Kvquant: Towards 10 million context length llm inference with kv cache quantization , author=. Advances in Neural Information Processing Systems , volume=

  56. [56]

    arXiv preprint arXiv:2504.15364 , year=

    KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments , author=. arXiv preprint arXiv:2504.15364 , year=

    arXiv

  57. [57]

    arXiv preprint arXiv:2406.02069 , year=

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling , author=. arXiv preprint arXiv:2406.02069 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2501.07278 , year=

    Lifelong learning of large language model based agents: A roadmap , author=. arXiv preprint arXiv:2501.07278 , year=

    arXiv

  59. [59]

    arXiv preprint arXiv:2503.09567 , year=

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models , author=. arXiv preprint arXiv:2503.09567 , year=

    Pith/arXiv arXiv

  60. [60]

    Advances in neural information processing systems , volume=

    Long-short transformer: Efficient transformers for language and vision , author=. Advances in neural information processing systems , volume=

  61. [61]

    arXiv preprint arXiv:2410.12876 , year=

    In-context kv-cache eviction for llms via attention-gate , author=. arXiv preprint arXiv:2410.12876 , year=

    Pith/arXiv arXiv

  62. [62]

    Advances in Neural Information Processing Systems , volume=

    Learning to compress prompts with gist tokens , author=. Advances in Neural Information Processing Systems , volume=

  63. [63]

    arXiv preprint arXiv:2502.11089 , year=

    Native sparse attention: Hardware-aligned and natively trainable sparse attention , author=. arXiv preprint arXiv:2502.11089 , year=

    Pith/arXiv arXiv

  64. [65]

    2025 , urldate =

    Qwen/Qwen3-4B-Instruct-2507 , date =. 2025 , urldate =

    2025

  65. [66]

    2025 , howpublished =

    Lazarevich, Ivan and Bick, David and Gupta, Harsh and Mukherjee, Srinjoy and Neema, Nishit and Ramakrishnan, Gokul and Venkatesh, Ganesh , title =. 2025 , howpublished =

    2025

  66. [67]

    2021 , eprint=

    BookSum: A Collection of Datasets for Long-form Narrative Summarization , author=. 2021 , eprint=

    2021

  67. [68]

    2024 , month =

    Chaitanya Singhal , title =. 2024 , month =

    2024

  68. [69]

    arXiv preprint arXiv:2407.20485 , year=

    A2sf: Accumulative attention scoring with forgetting factor for token pruning in transformer decoder , author=. arXiv preprint arXiv:2407.20485 , year=

    Pith/arXiv arXiv

  69. [70]

    arXiv preprint arXiv:2408.03675 , year=

    Nacl: A general and effective kv cache eviction framework for llms at inference time , author=. arXiv preprint arXiv:2408.03675 , year=

    Pith/arXiv arXiv

  70. [71]

    arXiv preprint arXiv:2509.09754 , year=

    LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation , author=. arXiv preprint arXiv:2509.09754 , year=

    Pith/arXiv arXiv

  71. [72]

    Advances in Neural Information Processing Systems , volume=

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time , author=. Advances in Neural Information Processing Systems , volume=

  72. [73]

    arXiv preprint arXiv:2402.02750 , year=

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache , author=. arXiv preprint arXiv:2402.02750 , year=

    Pith/arXiv arXiv

  73. [74]

    arXiv preprint arXiv:2402.12065 , year=

    Wkvquant: Quantizing weight and key/value cache for large language models gains more , author=. arXiv preprint arXiv:2402.12065 , year=

    Pith/arXiv arXiv

  74. [75]

    arXiv preprint arXiv:2410.21465 , year=

    Shadowkv: Kv cache in shadows for high-throughput long-context llm inference , author=. arXiv preprint arXiv:2410.21465 , year=

    Pith/arXiv arXiv

  75. [76]

    arXiv preprint arXiv:2409.10516 , year=

    Retrievalattention: Accelerating long-context llm inference via vector retrieval , author=. arXiv preprint arXiv:2409.10516 , year=

    Pith/arXiv arXiv

  76. [77]

    arXiv preprint arXiv:2405.17424 , year=

    LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence , author=. arXiv preprint arXiv:2405.17424 , year=

    Pith/arXiv arXiv

  77. [78]

    arXiv preprint arXiv:2503.12491 , year=

    Cake: Cascading and adaptive kv cache eviction with layer preferences , author=. arXiv preprint arXiv:2503.12491 , year=

    arXiv

  78. [79]

    arXiv preprint arXiv:2502.01941 , year=

    Can LLMs Maintain Fundamental Abilities under KV Cache Compression? , author=. arXiv preprint arXiv:2502.01941 , year=

    Pith/arXiv arXiv

  79. [80]

    International conference on machine learning , pages=

    Transformers are rnns: Fast autoregressive transformers with linear attention , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  80. [81]

    arXiv preprint arXiv:2006.04768 , year=

    Linformer: Self-attention with linear complexity , author=. arXiv preprint arXiv:2006.04768 , year=

    Pith/arXiv arXiv 2006

Showing first 80 references.