GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

Junyi Wu; Leyang Chen; Yulun Zhang; Zhiteng Li

arxiv: 2605.15852 · v2 · pith:EUVLVGO5new · submitted 2026-05-15 · 💻 cs.CV

GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

Leyang Chen , Junyi Wu , Zhiteng Li , Yulun Zhang This is my paper

Pith reviewed 2026-05-20 19:49 UTC · model grok-4.3

classification 💻 cs.CV

keywords 3D reconstructionstreaming videoKV cache evictiontoken managementgeometry-aware pruningefficient inferencemonocular videoonline cache management

0 comments

The pith

GHOST uses a model's own 3D geometry outputs to evict redundant KV-cache tokens online during streaming reconstruction from video.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces GHOST as a training-free way to manage the growing key-value cache in long monocular video sequences for 3D reconstruction. Instead of cutting the cache to fixed anchor frames or using attention scores that ignore scene structure, it scores tokens by how important they are to the current 3D geometry prediction. Three components work together: a dual-level importance score, protection for special tokens, and per-layer budget allocation guided by cosine similarity between layers. Experiments show this keeps reconstruction quality high on standard benchmarks while shrinking the cache by roughly half and speeding inference by 1.75 times over prior methods.

Core claim

GHOST is a geometry-hierarchical online streaming token eviction method that exploits the model's own 3D geometry outputs to decide which tokens to retain in the KV cache. It combines a hierarchical dual-level importance scoring scheme, a privilege mechanism that shields special tokens from eviction, and a cosine-similarity-guided layer-wise budget allocation strategy. This framework runs without additional training and directly addresses the linear growth of cache memory in long-sequence 3D reconstruction tasks.

What carries the argument

hierarchical dual-level importance scoring with privilege protection and cosine-similarity layer-wise budget allocation that uses the model's 3D geometry predictions as the eviction signal

If this is right

KV cache memory usage drops by nearly half while reconstruction quality stays comparable to full-cache baselines.
Inference runs 1.75 times faster than existing state-of-the-art streaming methods on the tested benchmarks.
The approach works without any extra training or fine-tuning steps.
The three components reinforce one another so that geometrically valuable tokens survive eviction even in extended sequences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same geometry-driven eviction idea could be tested on other long-context 3D tasks such as novel-view synthesis from video.
If the geometry outputs degrade on out-of-distribution scenes the eviction decisions would likely become unreliable, suggesting a need for fallback heuristics.
Layer-wise budget allocation might generalize to other transformer-based 3D models that output intermediate geometric features.
Real-time deployment on memory-constrained hardware becomes more feasible once the cache size is decoupled from sequence length.

Load-bearing premise

The model's 3D geometry outputs are sufficiently reliable and informative to serve as the basis for online token eviction decisions without causing quality degradation across diverse scenes and benchmarks.

What would settle it

Running GHOST on the same long video sequences and benchmarks as the full-cache baseline and measuring a clear drop in reconstruction metrics such as PSNR, SSIM, or surface accuracy would show the eviction strategy harms quality.

Figures

Figures reproduced from arXiv: 2605.15852 by Junyi Wu, Leyang Chen, Yulun Zhang, Zhiteng Li.

**Figure 1.** Figure 1: Radar comparison across 7-Scenes, NRGBD and Bonn (averaged over all input lengths; outer = better). GHOST consistently dominates all baselines on every axis. Transformer models [17] have achieved remarkable results in 3D reconstruction from monocular images [21, 11, 18], learning to predict dense depth, point maps, and camera poses in a single forward pass. VGGT [18] extends this to multi-view sequences b… view at source ↗

**Figure 2.** Figure 2: Correlation between Key-sim score and two frame attributes: Left: Negligible linear correlation between Key-sim score and camera pose change (ρ = −0.07); right: Moderate positive linear correlation between Key-sim score and depth gradient variance (ρ = +0.31). Dashed lines denote linear fitting trends. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Figure 3: Top row: Raw RGB inputs from Long3D Lecture Hall, 7-Scenes Heads, 7-Scenes Chess, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Figure 4: GHOST inference pipeline. Offline: Cosine-similarity profiling allocates per-layer budgets. Online: An eviction mode that prunes KV cache to layer-wise budget computed offline with Geometry-Hierarchical Importance scoring and Special token boost . GHOST assigns per-patch importance ϕ(t, p) = wf sframe(t) + wkstoken(t, p), where sframe combines camera motion scam, depth variance sgeo, and recency stemp, whi… view at source ↗

**Figure 5.** Figure 5: Layer-wise budget allocation guided by cosine similarity. Larger input–output colour discrepancy and larger arrows indicate lower ρ¯ℓ; the cylinder shows how Btotal is distributed, with such layers receiving larger Bℓ. Camera tokens ct and register tokens {r i t} encode global scene geometry state and structural priors. Evicting these tokens can corrupt pose estimation and scene globalisation, yet standar… view at source ↗

**Figure 6.** Figure 6: Qualitative reconstruction comparison on 7-Scenes (Chess, Fire, Heads, Office, Kitchen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: Per-layer cosine similarity ρ¯ℓ (blue) and GHOST budget Bℓ (orange, τ=0.5). Lower similarity layers receive larger budgets [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

**Figure 8.** Figure 8: Accuracy Mean (↓) versus sequence length on the Long3D benchmark. The shaded region highlights the gap between GHOST and InfiniteVGGT. GHOST’s advantage over key-similarity eviction (InfiniteVGGT) grows with sequence length, confirming that geometry-aware eviction scales more gracefully to very long sequences. Limitations. GHOST is not directly applicable to architectures that lack any causal structure (e.… view at source ↗

read the original abstract

Streaming 3D reconstruction from long monocular video sequences requires maintaining a key-value (KV) cache that grows linearly with sequence length, creating a severe memory bottleneck. Existing approaches either truncate the cache to a fixed set of anchor frames, leading to reconstruction quality degradation, or rely on attention-score heuristics that are agnostic to 3D scene structure, failing to preserve geometrically valuable tokens. To address these problems, we present GHOST (Geometry-Hierarchical Online Streaming Token Eviction), a training-free KV cache management framework that exploits the model's own 3D geometry outputs to evict redundant tokens online. GHOST introduces three mutually reinforcing innovations: a hierarchical dual-level importance scoring scheme, a privilege mechanism that protects special tokens from eviction, and a cosine-similarity-guided layer-wise budget allocation. Experiments on various benchmarks show that GHOST preserves excellent reconstruction quality while cutting the KV cache by nearly half and delivering 1.75x faster inference compared to state-of-the-art methods. Our code is available at https://github.com/lokiniuniu/GHOST.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

GHOST gives a practical training-free way to evict KV tokens using the model's own 3D geometry outputs, but the early-frame reliability issue is a real soft spot that needs checking.

read the letter

Hi, the main takeaway is that this paper tackles the linear growth of KV cache in streaming monocular 3D reconstruction by scoring tokens for eviction with geometry information the model already computes. It avoids both crude truncation and pure attention heuristics. The three pieces are a hierarchical dual-level scorer, a privilege guard for special tokens, and cosine-similarity allocation of per-layer budgets. All of it runs without retraining, which keeps it simple to apply. The reported gains are roughly halving the cache size and 1.75 times faster inference while holding reconstruction quality on the benchmarks they tried. Public code is a plus for anyone who wants to reproduce or extend it. The approach feels like a sensible engineering step for memory-constrained settings such as robotics or AR pipelines that ingest long videos. On the downside, the stress-test concern about early frames lands. With minimal parallax at the start, the geometry signal is likely weak or biased, and any tokens evicted then stay gone. The privilege set is small and fixed, so it may not catch systematic early mis-ranking across many tokens. The abstract claims strong results but gives no experimental details, baselines, or variance numbers, so it is hard to judge whether the quality preservation holds when geometry is uncertain. This work is aimed at practitioners who already run 3D reconstruction models and need to stretch them to longer sequences without blowing up memory. A reader focused on efficient inference or KV cache tricks would get concrete ideas from the scoring and allocation rules. It deserves a serious referee because the problem is real and the proposed combination is specific enough to evaluate. I would send it out for review rather than desk reject.

Referee Report

2 major / 0 minor

Summary. The paper presents GHOST, a training-free KV cache management framework for streaming 3D reconstruction from long monocular video sequences. It uses the model's 3D geometry outputs to perform online token eviction via a hierarchical dual-level importance scoring scheme, a privilege mechanism for special tokens, and cosine-similarity-guided layer-wise budget allocation. The authors claim that this approach maintains excellent reconstruction quality while reducing the KV cache by nearly half and achieving 1.75x faster inference compared to state-of-the-art methods.

Significance. If the results hold, GHOST represents a meaningful advance in efficient long-sequence 3D reconstruction by incorporating geometric structure into cache eviction decisions rather than relying on generic attention heuristics. The training-free design and availability of code are positive aspects that facilitate reproducibility and adoption.

major comments (2)

The abstract reports positive benchmark results but provides no details on experimental setup, baselines, error bars, or potential post-hoc choices in eviction rules. This makes it impossible to verify if the data supports the claim of preserved quality with halved cache.
The approach assumes that the model's 3D geometry outputs are reliable from the first frame for making eviction decisions. However, in streaming monocular video with minimal initial parallax, these outputs are likely low-confidence or biased, risking permanent eviction of geometrically critical tokens. The privilege mechanism protects only a small fixed set and does not address this systematic early mis-ranking issue.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments and the positive assessment of GHOST's significance and reproducibility. We respond point-by-point to the major comments below, indicating planned revisions where appropriate.

read point-by-point responses

Referee: The abstract reports positive benchmark results but provides no details on experimental setup, baselines, error bars, or potential post-hoc choices in eviction rules. This makes it impossible to verify if the data supports the claim of preserved quality with halved cache.

Authors: We agree that the abstract's brevity omits these specifics, which are instead provided in the main text. Section 4 details the experimental setup (datasets, hardware, streaming protocol), baselines (including prior KV-cache eviction and streaming 3D methods), and evaluation metrics. Section 5 reports results with error bars from multiple runs and confirms that eviction rules are fixed and deterministic with no post-hoc tuning. To improve immediate verifiability, we will revise the abstract to briefly note the benchmarks used and that quality preservation is shown with standard deviations. revision: yes
Referee: The approach assumes that the model's 3D geometry outputs are reliable from the first frame for making eviction decisions. However, in streaming monocular video with minimal initial parallax, these outputs are likely low-confidence or biased, risking permanent eviction of geometrically critical tokens. The privilege mechanism protects only a small fixed set and does not address this systematic early mis-ranking issue.

Authors: This is a legitimate concern for the bootstrap phase. GHOST's hierarchical dual-level scoring combines immediate geometry cues with longer-term consistency, while the privilege mechanism protects both a fixed set of special tokens and dynamically high-importance ones. Because eviction decisions are made online and continuously, early low-confidence rankings can be revisited as parallax accumulates. Our experiments (Section 5 and supplementary ablations) show robust final reconstruction quality under streaming conditions. To strengthen the manuscript, we will add a short discussion subsection analyzing early-frame behavior and any observed sensitivity to initial parallax. revision: partial

Circularity Check

0 steps flagged

No circularity: heuristic training-free method with no self-referential derivation

full rationale

The paper presents GHOST as a training-free KV cache management framework that exploits the model's own 3D geometry outputs for online token eviction via hierarchical scoring, privilege mechanism, and cosine-similarity allocation. No mathematical derivation chain, parameter fitting, or equations are described that reduce predictions or results to inputs by construction. The approach is explicitly heuristic and relies on external model outputs plus empirical benchmark validation rather than any closed self-referential loop, making the central claims self-contained against external testing.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the unverified reliability of geometry outputs for eviction and the effectiveness of the proposed heuristics across scenes; no free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)

domain assumption Model's 3D geometry outputs can be used directly to identify redundant tokens without introducing reconstruction errors.
Eviction decisions depend on these outputs being accurate guides for importance.

pith-pipeline@v0.9.0 · 5726 in / 1126 out tokens · 35641 ms · 2026-05-20T19:49:19.101842+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

GHOST scores each cached token by a hierarchical dual-level importance signal derived entirely from the model’s own outputs: a frame-level component integrating camera pose change, depth gradient variance, and temporal recency... a token-level component integrating visual saliency, depth confidence, and 3D point confidence
IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean alpha_pin_under_high_calibration unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

cosine-similarity-guided layer-wise budget allocation... πℓ ∝ exp(aℓ/τ) where aℓ = 1−ρ̄ℓ

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.