LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

Jiayi Yao; Junchen Jiang; Kuntai Du; Zhuohan Gu

REVIEW

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.13009 v2 pith:FGZWM7QN submitted 2024-11-20 cs.LG cs.CL

LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

Zhuohan Gu , Jiayi Yao , Kuntai Du , Junchen Jiang This is my paper

classification cs.LG cs.CL

keywords attentionllmsllmsteersteeringperformancebalancebaselinescompared

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.

LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

discussion (0)