Turbocharging Web Automation: The Impact of Compressed History States

Haofu Liao; Peng Tang; Srikar Appalaraju; Xiyue Zhu

REVIEW 4 major objections 5 minor 19 references

Turbocharging Web Automation: The Impact of Compressed History States

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Compressing an agent's five most recent web pages into 256-token summaries improves next-action accuracy by 1.2-5.4 absolute points.

desk verdict A sensible Perceiver-style history compressor that plausibly helps web agents, but the headline gains are compromised by test-set hyperparameter selection and an undisclosed oracle-history assumption. read the letter →

arxiv 2507.21369 v1 pith:43HFXJXA submitted 2025-07-28 cs.CL

Xiyue Zhu , Peng Tang , Haofu Liao , Srikar Appalaraju This is my paper

classification cs.CL

keywords webautomationagentshistorycompressioncross-attentionlanguageinstructionsnext-actionpredictionMind2WebLINX

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Web automation agents usually decide the next click from the current page plus a record of past actions, ignoring what earlier pages looked like. This paper argues that past page states carry task-relevant information, and that the reason prior work left them out is their verbosity, not their irrelevance. Its proposed fix is a trained History Compressor that turns each past state into a fixed-length, 256-token representation guided by the instruction and history, so a model can use up to five past states without blowing up the input. On two public web-navigation benchmarks, the compressed-history model beats the no-history baseline by 1.2-5.4 absolute accuracy points across all splits and metrics. The result matters because it offers a cheap way to give web agents memory of where they have been, rather than only what they currently see.

What carries the argument

The central mechanism is the History Compressor, a learned module that maps each verbose history state to a fixed-length token sequence. It is a cross-attention compressor: a small set of learnable query vectors first self-attend, then cross-attend to the history state, the recorded actions, and the language instruction, pass through a feed-forward block, and then exchange information with neighboring history inputs through a fusion layer (channel-wise concatenation followed by a linear projection). The same weights are shared across all history inputs, and the output queries become the compact representation of that history. The paper uses 256 queries per history, up to five histories, and a two-stage training scheme in which zero-initialized attention gradually lets the transformer trust the compressed history representations before all modules are trained jointly. This mechanism is doing the work of distilling sparse, task-relevant information while keeping inference cost close to the no-history baseline.

What would settle it

Re-run the same trained model in a live browser loop where each predicted action actually changes the page and the observed successor state becomes the history input, then compare task success against feeding ground-truth states; if the gain over the no-history baseline vanishes or reverses in the self-generated condition, the central claim depends on oracle-history availability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that past web states improve next-action prediction once they are distilled into compact, task-relevant representations. The History Compressor does this by running a fixed set of learnable queries through self-attention, cross-attention over each history input (HTML state, action history, instruction), a feed-forward layer, and a fusion step that lets neighboring histories communicate; the resulting fixed-length representations are concatenated with the current state and fed to the transformer. With up to five histories compressed to 256 tokens each, the approach outperforms the no-history baseline by 1.2-5.4 absolute points on both benchmarks, and it also beats untrained alternatives: pruning the top HTML elements or asking a zero-shot LLM to summarize past pages. Ablations show that accuracy generally rises with more history inputs, that 256 tokens per history is the sweet spot, and that the fusion module is responsible for a consistent share of the gain. The paper reads this as evidence that history states are a usable signal for web agents once their verbosity is controlled.

Load-bearing premise

The load-bearing premise is that the history states fed to the model during training and testing come from the correct, ground-truth trajectory; in real use an agent's own earlier mistake would change those states, and the reported accuracy gains may not survive that change.

Editorial extensions

If this is right

Web agents should include past page states, not just past actions, when selecting the next element to act on.
The 1.2-5.4 absolute-point gains appear on zero-shot splits covering unseen tasks, websites, and domains, so history helps generalization, not just memorized trajectories.
Because each history costs a fixed 256 tokens, the approach can be added to existing agents with modest memory and latency overhead compared with pruning-based compression.
The paper's ablations show gains grow with number of histories up to five, suggesting the fixed budget is not yet saturated.
Untrained compressors such as pruning or LLM summarization can hurt accuracy, so the useful history signal is not just shorter page text but task-conditioned distillation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the histories in the reported experiments come from ground-truth trajectories rather than the agent's own rollouts, the real-world gain could be smaller; a rollout-based re-evaluation would settle it.
The same cross-attention compressor should transfer to screenshot-based states, since it only needs a sequence of tokens, but whether a 256-token budget is enough for visual information is untested.
The 79% higher latency over the no-history baseline may be the main deployment bottleneck, so smaller compressors or fewer queries could trade a little accuracy for speed in practice.
The optimal 256-token budget likely depends on page complexity, so adaptive per-page budgets could improve on the single fixed size.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the core idea is reasonable and the paper is honest about its own limitations, but I would not take the 1.2–5.4% gains at face value until the hyperparameters are selected off the test splits.

What’s actually new: a Perceiver-style compressor that turns each past web state into a fixed-length set of query vectors conditioned on the instruction and actions, with a fusion layer across history steps and a two-stage training recipe. The ingredients are established, but the combination applied to web-automation history states is a fair incremental contribution. The ablations over number of histories and compression length give useful guidance, and the comparison to pruning and zero-shot LLM compressors gives the right context.

What the paper does well: the main tables are internally consistent, the gains show up on both Mind2Web and WebLINX, and Table 3 makes a practical case that compression matters for inference cost. The limitations section is refreshingly concrete—it admits the 79% latency increase over no-history inputs, which many papers would bury.

The soft spots are real but not fatal. The stress-test note is correct: Tables 4 and 5 pick N=5 and 256 tokens on the same Mind2Web test splits that produce Table 1, so the reported margins are partly in-sample. That needs a held-out validation split. Second, there are no error bars or seeds, and small gaps like the 0.31-point difference between 256 and 512 tokens on Cross-Task could easily flip. Third, the paper never states whether history states come from ground-truth trajectories or model rollouts; if the latter, errors compound and the measured gains may not transfer. Fourth, there is no capacity-matched control—adding a trainable compressor adds parameters, so some of the gain could be capacity rather than history signal. The fusion ablation is a partial control, but it does not isolate the historical information itself.

None of these kill the paper. The mechanism is plausible, and the internal numbers are consistent. But the headline numbers are provisional. Who this is for: people working on web agents and long-horizon state compression. It deserves a serious referee; the flaws are correctable. I would ask for an independent validation split, error bars, and a clear statement about where history states come from before trusting the numbers.

Referee Report

4 major / 5 minor

Summary. The paper proposes Web History Compressor, a module that compresses each of the N most recent web states into a fixed number of learnable-query tokens before feeding them, together with the current state, history actions, and instruction, into a Flan-T5-based MindAct model. The compressor uses Perceiver-style self/cross-attention and a history fusion module, and is trained in two stages with zero-initialized attention. On Mind2Web and WebLINX, the authors report 1.2-5.4% absolute accuracy gains over the no-history MindAct baseline, and compare against pruning, LLM-summarization, and LLMLingua compression alternatives, with an inference-cost analysis in Table 3.

Significance. The question is practically important: web states are verbose, and a fixed-length compressed representation that lets a model exploit past states at modest cost would be a useful building block for web agents. The paper's comparisons against cheap off-the-shelf compressors and its inference-cost measurements are valuable, and the reported gains are internally consistent with the tables. However, the evidence is not yet conclusive because the main design choices (N=5, 256 tokens) are selected on the same Mind2Web test splits used for the headline results, there are no variance estimates, and there is no control isolating the contribution of history information from added model capacity and additional training. If these issues are addressed, the result would be a solid empirical contribution.

major comments (4)

[Appendix C.1/C.2, Tables 4-5] The number of history inputs N=5 and the compression length of 256 tokens are selected by running ablations directly on the Mind2Web test splits (Cross-Task, Cross-Website, Cross-Domain), which are the same evaluation sets reported in Table 1. This makes the headline configuration an in-sample selection on the test data. The margins involved are small enough that this matters: on Cross-Task the element-accuracy gain from 4 to 5 histories is 1.11 points (44.69 to 45.80), and 256 tokens beats 512 by only 0.31 points (45.80 vs 45.49). Please select these hyperparameters on a validation split derived from the training set, or on a separate development set, and then report Table 1 with the fixed configuration; if that is impossible, the claims should be softened and the selection bias acknowledged.
[Section 2.2, Tables 1-2] There is no control separating the effect of history information from the effect of added model capacity and an additional training stage. The full model has a randomly initialized compressor with extra parameters and is trained in two stages, whereas the MindAct baseline has neither. The observed gains could therefore stem from fine-tuning or capacity rather than from the content of the history states. Please add an ablation that keeps all components and training stages identical but removes or randomizes the history information, for example by replacing each history state with a fixed placeholder, or by shuffling history states across steps. This is needed to support the causal statement that history states 'turbocharge' web automation.
[Section 3.1] The paper never states whether the past states and actions fed to the model during evaluation come from the ground-truth demonstration trajectories or from the model's own, possibly erroneous, rollouts. Since a deployment agent must operate on self-generated histories, the benchmark protocol matters for external validity. Please state explicitly the source of the history inputs, and if only oracle histories are used, add a discussion or a small experiment under self-generated rollouts.
[Tables 1-7] No error bars, seeds, or significance tests are reported for any experimental condition. Many of the differences that support the claims are under one or two points (e.g., Cross-Domain element accuracy 32.65 vs 31.40; Cross-Website 256 vs 512 tokens 32.17 vs 31.56), so it is impossible to know whether these are stable effects or seed noise. Please report multiple seeds or bootstrap confidence intervals at least for the main tables.

minor comments (5)

[Tables 1, 2, 4] The header 'Marco Element Acc' should read 'Macro Element Acc'; Table 2 also has a stray 'PPruning' instead of 'Pruning'.
[Appendix C.2] The module is called a 'q-former-based compressor' in Appendix C.2, while Section 2.1 describes a Perceiver-style design; please align the terminology.
[Table 3] The row 'No compressor' and the column header 'Average Maximum # tokens / history' are unclear; please explain what these entries mean and what the units are.
[References] The reference list contains duplicate entries for Mind2Web (2023 and 2024 versions) and WebLINX (two 2024 entries); please consolidate them.
[Section 3.2, Table 6] The text states that the fusion module yields 'consistent improvements', but Table 6 shows that on Cross-Website the without-fusion variant is better on all four metrics (element accuracy 32.56 vs 32.17, macro element 36.15 vs 35.71, step accuracy 28.84 vs 28.73, macro step 31.95 vs 31.83). Please qualify this claim and discuss why fusion can hurt on Cross-Website.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the web-history-compressor results are measured against held-out Mind2Web and WebLINX test splits, and no claimed prediction is equivalent to a fitted input by construction.

full rationale

The paper's central claim is an empirical accuracy comparison. The history compressor is trained on the official training splits and evaluated on the official test splits; the transformer backbone (MindAct/Flan-T5) and the compressor architecture are taken from independent sources (Perceiver, LLMLingua, MindAct), not from a self-citation chain. The two-stage training procedure and zero-initialized attention are implemented as described, with the compressor weights randomly initialized and trained on the training data. The reported 1.2–5.4% absolute improvements are measurements on the test splits, not quantities derived from the model's own definitions. The self-citations (Park et al. 2025; Gao et al. 2024) appear only in contextual citation lists and are not load-bearing for the numerical claims. The Limitations section's acknowledgment of 79% higher inference latency is an honest cost disclosure, not a circularity. The only methodological concern visible in the manuscript is that Appendix C.1/C.2 select the number of history inputs (N=5) and the compressed length (256 tokens) using the same Mind2Web test splits reported in Table 1; that is a test-set selection / overfitting risk, not circularity, because the final numbers are still empirical measurements and no parameter is renamed as a prediction. Under the stated criteria, there is no step where an equation reduces to its own input or where a fitted parameter is presented as a derived prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The history compressor and learnable queries are model components, not independent postulates. The main free choices are hyperparameters selected on the test splits, and the main unstated premise is that histories come from the ground-truth trajectory.

free parameters (4)

Number of learnable queries / compressed length per history = 256
Chosen by running the model on Mind2Web test splits (Appendix C.2, Table 5) rather than a held-out validation set; this directly affects reported accuracy.
Number of history inputs N = 5
Chosen by test-split ablation (Appendix C.1, Table 4); performance increases with N up to 5, and 5 is used in the main results.
Number of history compressor layers M = 2
Hand-selected in Section 2.2, not ablated; capacity affects how much history information is retained.
Zero-init attention factor = learned
The scale factor controlling attention to history representations is learned during first-stage training (Section 2.2, citing Zhang et al. 2024).

assumptions (4)

domain assumption History states contain task-relevant information not available in the current state or in the action history.
Motivates the entire method (Section 1, Figure 1). If false, compression adds nothing and the reported gains would have to come from model capacity.
domain assumption A fixed-length 256-token representation can preserve enough of a history state for future action prediction.
Central modeling bet of Section 2.1; no information-theoretic or empirical analysis of what is lost in compression is provided.
domain assumption History states and actions used in evaluation are the ground-truth trajectory rather than the agent's own rollouts.
Section 3.1 says official splits are followed, but the paper never states whether past states are oracle annotations or self-generated. Real agents would feed their own (possibly incorrect) past states back.
domain assumption MindAct with Flan-T5-base is a fair and representative base model for testing the compressor.
All experiments use one base model (Section 2.2); the Limitations section acknowledges the method is not tested on stronger or multimodal models.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Turbocharging Web Automation: The Impact of Compressed History States." pith.science (2026). https://pith.science/paper/43HFXJXA

@misc{pith2026250721369,
  author       = {Pith},
  title        = {Pith review of: Turbocharging Web Automation: The Impact of Compressed History States},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/43HFXJXA}},
  note         = {Machine review of arXiv:2507.21369}
}

read the original abstract

Language models have led to a leap forward in web automation. The current web automation approaches take the current web state, history actions, and language instruction as inputs to predict the next action, overlooking the importance of history states. However, the highly verbose nature of web page states can result in long input sequences and sparse information, hampering the effective utilization of history states. In this paper, we propose a novel web history compressor approach to turbocharge web automation using history states. Our approach employs a history compressor module that distills the most task-relevant information from each history state into a fixed-length short representation, mitigating the challenges posed by the highly verbose history states. Experiments are conducted on the Mind2Web and WebLINX datasets to evaluate the effectiveness of our approach. Results show that our approach obtains 1.2-5.4% absolute accuracy improvements compared to the baseline approach without history inputs.

Figures

Figures reproduced from arXiv: 2507.21369 by the authors.

**Figure 2.** (a) Overall architecture of our model. Our model takes the current input and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages

[1]

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

arXiv 2024
[2]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

2024
[3]

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

2024
[4]

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. https://arxiv.org/abs/2306.06070 Mind2web: Towards a generalist agent for the web . Preprint, arXiv:2306.06070

arXiv 2023
[5]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024
[6]

Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854

arXiv 2023
[7]

Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Zhuowen Tu, Vijay Mahadevan, and Stefano Soatto. 2024. Enhancing vision-language pre-training with rich supervisions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13480--13491

work page 2024
[8]

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856

arXiv 2023

Show all 19 references

[9]

Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. 2021. https://arxiv.org/abs/2103.03206 Perceiver: General perception with iterative attention . Preprint, arXiv:2103.03206

2021 arXiv
[10]

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.825 LLML ingua: Compressing prompts for accelerated inference of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

2023 doi
[11]

Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lun Chao. 2024. Dual-view visual contextualization for web navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14445--14454

2024
[12]

Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802

2018 arXiv
[14]

Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. https://arxiv.org/abs/2402.05930 Weblinx: Real-world website navigation with multi-turn dialogue . Preprint, arXiv:2402.05930

2024
[15]

Manmatha, and Shabnam Ghadar

Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R. Manmatha, and Shabnam Ghadar. 2025. R-vlm: Region-aware vision language model for precise gui grounding. In Findings of ACL

2025
[16]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

2020
[17]

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744--20757

2022
[18]

Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2024. https://arxiv.org/abs/2303.16199 Llama-adapter: Efficient fine-tuning of language models with zero-init attention . Preprint, arXiv:2303.16199

2024 arXiv
[19]

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614

2024 arXiv
[20]

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

arXiv 2024

[2] [2]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

2024

[3] [3]

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

2024

[4] [4]

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. https://arxiv.org/abs/2306.06070 Mind2web: Towards a generalist agent for the web . Preprint, arXiv:2306.06070

arXiv 2023

[5] [5]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

[6] [6]

Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854

arXiv 2023

[7] [7]

Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Zhuowen Tu, Vijay Mahadevan, and Stefano Soatto. 2024. Enhancing vision-language pre-training with rich supervisions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13480--13491

work page 2024

[8] [8]

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856

arXiv 2023