Neural Prioritisation for Web Crawling

Francesca Pezzuti; Nicola Tonellotto; Sean MacAvaney

REVIEW 4 major objections 4 minor 40 references

Neural Prioritisation for Web Crawling

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that prioritising the crawling frontier with a neural quality estimator—propagating semantic quality through the link graph—substantially improves early crawling and ranking for natural language queries, with gains up to…

desk verdict Solid incremental extension of the authors' own oracle work; the propagation policies are reasonable, the RQ gains are real but likely overstated given the sparse baseline and undisclosed train/eval overlap. read the letter →

arxiv 2506.16146 v2 pith:LA2OYGAV submitted 2025-06-19 cs.IR

Francesca Pezzuti , Sean MacAvaney , Nicola Tonellotto This is my paper

classification cs.IR

keywords webcrawlingprioritisationneuralqualityestimationsemanticnaturallanguagesearchfrontierharvestrateClueWeb22

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a web crawler can be made substantially better at finding pages that answer natural language questions by steering its frontier with a neural semantic-quality score, rather than relying on link structure or popularity. The proposed policies QFirst and QMin propagate quality through the link graph: a newly discovered page inherits the quality estimate of the page linking to it, and QMin additionally lowers a page's priority whenever a lower-quality link to it appears. In simulated crawls over the English subset of ClueWeb22-B, these policies beat breadth-first search on natural language queries by large margins—up to +149% harvest rate (the fraction of crawled pages relevant to some query), +152% maxNDCG (the ranking quality an ideal ranker could achieve), and +139% nDCG@10 (the ranking quality of a BM25-plus-neural-rerank pipeline)—while staying essentially on par for keyword queries and collecting the same relevant pages up to 1.6× faster. If this holds beyond the simulated setting, crawlers can align with the ongoing shift from keyword search to conversational search without sacrificing classical keyword effectiveness. The authors frame the work as opening a line of research rather than a complete solution.

What carries the argument

The machinery is quality propagation through the inlinking neighbourhood. A neural quality estimator $M_\theta$—a lightweight transformer trained, query-independently, to score how likely a document is to be relevant to an arbitrary query—scores every downloaded page. QFirst then enqueues each newly discovered URL with priority equal to the parent page's score, never updating it; QMin enqueues with the parent's score and, whenever another link to the same URL is found, sets the priority to the minimum of the current priority and the new ancestor's score. The oracle QOracle, which uses the page's own text score as priority before download, provides an upper bound. This design converts semantic quality from a global, corpus-wide property into a local, propagable signal with $O(n)$ time and $O(n)$ space cost, no full-graph storage, and no tuning hyperparameters—which is how the gains are achieved at early crawl stages.

What would settle it

Measure the rank correlation between a page's QT5-Small quality score and the mean score of its inlinking parents across a large sample of a held-out web corpus. If the correlation is near zero or negative, the quality-propagation policies lose their grounding; alternatively, run the same ClueWeb22-style crawl comparison on a web graph engineered so that low-quality pages link heavily to high-quality pages and check whether the neural policies still beat BFS.

Watch

Extended reading notes

Core claim

The central claim is empirical: neural crawling policies—where a query-independent language model scores each downloaded page and that score becomes the priority for the URLs it points to—can dramatically improve early-crawl effectiveness for natural language queries. Because a page's text is not available before download, the practical policies substitute the quality of the linking page; QFirst uses that score once, and QMin maintains the minimum score among all inlinking ancestors. On the English subset of ClueWeb22-B with the Researchy Questions query set, QMin comes close to or matches the oracle policy that sees each page's true quality, and both practical policies outperform BFS throughout the crawl. The same framework remains competitive on keyword queries from MS MARCO Web Search, with the oracle's keyword advantage fading late in the crawl because the greedily built corpus accumulates distracting pages. The authors conclude that semantic quality is a usable, scalable signal for frontier prioritisation, and that crawlers should be redesigned around it.

Load-bearing premise

The load-bearing premise is that pages of similar semantic quality tend to link to each other, so the quality of a newly discovered page can be approximated by the quality of the pages that link to it; if that correlation is weak in a given web graph, QFirst and QMin simply propagate noise and the reported gains would not transfer.

Editorial extensions

If this is right

Crawlers for natural language search no longer need a complete or growing web graph: a per-page forward pass of a small neural estimator replaces global PageRank-style computation, making the approach easier to distribute across machines.
The reported speedups mean that to harvest the same number of relevant natural-language pages, a BFS crawler must download roughly 60% more pages than QMin, implying direct savings in bandwidth, storage, and crawl time.
The fact that QMin tracks the oracle on natural language queries indicates that most valuable pages for this query type are reachable through high-quality links, so a min-over-ancestors update recovers most of the theoretical benefit.
Keyword queries are a genuine trade-off: while harvest rate and maxNDCG improve early, nDCG@10 stays only comparable to BFS, and the paper attributes this to valuable pages that are reachable only through low-quality links.
Because the estimator is query-independent and cheap, the same crawler can be re-fine-tuned as search behaviour drifts, allowing the frontier policy to track emerging definitions of semantic quality without structural changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment, implied but not run here, is to measure the actual correlation between page quality scores and the mean score of inlinking parents on a fresh crawl; the entire QFirst/QMin family stands or falls on that correlation.
The paper itself lists open issues—real multi-threaded crawling, fairness and transparency of the estimator, and adversarial manipulation—so the headline gains should be read as results of a controlled simulation, not a deployment guarantee.
A boundary case is the 'deep but valuable' page reachable only through low-quality link farms: QMin would postpone it indefinitely, while QFirst's noisier exploration has a chance of finding it, hinting that a small amount of random exploration mixed into QMin could recover both benefits.
If natural language queries keep growing, crawl logs generated by these policies could become supervision for the next quality estimator, creating a feedback loop in which the crawler's priorities and the ranker's notion of relevance co-evolve.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid incremental extension of the authors' own earlier oracle-based work. The two practical policies (QFirst, QMin) propagate neural quality scores through the link graph, and the experiments on ClueWeb22-B show large early-crawl gains on Researchy Questions. The main thing to know before citing: the baseline is only BFS, and the training/evaluation split for MSM-WS is undisclosed, which leaves real room for inflated numbers.

What's actually new: the propagation idea. Prior work by the same group [31] only speculated that neighbor quality correlates; here they implement it in two flavors and test it. The RQ evaluation is a genuinely new result, and the reported speedups (up to 1.6×) are plausible. The paper also does a good job of positioning itself relative to Craw4LLM and other LLM-crawling work, and the complexity analysis is straightforward and honest.

Where the soft spots are: first, the baseline is just BFS. The authors justify excluding PageRank due to small graph size, and that's defensible, but the abstract says 'compared to existing crawling techniques' when they only compare to one. Second, there are no confidence intervals on the crawl-level curves, only pointwise significance tests at each time point. Third, and most important: the QT5 estimator is trained on MSM-WS positive labels, and the MSM-WS evaluation uses a filtered subset of MSM-WS queries without stating whether they are held out. That's a potential circularity for the keyword results. The stress-test note suggests the RQ results could also be inflated if the RQ relevant documents overlap with the MSM-WS training positives. That concern is speculative — both come from CW22 click logs, but RQ is a different query set, and the propagation mechanism still has to work through the link graph regardless of what the estimator recognizes. The overlap could inflate absolute numbers, but it wouldn't explain the relative advantage over BFS unless the estimator's training created a specific advantage for those pages. The paper doesn't report the overlap, so the magnitude is unknown. I'd like to see that checked, but I don't think it's load-bearing.

Who this is for: people working on web crawling, search engine efficiency, and the intersection of LLMs and IR. It's a short conference paper, not a definitive study. If a serious editor sent it to review, the reviewers could ask for the train/eval split, confidence intervals, and at least one more baseline (e.g., a simple content-based heuristic). But the core idea is worth engaging with, and the paper is well written and reproducible (code is on GitHub).

Recommendation: let it go to review, but don't accept the numbers as final.

Referee Report

4 major / 4 minor

Summary. The paper proposes two neural crawling policies, QFirst and QMin, that use an LLM-based semantic quality estimator (QT5-Small) to prioritise URLs in the frontier, together with an upper-bound oracle policy QOracle, and compares them against BFS on the English subset of ClueWeb22-B. The evaluation uses MS MARCO Web Search (keyword queries) and Researchy Questions (natural-language queries), reporting harvest rate, maxNDCG, nDCG@10 of a BM25-to-MonoELECTRA pipeline, and mean speedup. The central claim is that the neural policies significantly improve early crawling and retrieval effectiveness, especially for natural-language queries, while remaining competitive on keyword queries.

Significance. If the results hold, the paper offers a practical and scalable way to inject semantic quality signals into general-purpose crawling, with potentially large gains for natural-language search. The strengths are the reproducible setting (public code, public corpus), the use of multiple IR evaluation metrics, and the explicit comparison with an oracle upper bound. The main risk is that the evaluation may be partly entangled with the training of the quality estimator, and the crawl-level results rest on a single simulation run, so the headline improvements could be inflated by memorisation rather than semantic generalisation.

major comments (4)

[§3.3 vs §3.2 and §4] The quality estimator QT5-Small was trained on positive documents sampled from the MSM-WS training set (9.1M documents with positive relevance labels from CW22), while the RQ relevance judgments in Table 1 are defined by taking the most-clicked page per query from a click distribution over the same CW22 corpus. The paper does not report the intersection between the 1,973 RQ relevant documents and the QT5 training positives, nor does it exclude RQ-relevant (or MSM-WS evaluation) documents from quality-estimator training. If the overlap is large, the large RQ gains in Figures 2–4 could partly reflect the estimator having already memorised the very pages used as relevance targets. Please measure the overlap and re-run the experiments with any overlapping relevant documents removed from QT5 training, or provide another explicit demonstration that the evaluation queries and labels are held out.
[§3.1, §3.6, Figures 2–4] The experimental section reports a single crawling simulation from '100k randomly selected seed URLs' and does not report multiple seed sets, confidence intervals, or any description of the unit of analysis for the significance tests. The paired t-test and Z-test in §3.6 are not specified as paired over queries, over time points, or over repeated crawls. With one crawl trajectory per policy, the filled-circle markers in Figures 2–4 do not support a claim that one policy is 'significantly' better at the crawl level. Please state the pairing explicitly and provide confidence bands or multiple-seed results; even a block bootstrap over time points would strengthen the central empirical claim.
[Abstract and §3.6] The abstract claims improvement 'compared to existing crawling techniques', but the only crawling baseline is BFS. The justifications for excluding PageRank on small graphs are reasonable, yet they leave the practical comparison against any link- or content-aware heuristic untested. Since BFS is a very weak baseline, the headline percentages (+149% HR, +152% maxNDCG, +139% nDCG@10) may overstate the practical advantage over state-of-the-art prioritisation. Please either narrow the claims to 'compared to BFS' throughout the abstract and conclusions, or add at least one additional non-trivial baseline (for example, indegree-based best-first or a text-similarity focused crawler).
[§2.1 and §4.3] The practical policies rely on the assumption that a page's semantic quality is related to the quality of the pages linking to it, and QMin further assumes that low-quality ancestors imply a low-quality descendant. This assumption is asserted with a citation but not directly validated on the crawled corpus. The MSM-WS results in §4.3 show that QMin can be on par with BFS, suggesting the assumption does not always hold. Please add a direct analysis of the correlation between QT5 scores of linked pages in CW22-B, or an ablation replacing the semantic proxy with a random or non-semantic proxy, to demonstrate that the observed gains come from semantic quality propagation rather than from the link-structure bias alone.

minor comments (4)

[§2.2] The phrase 'one-the-fly quality estimation' should be 'on-the-fly quality estimation'.
[§5] In the Conclusions, 'their robustness to such attacks should has yet to be explored' should read 'has yet to be explored'.
[§3.6 and References] Reference [16] is cited as 'Helge et al.' in the text but listed as 'Holzmann Helge, Anand Avishek, and Khosla Megha'; the in-text citation should be consistent with the reference list, e.g., 'Holzmann et al.'.
[§3.3] The description of QT5 training says the model 'converged after 1.6M training instances', but the paper does not report the validation set used for early stopping or the final model's quality-estimation accuracy on held-out data; reporting these would help readers judge the strength of the estimator.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the core RQ result relies on a label source independent of QT5 training, though the MSM-WS arm and a self-cited quality-propagation premise warrant minor caveats.

full rationale

The central claim is an empirical result: neural crawling policies improve HR, maxNDCG, and nDCG@10 early in a crawl. The headline numbers for Researchy Questions (RQ) are not circular: QT5-Small is trained on MSM-WS positive labels, while the RQ relevance labels are the most-clicked page per RQ query, a different label source, so the RQ improvements are a genuine forward test of the policy. The quality-propagation premise for QFirst/QMin is supported in part by the authors' prior work [31], but the policies are evaluated against RQ data not used to fit QT5, so that self-citation is motivational rather than load-bearing. The MSM-WS arm is less clean: Section 3.3 samples QT5 training positives from 'the MSM-WS training set,' and Section 4.1 then reports HR/maxNDCG on MSM-WS, without the paper stating that the evaluation queries are held out from those training labels; if the same query set is used, that arm would measure re-ranking of training labels. The paper also does not quantify overlap between RQ relevant documents and MSM-WS training positives. These are unquantified risks to be disclosed, not by-construction reductions demonstrated in the text. QOracle is explicitly an upper bound within the chosen estimator family, not an absolute ceiling. Accordingly, no circular step meets the evidence bar; score 2 reflects the minor self-citation and the need for split/overlap transparency.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No theoretical derivation is present, so there are no fitted constants to audit. The empirical claims rest on the trained QT5-Small model (weights fitted to MSM-WS training labels) and on the domain assumptions listed. The paper introduces no new entities. The main unstated dependency is the evaluation split for MSM-WS, which is not disclosed.

assumptions (5)

domain assumption A page's semantic quality is positively correlated with the quality of the pages that link to it.
Section 2.1: 'the quality of a page is related to the quality of the pages it is connected to.' This underpins QFirst, which assigns a new page the quality score of its parent, and QMin, which takes the minimum over ancestors.
domain assumption A page linked only from low-quality pages is unlikely to be high-quality.
Section 2.1: 'if a page is linked to a low-quality page, it is highly unlikely to be of high-quality.' This is the basis for QMin's priority reduction upon rediscovery; not validated in the paper.
domain assumption QT5-Small, trained on MSM-WS relevance labels, provides a query-independent quality signal that generalizes to unseen queries.
Section 3.3: the estimator is trained on the MSM-WS training set and then used to prioritize for both MSM-WS and RQ evaluations. The RQ results depend on this generalization.
domain assumption BFS is a strong general-purpose baseline on small graphs, and PageRank is unreliable on partial graphs.
Section 3.6 cites [3,16,23] to justify excluding PageRank and using BFS. The comparative claims are therefore only against BFS.
domain assumption A static simulation on ClueWeb22-B English head pages approximates real crawling conditions.
Section 3.1 uses a single-threaded simulation over a static corpus; limitations are acknowledged in Section 5, but the effectiveness numbers rely on this approximation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Neural Prioritisation for Web Crawling." pith.science (2026). https://pith.science/paper/LA2OYGAV

@misc{pith2026250616146,
  author       = {Pith},
  title        = {Pith review of: Neural Prioritisation for Web Crawling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LA2OYGAV}},
  note         = {Machine review of arXiv:2506.16146}
}

read the original abstract

Given the vast scale of the Web, crawling prioritisation techniques based on link graph traversal, popularity, link analysis, and textual content are frequently applied to surface documents that are most likely to be valuable. While existing techniques are effective for keyword-based search, both retrieval methods and user search behaviours are shifting from keyword-based matching to natural language semantic matching. The remarkable success of applying semantic matching and quality signals during ranking leads us to hypothesize that crawling could be improved by prioritizing Web pages with high semantic quality. To investigate this, we propose a semantic quality-driven prioritisation technique to enhance the effectiveness of crawling and align the crawler behaviour with recent shift towards natural language search. We embed semantic understanding directly into the crawling process -- leveraging recent neural semantic quality estimators to prioritise the crawling frontier -- with the goal of surfacing content that is semantically rich and valuable for modern search needs. Our experiments on the English subset of ClueWeb22-B and the Researchy Questions query set show that, compared to existing crawling techniques, neural crawling policies significantly improve harvest rate, maxNDCG, and search effectiveness during the early stages of crawling. Meanwhile, crawlers based on our proposed neural policies maintain comparable search performance on keyword queries from the MS MARCO Web Search query set. While this work does not propose a definitive and complete solution, it presents a forward-looking perspective on Web crawling and opens the door to a new line of research on leveraging semantic analysis to effectively align crawlers with the ongoing shift toward natural language search.

Figures

Figures reproduced from arXiv: 2506.16146 by the authors.

**Figure 2.** Assessment of neural crawling policies and BFS in terms of Harvest Rate. MSM-WS (left) and RQ (right). Statistically [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Assessment of neural crawling policies and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Assessment of neural crawling policies and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 38 canonical work pages

[31]

Francesca Pezzuti, Ariane Mueller, Sean MacAvaney, and Nicola Tonellotto. 2025. Document Quality Scoring for Web Crawling. (2025). arXiv:2504.11011

work page Pith review arXiv 2025
[1]

Web Structure, Dynamics and Page Quality

Ricardo Baeza-Yates, Felipe Saint-Jean, and Carlos Castillo. 2002. "Web Structure, Dynamics and Page Quality". In Proc. SPIRE. 117–132

work page 2002
[2]

Monica Bianchini, Marco Gori, and Franco Scarselli. 2005. Inside PageRank.ACM TOIT 5 (2005), 92–128

work page 2005
[3]

Paolo Boldi, Massimo Santini, and Sebastiano Vigna. 2004. Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations. In Proc. W A W. 168–180

work page 2004
[4]

Soumen Chakrabarti, Martin van den Berg, and Byron Dom. 1999. Focused crawling: a new approach to topic-specific Web resource discovery. Computer Networks 31, 11-16 (1999), 1623–1640

work page 1999
[5]

Xuejun Chang, Debabrata Mishra, Craig Macdonald, and Sean MacAvaney. 2024. Neural Passage Quality Estimation for Static Pruning. In Proc. SIGIR. 174–185

work page 2024
[6]

Jia Chen, Jiaxin Mao, Yiqun Liu, Fan Zhang, Min Zhang, and Shaoping Ma. 2021. Towards a Better Understanding of Query Reformulation Behavior in Web Search. In Proc. WWW. 743–755

work page 2021
[7]

Qi Chen, Xiubo Geng, Corby Rosset, Carolyn Buractaon, Jingwen Lu, Tao Shen, Kun Zhou, Chenyan Xiong, Yeyun Gong, Paul Bennett, Nick Craswell, Xing Xie, Fan Yang, Bryan Tower, Nikhil Rao, Anlei Dong, Wenqi Jiang, Zheng Liu, Mingqin Li, Chuanjie Liu, Zengzhong Li, Rangan Majumder, Jennifer Neville, Andy Oakley, Knut Magne Risvik, Harsha Vardhan Simhadri, Ma...

work page 2025

Show all 40 references

[8]

Junghoo Cho, Hector Garcia-Molina, and Lawrence Page. 1998. Efficient crawling through URL ordering. Computer Networks 30, 1–7 (1998), 161–172

1998
[9]

Le, and Christopher D

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In Proc. ICLR

2020
[10]

De Bra and Reinier D.J

Paul M.E. De Bra and Reinier D.J. Post. 1994. Information retrieval in the World- Wide Web: Making client-based searching feasible. Computer Networks 27, 2 (1994), 183–192

1994
[11]

Marc Ehrig and Alexander Maedche. 2003. Ontology-focused crawling of Web documents. In Proc. SAC. 1174–1178

2003
[12]

Elsas and Susan T

Jonathan L. Elsas and Susan T. Dumais. 2010. Leveraging temporal dynamics of document content in relevance ranking. In Proc. WSDM. 1–10

2010
[13]

Dennis Fetterly, Nick Craswell, and Vishwa Vinay. 2009. Measuring the Search Effectiveness of a Breadth-First Crawl. In Proc. ECIR. 388–399

2009
[14]

Dennis Fetterly, Nick Craswell, and Vishwa Vinay. 2009. The impact of crawl policy on web search effectiveness. In Proc. SIGIR. 580–587

2009
[15]

Ido Guy. 2016. Searching by Talking: Analysis of Voice Queries on Mobile Web Search. In Proc. SIGIR. 35–44

2016
[16]

Holzmann Helge, Anand Avishek, and Khosla Megha. 2019. Estimating PageRank deviations in crawled graphs. Applied Network Science 4, 86 (2019)

2019
[17]

Maarek, Dan Pelleg, Menachem Shtal- haim, and Sigalit Ur

Michael Hersovici, Michal Jacovi, Yoelle S. Maarek, Dan Pelleg, Menachem Shtal- haim, and Sigalit Ur. 1998. The shark-search algorithm. An application: tailored Web site mapping. Computer Networks 30, 1–7 (1998), 317–326

1998
[18]

Andrey Kolobov, Yuval Peres, Eyal Lubetzky, and Eric Horvitz. 2019. Optimal Freshness Crawl Under Politeness Constraints. In Proc. SIGIR. 495–504

2019
[19]

Dirk Lewandowski and Nadine Höchstötter. 2008. Web Searching: A Quality Measurement Perspective. Springer, 309–340

2008
[20]

Andrew McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. 2000. Au- tomating the Construction of Internet Portals with Machine Learning.Information Retrieval 3, 2 (2000), 127–163

2000
[21]

Filippo Menczer, Gautam Pant, and Padmini Srinivasan. 2004. Topical web crawlers: Evaluating adaptive algorithms. ACM TOIT 4, 4 (2004), 378–419

2004
[22]

Filippo Menczer, Gautam Pant, Padmini Srinivasan, and Miguel E. Ruiz. 2001. Evaluating topic-driven web crawlers. In Proc. SIGIR. 241–249

2001
[23]

Marc Najork and Janet L. Wiener. 2001. Breadth-first crawling yields high-quality pages. In Proc. WWW. 114–118

2001
[24]

Christopher Olston and Sandeep Pandey. 2008. Recrawl scheduling based on information longevity. In Proc. WWW. 437–446

2008
[25]

Liudmila Ostroumova, Ivan Bogatyy, Arseniy Chelnokov, Alexey Tikhonov, and Gleb Gusev. 2014. Crawling Policies Based on Web Page Popularity Prediction. In Proc. ECIR. 100–111

2014
[26]

Arnold Overwijk, Chenyan Xiong, and Jamie Callan. 2022. ClueWeb22: 10 Billion Web Documents with Rich Information. In Proc. SIGIR. 3360–3362

2022
[27]

Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank Citation Ranking: Bringing Order to the Web. In Proc. WWW

1999
[28]

Sandeep Pandey, Sourashis Roy, Christopher Olston, Junghoo Cho, and Soumen Chakrabarti. 2005. Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results. In Proc. VLDB. 781–792

2005
[29]

Gautam Pant, Shannon Bradshaw, and Filippo Menczer. 2003. Search Engine- Crawler Symbiosis: Adapting to Community Interests. In Proc. TPDL. 221–232

2003
[30]

Gautam Pant and Padmini Srinivasan. 2006. Link contexts in classifier-guided topical crawlers. IEEE TKDE 18, 1 (2006), 107–122

2006
[32]

Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, and Nikhil Rao

Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C. Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, and Nikhil Rao. 2024. Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents. (2024). arXiv:2402.17896

2024 arXiv
[33]

Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. Commun. ACM 18, 11 (1975), 613–620

1975
[34]

Halevy, Nicola Tonellotto, and Fabrizio Silvestri

Artsiom Sauchuk, James Thorne, Alon Y. Halevy, Nicola Tonellotto, and Fabrizio Silvestri. 2022. On the Role of Relevance in Natural Language Processing Tasks. In Proc. SIGIR. 1785–1789

2022
[35]

Khan Tajmir, Rashid Umer, and Rehman Abdur. 2024. End-to-end pseudo rele- vance feedback based vertical web search queries recommendation. Multimedia Tools and Applications 83, 31 (2024), 75995–76033

2024
[36]

Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, and Luke Gallagher

Johanne R. Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, and Luke Gallagher. 2024. What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild. In Proc. SIGIR. 2703–2707

2024
[37]

Lalit Kumar Tyagi, Anish Gupta, and Vibhash Singh Sisodia. 2023. A New Era of Web Mining: Innovative Approaches in Focused Web Crawling for Domain- Specific Information. In Proc. ICTACS. 1–6

2023
[38]

Ryen W. White. 2024. Advancing the Search Frontier with AI Agents. Commun. ACM 67, 9 (2024), 54–65

2024
[39]

Yutong Xie, Zhaoying Pan, Jinge Ma, Luo Jie, and Qiaozhu Mei. 2023. A Prompt Log Analysis of Text-to-Image Generation Systems. In Proc. WWW. 3892–3902

2023
[40]

Shi Yu, Zhiyuan Liu, and Chenyan Xiong. 2025. Craw4LLM: Efficient Web Crawling for LLM Pretraining. (2025). arXiv:2502.13347

2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [31]

Francesca Pezzuti, Ariane Mueller, Sean MacAvaney, and Nicola Tonellotto. 2025. Document Quality Scoring for Web Crawling. (2025). arXiv:2504.11011

work page Pith review arXiv 2025

[2] [1]

Web Structure, Dynamics and Page Quality

Ricardo Baeza-Yates, Felipe Saint-Jean, and Carlos Castillo. 2002. "Web Structure, Dynamics and Page Quality". In Proc. SPIRE. 117–132

work page 2002

[3] [2]

Monica Bianchini, Marco Gori, and Franco Scarselli. 2005. Inside PageRank.ACM TOIT 5 (2005), 92–128

work page 2005

[4] [3]

Paolo Boldi, Massimo Santini, and Sebastiano Vigna. 2004. Do Your Worst to Make the Best: Paradoxical Effects in PageRank Incremental Computations. In Proc. W A W. 168–180

work page 2004

[5] [4]

Soumen Chakrabarti, Martin van den Berg, and Byron Dom. 1999. Focused crawling: a new approach to topic-specific Web resource discovery. Computer Networks 31, 11-16 (1999), 1623–1640

work page 1999

[6] [5]

Xuejun Chang, Debabrata Mishra, Craig Macdonald, and Sean MacAvaney. 2024. Neural Passage Quality Estimation for Static Pruning. In Proc. SIGIR. 174–185

work page 2024

[7] [6]

Jia Chen, Jiaxin Mao, Yiqun Liu, Fan Zhang, Min Zhang, and Shaoping Ma. 2021. Towards a Better Understanding of Query Reformulation Behavior in Web Search. In Proc. WWW. 743–755

work page 2021

[8] [7]

Qi Chen, Xiubo Geng, Corby Rosset, Carolyn Buractaon, Jingwen Lu, Tao Shen, Kun Zhou, Chenyan Xiong, Yeyun Gong, Paul Bennett, Nick Craswell, Xing Xie, Fan Yang, Bryan Tower, Nikhil Rao, Anlei Dong, Wenqi Jiang, Zheng Liu, Mingqin Li, Chuanjie Liu, Zengzhong Li, Rangan Majumder, Jennifer Neville, Andy Oakley, Knut Magne Risvik, Harsha Vardhan Simhadri, Ma...

work page 2025