Pith. sign in

REVIEW 4 major objections 4 minor 36 references

This paper claims that the cumulative discovery curve and pairwise containment are two projections of a single urn process, so their disagreement quantifies population heterogeneity and recovers a persistent core fraction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:34 UTC pith:3PWPQPOD

load-bearing objection Useful discovery-curve derivation, undercut by an algebraic error in the two-component containment formula. the 4 major comments →

arxiv 2607.13636 v1 pith:3PWPQPOD submitted 2026-07-15 physics.soc-ph cs.DLcs.IR

Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls

Michael Paris , Hande Celikkanat , Luca Foppiano This is my paper
classification physics.soc-ph cs.DLcs.IR
keywords web archivecrawl coveragediscovery curveurn modeltwo-component modelURL lifetimepairwise containmentlongitudinal crawl
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to give web-archive operators a way to measure what their crawler actually sees — coverage, churn, and the split between a persistent core and an ephemeral shell — using only the URL counts already published with each crawl. It introduces the discovery curve, the cumulative number of distinct URLs seen over a window of consecutive crawls, and shows that under the standard urn model it has a closed-form expression in the same two parameters (per-round coverage c and survival α) that govern pairwise containment. The central claim is that containment and the discovery curve are two projections of one process: when the URL population is homogeneous, independent fits agree on (α,c), and any disagreement is itself a measurement of heterogeneity. Applied to two long-running archives with opposing crawl designs, the two projections disagree, and a two-component urn with a persistent core fraction κ reconciles them. The paper concludes that operators can recover a small dictionary of interpretable quantities — coverage, churn, unique rate, life expectancy, resolve time, hidden time, and core fraction — from aggregate crawl statistics alone.

Core claim

The paper's central discovery is that the cumulative discovery curve U(s,T) — the number of distinct URLs seen across a window of T crawls — is a closed-form function of the same urn parameters (α,c) as the pairwise containment g(Δt)=c α^Δt. Fitting the two projections independently on the same archive therefore becomes a falsifiability test: agreement confirms a homogeneous population; disagreement quantifies heterogeneity. On the two archives studied, the homogeneous fits disagree, and the paper shows that a two-component urn — an immortal, fully-sampled core of fraction κ plus an ephemeral shell with its own (α∂,c∂) — reconciles the two projections, with κ around 0.4 at domain granularity

What carries the argument

The central object is the discovery curve U(s,T): the cumulative URL footprint over a sliding window of T crawls starting at crawl s. Under the homogeneous urn — each round samples a fraction c of the population and retains fraction α — it follows a closed form with a linear tail; the asymptotic slope ν∞ = (1−α)/(1−α(1−c)) and intercept I = 1 + ρ∞ τ_h invert to (α,c). The load-bearing device is the two-component urn, which partitions the population into an immortal, fully-sampled core of fraction κ and an ephemeral shell with parameters (α∂,c∂). Containment then acquires a persistent floor at κ, and the discovery curve splits additively into a saturated core term plus the shell's homogeneous

Load-bearing premise

The reconciliation rests on assuming the core is immortal and fully sampled and the shell is homogeneous with a single (α∂,c∂); the paper's own residual on c∂ shows the shell is not homogeneous, so the fitted κ is only as trustworthy as that idealization.

What would settle it

Generate a synthetic homogeneous urn with known (α,c) using the paper's exact sampling-and-churn step, then run the two-projection fits: if the two homogeneous fits disagree or the two-component fit returns a nonzero κ, the disagreement diagnostic and core fraction are artifacts of the fitting procedure rather than measurements of real population structure.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • From the rolling unique-URL counts a crawler already publishes, an operator can estimate per-round coverage c, survival α, and the persistent-core fraction κ, with no external ground truth.
  • The two projections form an internal falsifiability check: if homogeneous fits to containment and the discovery curve agree, the simple urn is adequate; if they disagree, the gap is a quantitative measure of population heterogeneity.
  • The fitted parameters convert directly into operator language: life expectancy ℓ=α/(1−α), sampled time η=cℓ, and hidden time η̄=(1−c)ℓ, so a raw crawl sequence becomes a small dictionary of planning quantities.
  • The structural prediction — both projections converging on a single (κ,α,c) triple — holds across two archives with opposite crawl designs, suggesting the decomposition reflects the underlying URL population rather than one pipeline's artifacts.
  • The remaining disagreement on shell coverage c∂ is itself a diagnostic: it shows the shell is not homogeneous and defines the target for a rank-resolved κ(r) extension.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If κ is stable across policy changes, it could serve as a calibration constant for comparing coverage across archives or tracking policy drift over time — a use the paper does not develop.
  • The discovery-curve closed form could be inverted to estimate the distribution of hidden epochs per URL, letting operators distinguish dormant-but-alive URLs from churned ones without a rebirth mechanism — a distinction the paper's α absorbs.
  • Because downstream datasets are built by filtering crawl output, these estimates give a quantitative upper bound on how much live web content is structurally absent from any snapshot, independent of later filtering quality.
  • A direct test: apply the two-projection diagnostic to a synthetic homogeneous urn with known (α,c); if the two-component fit returns κ above the noise floor, the core fraction is an artifact of the fitting procedure rather than a real population structure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops an urn model of longitudinal web crawls in which each round samples a fraction c of a fixed-size URL population and then churns a fraction 1−α, replacing it with new URLs. It derives closed forms for pairwise containment g(Δt)=cα^Δt and for the cumulative discovery curve U(t), inverts the latter to recover (α,c), and extends the model to a two-component population with a persistent core fraction κ and an ephemeral shell (α∂,c∂). It applies the framework to Common Crawl (2020–2025, domain granularity) and the German Academic Web (URL granularity), claiming that independent fits to containment and to the discovery curve agree on a common κ≈0.4 for Common Crawl and on a single triple for GAW, while a residual on c∂ indicates shell heterogeneity.

Significance. If the central claim holds, the paper offers a practical way for crawl operators to infer coverage, persistence, and a core/shell decomposition from publicly published crawl statistics, without instrumenting the crawler. The derivations in §3.3–3.5 are explicit and algebraically checkable, and the operator-language translations (lifetime, sampled time, hidden time) are a useful contribution. The two-archive comparison (Common Crawl vs. German Academic Web) is a valuable cross-validation design. However, the central two-component result rests on a containment formula that is algebraically incorrect, and the empirical support lacks uncertainty quantification and full numeric reporting. These issues affect the paper's main claim and require substantial revision.

major comments (4)
  1. [§3.6, Eq. (16)] The two-component containment formula is algebraically incorrect. With M = N[κ+(1−κ)c∂] from Eq. (17), the core contributes κN intersections and the shell contributes N(1−κ)c∂² α∂^Δt, so the correct expression is g(Δt) = [κ+(1−κ)c∂² α∂^Δt] / [κ+(1−κ)c∂]. Eq. (16) omits the denominator and uses c∂ instead of c∂²; its long-run floor is κ, whereas the model's floor is μ=κ/[κ+(1−κ)c∂] (≈0.45 for κ≈0.4, c∂≈0.8). Eq. (18) already uses μ, so Eq. (16) is internally inconsistent with Eq. (18). The parameters reported in Fig. 4 and the claimed cross-source agreement on κ are therefore not parameters of the model as defined; the regressions must be re-run with the corrected formula.
  2. [§4.2–4.3] Empirical support is under-specified. No error bars, confidence intervals, or goodness-of-fit statistics are reported for any fitted (α,c,κ); the only numeric values in the text are κ(d)≈0.4 and the GAW triple (0.06,0.62,0.51). The central claim that the two projections 'agree' on κ requires a quantitative comparison with uncertainties; without them, the agreement may be coincidental, and the residual on c∂ (Fig. 4) is not quantified. Provide a table of fitted parameters, standard errors, and residual metrics for both archives.
  3. [§4.1] The mapping from the published url_last_N rolling counts to the discovery curve U(s,T) needs clarification for irregular cadence. url_last_N gives unique URLs over the last N crawls, but Common Crawl cadence varies (biweekly to 1–2 month gaps), so a fixed N corresponds to different time spans; Fig. 2b plots against elapsed time, while the model formulas (3)–(5) treat t as a discrete round index. State explicitly whether the fits use crawl index or wall-clock time, and how the rolling-window aggregation (which may count a URL once even if it appears in multiple crawls) matches the model's per-round deduplicated union.
  4. [§3.6/§5] The model's own residual on c∂ shows that the shell is not homogeneous, so the two-component urn is an idealization. The paper acknowledges this but still claims the two projections are 'reconciled' by κ. The reconciliation is conditional on a homogeneous-shell assumption that the data reject. Please quantify the residual and show that κ is robust to shell heterogeneity (e.g., by fitting a rank-resolved or multi-component shell), or temper the central claim accordingly.
minor comments (4)
  1. [§3.6] After Eq. (16), the text says the core contributes a 'persistent floor at κ'; given the correct normalization, the floor is μ, not κ. This wording should be corrected along with the formula.
  2. [§4.3] The GAW triple (κ,α,c)≈(0.06,0.62,0.51) is not clearly defined: is this the homogeneous fit or the two-component fit? Are these (κ,α∂,c∂) or (κ,α,c)? Clarify which parameters are being reported.
  3. [§1, §4] The paper uses 'cross-source' to mean two projections within the same archive and also for the two-archive comparison. Consider 'cross-projection' for the former to avoid ambiguity.
  4. [References] References [22] and [35] are arXiv preprints; if published versions exist, they should be cited.

Circularity Check

0 steps flagged

No significant circularity: the two-projection agreement is an over-identification test, and the main self-citation is not load-bearing.

full rationale

The derivation chain is not circular. The containment formula g=c α^Δt is cited from the same first author's prior work [22], but it is a two-line consequence of the urn assumptions in §3.1 (a sampled URL survives Δt rounds with probability α^Δt and is sampled in the later crawl with probability c), and it is tested against two external archives rather than assumed into the fitted values; the self-citation is therefore not load-bearing. The central 'two projections' claim is an over-identification check: parameters fitted independently to containment and to the discovery curve are overlaid on both observables (Fig. 3, Fig. 4), so agreement is not achieved by construction. When the homogeneous fits disagree, the two-component model adds one parameter and is then re-fit to each projection; the reported convergence on κ≈0.4 is a non-vacuous consistency across two different estimating equations, and the paper's own admitted residual on c∂ (§4.2, §5) shows the reconciliation is not artificially forced. The GAW check is a genuinely external archive. The manuscript also flags limitations ('This is a conjecture, not a theorem'; 'the shell itself is not homogeneous'), which weakens the scientific claim but does not indicate circularity. The algebraic objection to Eq. (16) — missing crawl-size normalization and c∂ vs. c∂² — would be a correctness/model-specification issue, not a circularity, and does not affect this verdict.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

The model uses a small number of fitted parameters (α, c, κ, α∂, c∂) to describe crawl dynamics. The core/shell decomposition is an ad hoc extension to reconcile two observables, and the assumptions of a rigid core and homogeneous shell are not independently verified. The free parameters are all estimated from the same public crawl statistics that the paper aims to explain.

free parameters (4)
  • per-round survival α = ≈0.62 (GAW URL level); varies by archive
    Recovered from containment and discovery curve fits; a free parameter of the urn model estimated from data.
  • per-round coverage c = ≈0.78 (Common Crawl domain), ≈0.51 (GAW URL)
    Estimated from the same fits; describes the fraction of the urn sampled each round.
  • core fraction κ = ≈0.4 (Common Crawl domain), ≈0.06 (GAW URL)
    Fitted as the containment floor or via two-component discovery curve; a free parameter introduced to reconcile the two projections.
  • shell survival α∂ and coverage c∂ = Not reported numerically in text; figure legends only
    Fitted shell parameters in the two-component model; values are not given in the prose, only in figures.
axioms (6)
  • domain assumption Uniform random sampling each round: each element is drawn with equal probability c, independent of rank or history.
    Real crawlers use host budgets and harmonic-centrality ranking, so this is an idealization. Invoked in §3.1.
  • domain assumption Constant urn size N each round, with churn replacing exactly N(1-α) elements by brand-new elements.
    Assumes population size and churn volume are stable; invoked throughout §3.
  • domain assumption No rebirth: an element that churns out never returns; any apparent comeback is modeled as an extended missed run.
    Explicitly acknowledged in §5; affects the interpretation of α.
  • ad hoc to paper Immortal, fully-sampled core: a fraction κ of URLs have (αK,cK)=(1,1).
    Introduced to explain the containment floor and reconcile the two projections; no independent evidence for a strictly rigid core. Invoked in §3.6.
  • ad hoc to paper Homogeneous shell: all non-core URLs share a single (α∂,c∂).
    The paper's own residual on c∂ contradicts this; the model is knowingly incomplete. Invoked in §3.6 and §4.2.
  • domain assumption The published rolling unique-URL statistics (url_last_N) correspond exactly to the cumulative distinct URLs over consecutive crawls.
    Pivoting these aggregates into U(s,T) assumes they are precise deduplicated unions; not independently verified.
invented entities (2)
  • Persistent core K no independent evidence
    purpose: Explains the persistent containment floor and reconciles the two projections by splitting the URL population into an immortal component.
    The core/shell split is a model construct fitted to the same data; no external measurement or mechanism is given to confirm that such a rigid class of URLs exists.
  • Ephemeral shell ∂K no independent evidence
    purpose: Carries the churning URL mass and accounts for fresh discoveries and the transient part of the discovery curve.
    The shell is defined as the residual after removing the core; its heterogeneity is acknowledged by the residual on c∂, but no independent evidence for a single shell component is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 11828 in / 12002 out tokens · 110512 ms · 2026-08-02T04:34:48.759190+00:00 · methodology

0 comments
read the original abstract

A longitudinal web crawl is a sequence of partial samples of an evolving URL population. Pairwise containment between two crawls is the standard probe; under a simple \emph{urn} model of the crawl -- each round samples a fraction of the URLs and replaces a fraction -- it recovers two interpretable rates, per-round survival $\alpha$ and coverage $c$, but treats the population as uniform and consumes one pair at a time. In this work, we define a formal language for talking about a crawl. We extend this analysis with the \emph{discovery curve} $U(s, T)$, the cumulative URL footprint over a sliding window of $T$ crawls starting at $s$, which under the same urn model is also a closed-form function of $(\alpha, c)$. Containment and the discovery curve are then two projections of one process: independent fits agree on $(\alpha, c)$ when the urn is homogeneous, so any disagreement is itself a measurement. Applied to Common Crawl (2020--2025, domain granularity) and to the German Academic Web (GAW, URL granularity), the two projections disagree on both archives, and a two-component urn with a persistent core fraction $\kappa$ alongside shell parameters $(\alpha_\partial, c_\partial)$ reconciles the disagreement. A residual on $c_\partial$ remains, signaling that the shell itself is not homogeneous; $\kappa$ is recorded as the scalar entry point to a rank-resolved generalization, which is left to follow-up work. \keywords{web archive \and crawl coverage \and discovery curve \and urn model \and two-component model \and URL lifetime}

Figures

Figures reproduced from arXiv: 2607.13636 by Hande Celikkanat, Luca Foppiano, Michael Paris.

Figure 1
Figure 1. Figure 1: Per-round area decomposition of the urn. The left box [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Discovery curve on Common Crawl, 2020–2025. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Common Crawl — homogeneous cross-source view at domain granularity, 2020–2025. Both panels overlay the domain-level data with two homogeneous urn curves: burgundy uses (αg, cg) fit on containment, navy uses (αU , cU ) fit on the discovery curve. If the domain population were homoge￾neous, the two curves would coincide on both panels; the size of the gap is the homogeneous-fit failure on this window and mot… view at source ↗
Figure 4
Figure 4. Figure 4: Common Crawl — two-component cross-source view at domain granularity, 2020–2025. Same domain observables as [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: German Academic Web — two-component validation. The same two-component cross-source check as [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 5 canonical work pages

  1. [1]

    In: Proceedings of the 2024 ACM Conference on F airness, Ac- countability, and Transparency

    Baack, S.: A critical analysis of the largest source for ge nerative ai training data: Common crawl. In: Proceedings of the 2024 ACM Conference on F airness, Ac- countability, and Transparency. pp. 2199–2208 (2024)

    2024

  2. [2]

    Computer Networks and ISDN syste ms 30(1-7), 379– 388 (1998)

    Bharat, K., Broder, A.: A technique for measuring the rela tive size and overlap of public web search engines. Computer Networks and ISDN syste ms 30(1-7), 379– 388 (1998)

    1998

  3. [3]

    https://doi.org/10.1016/S1389-1286(00)00045-1

    Brewington, B.E., Cybenko, G.: How dynamic is the Web? Com puter Networks 33(1–6), 257–276 (2000). https://doi.org/10.1016/S1389-1286(00)00045-1

    work page doi:10.1016/s1389-1286(00)00045-1 2000

  4. [4]

    Biometrics 43(4), 783–791 (1987)

    Chao, A.: Estimating the population size for capture-rec apture data with unequal catchability. Biometrics 43(4), 783–791 (1987). https://doi.org/10.2307/2531532

    work page doi:10.2307/2531532 1987

  5. [5]

    University of California Pub lications on Statistics 1, 131–160 (1951)

    Chapman, D.G.: Some properties of the hypergeometric dis tribution with applica- tions to zoological censuses. University of California Pub lications on Statistics 1, 131–160 (1951)

    1951

  6. [6]

    In: Proc

    Cho, J., Garcia-Molina, H.: The evolution of the web and im plications for an incremental crawler. In: Proc. VLDB. pp. 200–209 (2000)

    2000

  7. [7]

    In: Proceedings of the 2000 ACM SIGMOD International Conferenc e on Management of Data

    Cho, J., Garcia-Molina, H.: Synchronizing a database to i mprove freshness. In: Proceedings of the 2000 ACM SIGMOD International Conferenc e on Management of Data. pp. 117–128. ACM (2000). https://doi.org/10.1145/342009.335391

    arXiv 2000

  8. [8]

    In: Proceedings of the 20 21 Conference on Em- pirical Methods in Natural Language Processing (EMNLP)

    Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., Gardner, M.: Documenting large webtext corpo ra: A case study on the Colossal Clean Crawled Corpus. In: Proceedings of the 20 21 Conference on Em- pirical Methods in Natural Language Processing (EMNLP). pp . 1286–1305 (2021). https://doi.org/10.18653/v1/2021.emnlp-main.98

    doi:10.18653/v1/2021.emnlp-main.98 2021

  9. [9]

    Zeitschrift für Angewandte Mathematik und Mechanik 3(4), 279–289 (1923)

    Eggenberger, F., Pólya, G.: Über die statistik verkettet er vorgänge. Zeitschrift für Angewandte Mathematik und Mechanik 3(4), 279–289 (1923). https://doi.org/10.1002/zamm.19230030407

    work page doi:10.1002/zamm.19230030407 1923

  10. [10]

    arXiv prepri nt arXiv:2101.00027 (2020), https://arxiv.org/abs/2101.00027

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., Leahy, C.: The Pile: An 800GB dataset of diverse text for language modeling. arXiv prepri nt arXiv:2101.00027 (2020), https://arxiv.org/abs/2101.00027

    Pith/arXiv arXiv 2020

  11. [11]

    In: Proceed- ings of the 6th international conference on Web engineering

    Gomes, D., Silva, M.J.: Modelling information persiste nce on the web. In: Proceed- ings of the 6th international conference on Web engineering . pp. 193–200 (2006)

    2006

  12. [12]

    Biometrika 40(3-4), 237–264 (1953)

    Good, I.J.: The population frequencies of species and th e estima- tion of population parameters. Biometrika 40(3-4), 237–264 (1953). https://doi.org/10.1093/biomet/40.3-4.237

    doi:10.1093/biomet/40.3-4.237 1953

  13. [13]

    Aca- demic Press (1978)

    Heaps, H.S.: Information Retrieval: Computational and Theoretical Aspects. Aca- demic Press (1978)

    1978

  14. [14]

    arXiv preprint arXiv:2001.08361 (2020), https://arxiv.org/abs/2001.08361

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., C hess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural l anguage models. arXiv preprint arXiv:2001.08361 (2020), https://arxiv.org/abs/2001.08361

    Pith/arXiv arXiv 2001

  15. [15]

    Journal of the American society for information science and technology 53(2), 162–171 (2002)

    Koehler, W.: Web page change and persistence—a four-yea r longitudinal study. Journal of the American society for information science and technology 53(2), 162–171 (2002)

    2002

  16. [16]

    Kolchin, V.F., Sevast’yanov, B.A., Chistyakov, V.P.: R andom Allocations. V. H. Winston & Sons, Washington, D.C. (1978)

    1978

  17. [17]

    Science 280(5360), 98–100 (1998)

    Lawrence, S., Giles, C.L.: Searching the World Wide Web. Science 280(5360), 98–100 (1998). https://doi.org/10.1126/science.280.5360.98 Discovery Curves and Core Persistence 15

    work page doi:10.1126/science.280.5360.98 1998

  18. [18]

    In: Advances in Neural In formation Processing Systems (NeurIPS), Datasets and Benchmarks Track (2024)

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre , S., Bansal, H., Guha, E., Keh, S., Arora, K., et al.: DataComp-LM: In search of the n ext generation of training sets for language models. In: Advances in Neural In formation Processing Systems (NeurIPS), Datasets and Benchmarks Track (2024)

    2024

  19. [19]

    US Department of Agriculture Circular 118, 1–4 (1930)

    Lincoln, F.C.: Calculating waterfowl abundance on the b asis of banding returns. US Department of Agriculture Circular 118, 1–4 (1930)

    1930

  20. [20]

    Foundations and Tr ends in Information Retrieval 4(3), 175–246 (2010)

    Olston, C., Najork, M.: Web crawling. Foundations and Tr ends in Information Retrieval 4(3), 175–246 (2010). https://doi.org/10.1561/1500000017

    work page doi:10.1561/1500000017 2010

  21. [21]

    In: Pro ceedings of the 58th Annual Meeting of the Association for Computational Lingui stics (ACL)

    Ortiz Suárez, P.J., Romary, L., Sagot, B.: A monolingual approach to contextu- alized word embeddings for mid-resource languages. In: Pro ceedings of the 58th Annual Meeting of the Association for Computational Lingui stics (ACL). pp. 1703– 1714 (2020). https://doi.org/10.18653/v1/2020.acl-main.156

    doi:10.18653/v1/2020.acl-main.156 2020

  22. [22]

    arXiv preprint arXiv:260 3.15416 (2026)

    Paris, M., Paris, G., Baumann, F.: Estimating absolute w eb crawl coverage from longitudinal set intersections. arXiv preprint arXiv:260 3.15416 (2026)

    2026

  23. [23]

    In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track (2024)

    Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mit chell, M., Raffel, C., Von Werra, L., Wolf, T.: The FineWeb datasets: Decanting the web for the finest text data at scale. In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track (2024)

    2024

  24. [24]

    In: Advances in Neural Information Processing System s (NeurIPS) (2023), https://arxiv.org/abs/2306.01116

    Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cap pelli, A., Alobei- dli, H., Pannier, B., Almazrouei, E., Launay, J.: The Refined Web dataset for Falcon LLM: Outperforming curated corpora with web data, an d web data only. In: Advances in Neural Information Processing System s (NeurIPS) (2023), https://arxiv.org/abs/2306.01116

    Pith/arXiv arXiv 2023

  25. [25]

    Report of the Danish Biological Station to th e Home Department 6, 1–48 (1896)

    Petersen, C.G.J.: The yearly immigration of young plaic e into the Limfjord from the German Sea. Report of the Danish Biological Station to th e Home Department 6, 1–48 (1896)

  26. [26]

    Journal of Machine Learning Research 21(140), 1–67 (2020), https://jmlr.org/papers/v21/20-074.html

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text- to-text transformer. Journal of Machine Learning Research 21(140), 1–67 (2020), https://jmlr.org/papers/v21/20-074.html

    2020

  27. [27]

    The American Mathematical Monthly 45(6), 348–352 (1938)

    Schnabel, Z.E.: The estimation of the total fish populati on of a lake. The American Mathematical Monthly 45(6), 348–352 (1938). https://doi.org/10.1080/00029890.1938.11990818

    arXiv 1938

  28. [28]

    In: Procee dings of the 62nd Annual Meeting of the Association for Computational Linguistics ( ACL)

    Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkin son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al.: Dolma: an open c orpus of three trillion tokens for language model pretraining research. In: Procee dings of the 62nd Annual Meeting of the Association for Computational Linguistics ( ACL). pp. 15725–15788 (2024), https://acl...

    2024

  29. [29]

    In: Proceedings of the 6th Inte rnational Work- shop on Consuming Linked Data (COLD’15) (2015)

    Stolz, A., Hepp, M.: Towards Crawling the Web for Structu red Data: Pitfalls of Common Crawl for E-Commerce. In: Proceedings of the 6th Inte rnational Work- shop on Consuming Linked Data (COLD’15) (2015)

    2015

  30. [30]

    Su, D., et al.: Nemotron-CC: Transforming Common Crawl i nto a refined long- horizon pretraining dataset (2024)

    2024

  31. [31]

    , Marivate, V., et al.: CommonLID: Re-evaluating state-of-the-art language iden tification performance on web data

    Suarez, P.O., Burchell, L., Arnett, C., Mosquera-Gómez , R., Hincapie-Monsalve, S., Vaughan, T., Stewart, D., Ostendorff, M., Abdulmumin, I. , Marivate, V., et al.: CommonLID: Re-evaluating state-of-the-art language iden tification performance on web data. arXiv preprint arXiv:2601.18026 (2026)

    Pith/arXiv arXiv 2026

  32. [32]

    In: Proceedings of the 16th ACM Web Science Confer ence

    Thompson, H.S.: Improved methodology for longitudinal web analytics using com- mon crawl. In: Proceedings of the 16th ACM Web Science Confer ence. pp. 59–69 (2024) 16 M. Paris et al

    2024

  33. [33]

    https://github.com/togethercomputer/RedPajama-Data (Apr 2023)

    Together Computer: RedPajama: An open source recipe to r eproduce LLaMA training dataset. https://github.com/togethercomputer/RedPajama-Data (Apr 2023)

    2023

  34. [34]

    In: Proceedin gs of the 15th Interna- tional Conference on World Wide Web

    Toyoda, M., Kitsuregawa, M.: What’s really new on the web ?: Identifying new pages from a series of unstable web snapshots. In: Proceedin gs of the 15th Interna- tional Conference on World Wide Web. pp. 233–241. WWW ’06, AC M, New York, NY, USA (2006). https://doi.org/10.1145/1135777.1135815

    arXiv 2006

  35. [35]

    arXiv preprint arXiv:2508.18544 (2025)

    Vanni, F., Lambert, D.: Urn modeling of random graphs acr oss granularity scales: A framework for origin-destination human mobility network s. arXiv preprint arXiv:2508.18544 (2025)

    Pith/arXiv arXiv 2025

  36. [36]

    In: Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Tra ck (2024)

    Weber, M., Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexa ndrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala , R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., Zhang, C.: RedPajama: an open dataset for training large language models. In: Advances in Neural Information Processing Systems (NeurIPS), Datasets an...

    2024