Pith. sign in

REVIEW 3 major objections 6 minor 12 references

OriginBlame: Record- and Token-Level Data Provenance for AI Training Datasets

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hash-linked provenance index turns author revocation into precise line- and token-level forget sets.

desk verdict Solid provenance system with a suspicious headline unlearning result; the 42% claim likely confounds set composition with selection quality. read the letter →

arxiv 2607.13037 v1 pith:FGZN2YOM submitted 2026-05-19 cs.AI

Haolin Xue This is my paper · ORCID
classification cs.AI
keywords dataprovenancemachineunlearningforgetsetauthorrevocationcontent-addressableindexingtoken-levelNPOover-deletion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the bottleneck in data removal from AI training is not the unlearning algorithm but the absence of a forget set. Its system, OriginBlame (ob), attaches author identity to each training record as it is written, carries that identity through tokenization and packing via SHA-256 hash links, and answers revocation requests with deterministic reverse queries. On 219k Wikipedia pages this cuts over-deletion from 101x to 1.3x, and on a 1.7B model a provenance-built forget set outperforms a same-size random one by 42% in forget perplexity while also improving retain perplexity by 23%. The system explicitly does not infer authorship: it propagates whatever attribution the source environment already has.

What carries the argument

The three-layer content-addressable index (authors ← sections ← document-index) linked by SHA-256 hashes, plus an independent token-index layer storing cumulative token ranges. The workhorse is the reverse query: author id → matching section hashes → document-index entries, which yields a forget set without reading data files. The token-index's cumulative-count trick means each entry knows its token range from the sum of predecessors, giving position-indexed provenance with no per-token records.

What would settle it

If a real crawl or dataset passes through processing stages that strip authorship before ob.track() can run, the provenance chain is empty and the 1.3x over-deletion claim does not materialize. A direct experiment: take a corpus, build ob provenance, then run NPO with ob's forget set versus an oracle forget set of the same author's true lines — if unlearning metrics are statistically equal, the system's precision is not buying the stated improvement.

Watch

Extended reading notes

Core claim

The central discovery is architectural: provenance at record and token granularity can be made cheap and queryable by storing three hash-linked layers — authors, sections, document-index — and a parallel token-index that records cumulative token counts during tokenization. Revocation is a lazy tag cascade: setting revoked=true on an author record, then traversing author → sections → document-index to produce the exact (file, line_hash) list. The token-index produces a bitmask over token-index entries directly usable by unlearning code, without per-token records. The paper's evidence that this matters is the NPO experiment: the same unlearning algorithm receives either ob's line-level forget

Load-bearing premise

OriginBlame's entire benefit depends on author identity being present when data first enters the pipeline — it propagates attribution but cannot discover it, and the paper concedes that incremental adoption is impossible once attribution has already been lost.

Editorial extensions

If this is right

  • Author revocation requests can be fulfilled at line or token granularity, eliminating dataset-level over-deletion for training corpora.
  • Unlearning algorithms like NPO can be fed provenance-built forget sets and show large quality gains; token-level bitmasks give a standard interface.
  • Provenance is auditable: an append-only operation log supports compliance checks, and the system needs no model access or GPU.
  • The same integration pattern works beyond text — git blame attribution for source code shows line-level provenance is domain-agnostic.
  • Because revocation is a reversible tag, compliance actions can be tested and rolled back before purge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If provenance-recording becomes standard at ingestion, the whole unlearning pipeline can be reframed: the hard part shifts from 'which algorithm' to 'what metadata do we keep at write time'.
  • The token-index implies a path to per-token attribution, which would connect to token-level unlearning and fine-grained copyright claims.
  • The 42/23 percent split suggests forget-set precision may matter more than algorithm choice; a fair test would pit provenance-built sets against influence-function-based selection with the same budget.
  • In collaborative corpora, line-level revocation still deletes co-authored content; the paper's data shows 100% of records sit in multi-author sections, so true author-level compliance may require partial attribution weights.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. OriginBlame (ob) is a provenance system that propagates authorship metadata through data-processing pipelines. It stores records in a three-layer content-addressable index (authors, sections, document-index) plus an optional token-index layer for packed/tokenized data. Queries resolve contributor identities into record- or token-level forget sets. The paper evaluates on zhwiki (219,555 pages) and Linux kernel sources, measuring revocation precision, reconcile recovery, query latency, framework overhead (HuggingFace Datasets, Datatrove), and a downstream NPO/RMU/gradient-ascent unlearning study on Qwen3-1.7B. Headline results are reduced over-deletion (101x to 1.3x), modest integration overhead, and a 42% improvement in NPO forget PPL for provenance-selected vs. random forget sets.

Significance. If the main unlearning comparison is valid, the paper addresses a real, under-served problem: converting an author-level revocation request into a fine-grained forget set for machine unlearning. The architecture is simple and deterministic, the core has no ML/GPU dependency, and the integration surface is small. The cross-domain Linux kernel experiment is a useful generality check. The strengths include machine-checkable deterministic query design and a clear separation of propagation from authorship discovery. However, the strongest downstream claim is not currently well supported (see Major Comments), and the token-index contribution is validated only at the level of overhead and latency, not correctness or downstream use. The paper's scope is also narrower than the abstract suggests because ob propagates, not discovers, authorship; this is acknowledged in Discussion but should be foregrounded early.

major comments (3)
  1. [§5.7, Table 8] The key unlearning comparison is not interpretable as reported. The two conditions use different forget sets—target-author lines vs. a same-size random subset of the corpus—and the text says all metrics are computed 'from the respective splits.' If that means the random condition's forget PPL/ROUGE is measured on the randomly selected lines, the comparison only shows that NPO degrades the model on whichever lines it was trained to forget; it does not show better targeting of the requester's content. The SFT row in Table 8 is also undefined under this reading because it cannot be a common baseline for two different splits. Fix the protocol: evaluate both conditions on a fixed held-out sample of the target author's lines and a fixed retain sample, and report the random condition's own-set metrics as secondary.
  2. [§5.7, experimental protocol] The unlearning experiments appear to be a single run per condition with no seeds, variance estimates, or significance tests. Given stochastic QLoRA/NPO, differences like retain PPL 4.23 vs 5.89 (Berthe) or 4.93 vs 6.04 (Antigng) could be within run-to-run noise. Evaluation subsets are also small (1,000 lines for PPL, 100 for ROUGE-L). Please report multiple seeds with means and standard deviations, and state whether the random forget set excludes target-author lines; if it does not, report the overlap between the random and line-level sets.
  3. [§4.5, §5.5] The token-level layer is advertised as a third contribution, but the evaluation validates only throughput, latency, and metadata preservation. There is no check that generate-set bitmasks correspond to the correct token ranges after ob clean merging, no correctness test against ground-truth token spans, and no downstream token-level unlearning experiment. Without this, the token-level claim remains a design assertion rather than a demonstrated capability. Add a correctness/alignment check and, if feasible, a small token-level forget-set experiment.
minor comments (6)
  1. [§5.2/§5.7] The terms 'record,' 'line,' and 'page' are used somewhat interchangeably. Clarify the unit of provenance: is a record one wiki page, one section, or one JSONL line? This affects how to read Table 3 and the unlearning forget sets.
  2. [§5.3, Appendix B] The reconcile mutation replaces 10% of alphabetic characters with random lowercase letters, which is ill-suited to zhwiki's predominantly Han text. Many lines may contain no alphabetic characters and therefore remain unchanged, inflating the hash-recovery rate. Report the actual fraction of mutated lines whose hashes changed, or use a CJK-appropriate mutation.
  3. [§5.3] Reconcile relies on nomic-embed-text-v1.5 with cosine threshold 0.85. No sensitivity analysis is provided; report how recovery varies with threshold and embedding model.
  4. [§5.5, Table 6] Datatrove throughput overhead is nonmonotonic (13.8%, 19.0%, 13.4%, 2.1%). Explain the pattern, and fill the missing full-scale Datatrove entry in Table 7.
  5. [General] The manuscript cites GitHub URLs for code but does not include an artifact or data in the submission. Please provide a stable commit hash, evaluation scripts, and the zhwiki-derived QA data or a generation script to support reproducibility.
  6. [§4.1] Minor wording issue: 'which ob clean later merges' is grammatically awkward; also, the WAL notation 'lock.{pid}→docidx.{pid}' should be defined the first time it appears.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the provenance derivation is self-contained and deterministic.

full rationale

OriginBlame's core chain—authors → sections → document-index/token-index—is a content-addressable storage and query design, not a statistical derivation. The forget set F={l∈D|author(l)=a_i} is the direct result of an ob show/revoke query over recorded metadata; there are no fitted parameters, no inverse mapping, and no prediction that is statistically forced. The over-deletion factors in Table 3 are arithmetic ratios of dataset-level to record-level deletion sizes. The reconcile numbers are measured recovery rates on synthetic mutations with a fixed seed and threshold, not fitted to the target result. The unlearning experiment in Section 5.7 compares a provenance-selected forget set against a same-size random subset; this is a legitimate head-to-head treatment comparison, and the 42%/23% figures are computed from Table 8. One caveat, noted but not circular: 'All metrics are computed on sampled subsets ... from the respective splits' means each condition's forget PPL is measured on its own forget set, so the random baseline's forget PPL is not measured on the target author's lines; this is an internal-validity concern for the downstream unlearning claim, not a circularity in the provenance derivation. The paper is also explicit that ob propagates rather than discovers authorship (Section 4.1) and lists incremental adoption as a limitation, which further confirms the system's claims are conditional on external metadata rather than self-referential. No self-citation is load-bearing; no uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are invented. The system's free parameters are limited to the reconcile threshold and (inherited) unlearning hyperparameters; the load-bearing assumption is upstream author attribution, which the paper explicitly states it does not discover.

free parameters (2)
  • nomic-embed-text-v1.5 cosine threshold 0.85 = 0.85
    Chosen by hand for the reconcile semantic-matching pass (Section 5.3). The reported recovery rates (96-98%) depend on this threshold, which is not fitted to a validation set and has no sensitivity analysis.
  • Unlearning hyperparameters (NPO beta 0.1, RMU alpha 100) = beta=0.1, alpha=100
    Taken from prior work / chosen without a sweep shown. Since the claim is relative (line vs random under identical hyperparameters), this is a less serious dependency, but the absolute effect sizes may shift with different hyperparameters.
assumptions (4)
  • standard math SHA-256 hashing with no normalization is collision-free at the evaluated scale
    Section 4.1 cites the birthday bound p < 1e-68 at n=2.2e5, k=256. Standard cryptographic assumption.
  • domain assumption Author identity is available from the source environment (MediaWiki revision history, git blame)
    The entire system propagates, not discovers, authorship. Section 4.1 and the limitations section make this explicit. All evaluation numbers depend on this assumption holding in the source data.
  • domain assumption Wiki pages and kernel files have complete contributor lists in the source dump
    Section 5.1 assumes the zhwiki dump's contributor list is accurate and complete; Section 5.6 assumes git blame gives truth for kernel authors. If these lists are incomplete, the forget sets are incomplete.
  • domain assumption Token counts of a document are identical across workers and merge order does not change the packed binary
    Section 4.5 relies on cumulative token offsets to reconstruct token ranges. The paper does not discuss tokenizer nondeterminism or sharding order effects on the packed binary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OriginBlame: Record- and Token-Level Data Provenance for AI Training Datasets." pith.science (2026). https://pith.science/paper/FGZN2YOM

@misc{pith2026260713037,
  author       = {Pith},
  title        = {Pith review of: OriginBlame: Record- and Token-Level Data Provenance for AI Training Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGZN2YOM}},
  note         = {Machine review of arXiv:2607.13037}
}
read the original abstract

When a data contributor requests removal, model trainers face a practical gap: unlearning algorithms require a forget set, yet no tool can locate which training records belong to a given author. Existing provenance systems operate at file or dataset level, forcing catastrophic over-deletion. We present ob, a record- and token-level data provenance system that propagates author identity through data processing pipelines and resolves revocation requests into precise forget sets via deterministic queries. Evaluation on 219,555 Wikipedia pages demonstrates that record-level provenance eliminates dataset-level over-deletion (from 101x to 1.3x), while integration adds 1.3-4.0% throughput overhead (HuggingFace) and 2.1-19.0% (Datatrove) on wiki data. On a 1.7B model, provenance-based forget sets improve unlearning by 42% over random baselines.

Figures

Figures reproduced from arXiv: 2607.13037 by the authors.

Figure 1
Figure 1. The unlearning pipeline gap. Existing meth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Three-layer reference architecture with hash [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The track() workflow. After hash computation, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Revocation levels in ob. Author revoke cas [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Two-phase reconcile strategy. Phase 1 at [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: NPO advantage of ob line-level forget sets [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 3 linked inside Pith

  1. [1]

    Look up author entries in the index → obtain section bucket prefixes

  2. [2]

    Read those section shards to find matching sections → look up section entries in the index →obtain document-index bucket prefixes

  3. [3]

    purge–author –index follows the same three- hop path but returns only (line_hash, file) pairs without expanding section and author details

    Read only those document-index shards to collect results. purge–author –index follows the same three- hop path but returns only (line_hash, file) pairs without expanding section and author details. Implementation details.Two optimizations re- duce I/O:

  4. [4]

    id": "<hash>

    MUSE: Machine unlearning six-way evalu- ation for language models. InProceedings of the International Conference on Learning Representa- tions (ICLR). The Linux Foundation. 2024. Delta Lake: Open-source storage layer for data lakes.https://delta.io. 11 Dennis Trautwein, Aravindh Raman, Gareth Tyson, Ig- nacio Castro, Will Scott, Moritz Schubotz, Bela Gipp...

    arXiv 2024

  5. [8]

    Batched bucket reads.At each hop, IDs are grouped by their 2-character prefix so each shard file is read at most once, even when hundreds of IDs map to the same bucket

  6. [9]

    Unified binary index.The binary index (OBIDXF02) uses type-tagged references to serve both the document-index and token-index layers

    Batch pre-loading.For show, all author records referenced by matched sections are loaded into memory in a single pass over the author shards before the document-index loop, replacing per-resultget_author() calls with constant-time dict lookups. Unified binary index.The binary index (OBIDXF02) uses type-tagged references to serve both the document-index an...

  7. [10]

    Lines with r <0.10 are edited by replacing 10% of their alphabetic characters with random lowercase letters

    Edit(10%): Each line is independently clas- sified by drawing r∼U(0,1) . Lines with r <0.10 are edited by replacing 10% of their alphabetic characters with random lowercase letters. For JSON records, only the text field is modified; other fields (metadata, section hashes) are preserved

  8. [11]

    Delete(5%): Lines with 0.10≤r <0.15 are removed entirely

Show all 12 references
  1. [12]

    Insertions are placed at random positions

    Insert(5%): 0.05×N new lines are gener- ated by copying random existing lines and modifying 15% of their alphabetic characters. Insertions are placed at random positions. 12 This design preserves the character-set distri- bution of the original data (unlike random byte- flippi...

  2. [2022]

    Nathaniel Li, Alexander Pan, Anjali Gopal, Sum- mer Yue, Daniel Berrios, Alice Gatti, Justin D

    The stack: 3 TB of permissively licensed source code.Transactions on Machine Learning Research. Nathaniel Li, Alexander Pan, Anjali Gopal, Sum- mer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin...

    2024 arXiv

  3. [2025]

    Databricks

    The forget-set identification problem.Ma- chine Learning, 114(247). Databricks. 2024. MLflow: A platform for the machine learning lifecycle.https://mlflow.org. Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. 2024. Attr...

    2024 arXiv

  4. [2026]

    InProceedings of The Web Conference 2026 (WWW ’26), New York, NY , USA

    Fine-grained traceability for transparent ML pipelines. InProceedings of The Web Conference 2026 (WWW ’26), New York, NY , USA. ACM. Andrea D’Angelo, Francesco Gullo, and Giovanni Stilo

    2026

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.