Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Self-serve entity resolution fails predictably in three ways; the fix is a matcher tournament, hard vetoes, and verified merging.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On six public benchmarks, the paper shows no single entity-matching algorithm dominates and recommends an automatic matcher tournament, separate precision/recall mechanisms, and verified-merge clustering to stop cascading false merges.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A genuinely useful practitioner's paper on ER failure modes, strongest on transitive-closure collapse and verified merge; the precision-ablation evidence has internal inconsistencies that need fixing before the lessons can be trusted at face value. the 3 major comments →

arxiv 2607.26298 v1 pith:KG5SG6M2 submitted 2026-07-28 cs.LG cs.AI

Entity Resolution in Practice: Lessons from a Self-Serve Pipeline

Kaushik Pavani , Ganga Aluri , Pravin Jadhav , Neeraj Prasad , Kiran Sanka This is my paper
classification cs.LG cs.AI
keywords entity resolutionrecord linkagematcher tournamenttransitive closureverified mergehard vetoesblockingknowledge distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a self-serve entity resolution pipeline — one that must handle a new dataset without a domain expert tuning every knob — fails in three predictable ways, each absent from the existing ER literature. First, no single matcher architecture wins across datasets, so the pipeline should train several families per dataset and auto-select the winner. Second, precision and recall break at different stages: recall is lost before the matcher runs because blocking misses pairs, while precision fails because sparse records look similar to everything, so the fix is diverse blocking plus hard rule-based vetoes, not a shared threshold. Third, connected-components clustering assumes transitivity, letting one false link chain unrelated records into a mega-cluster; replacing it with a 'verified merge' step that re-scores cross-cluster pairs before merging recovers recall without propagating error. If these lessons hold, ER teams should adopt all three as defaults rather than escalation paths.

Core claim

On six public benchmarks from 864 to 5 million records, each capped at 10K training labels, the authors find that a tournament over three matcher families (DeepMatcher, LightGBM, GAT) never has a single winner: DeepMatcher and LightGBM each win three benchmarks, and GAT wins none. Recall is lost at the blocking stage, so they union an HNSW ensemble with identifier-based indexing; precision is improved by binning candidate pairs by shared populated fields (learned per-bin thresholds with monotonicity) and by hard per-field vetoes that zero out a match when an identifier field disagrees. The most striking result is in clustering: connected components collapses on the two largest datasets (Pair

What carries the argument

The carrying mechanism is a three-part self-serve pipeline: (1) a per-dataset matcher tournament that trains DeepMatcher, LightGBM, and GAT on shared pairs and embeddings and auto-selects the winner; (2) a precision/recall split — recall via an ensemble of HNSW indices plus an exact-match identifier blocking index, precision via sparsity-binned confidence thresholds and hard per-field vetoes that no learned matcher can override; (3) 'verified merge' clustering, which replaces blind transitive closure with center-based assignment followed by direct cross-cluster scoring of representative pairs, blocking any merge on a single veto or sub-threshold score. The verified-merge step carries the cla

Load-bearing premise

The central claim stands on the comparability of the Table V baseline (51.6% purity on Restaurants) to the Table III tournament winner under the same threshold and clustering regime; if the two tables measure different configurations, the veto gains are an artifact.

What would settle it

Re-run the precision ablation on Restaurants and MB 200K starting from the exact global-threshold tournament-winner clusters used in Table III; if cluster purity is already at or near the +Vetoes column before adding vetoes, the reported gains are an artifact. Conversely, if purity drops to the Table V baseline level, the lesson holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Committing to one matcher architecture forfeits at least a third of unseen datasets; the tournament is free because all matchers share the same training pairs and embeddings.
  • Tuning the matcher threshold cannot fix recall — a pair never retrieved by blocking is lost forever; recall must be repaired by diversifying the retriever, not by loosening the matcher.
  • Hard rule-based vetoes raise cluster purity substantially where identifier fields are common (Restaurants +32.8 pp, MB 200K +12 pp purity), because they impose business rules a soft classifier cannot learn from finite data.
  • Connected-components clustering is only safe when matcher precision is very high, and verified merge preserves or improves Pair-F1 on 4/6 datasets while capping the damage on the other two.
  • The teacher-student setup makes the tournament practical: LLM labeling costs about $450 per million pairs, the distilled winner about $12 per million pairs — a 37.5x reduction keeping inference on lightweight models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors leave implicit: verified merge's 'any negative evidence blocks the merge' rule is a stepping stone toward correlation-clustering formulations of ER that explicitly optimize a non-transitive objective, rather than post-hoc patching transitive closure.
  • Because the paper reports the SOP as an inspectable audit trail, a testable follow-up is whether practitioners can debug a bad merge by reading the YAML rules — and whether SOP revisions measurably cut the 'months of dead-end experiments' the authors describe.
  • The authors acknowledge GAT never winning is a product of the 10K-label cap; an experiment that scales labels on the two largest benchmarks could reveal a graph-based matcher overtaking at higher budgets.
  • The precision floor of verified merge (it cannot save MB 200K at pairwise precision ~0.51) suggests combining the merge gate with active LLM adjudication, asking the teacher to re-score the few surviving cross-cluster pairs before committing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper is a practitioner-oriented study of a self-serve entity-resolution pipeline built around a YAML SOP, LLM-conditioned labeling, distillation, and a block-match-cluster architecture. On six public benchmarks ranging from 864 to 5M records, the authors advance three lessons: (L1) no single matcher family wins across datasets, so an automatic tournament among DeepMatcher, LightGBM, and GAT is recommended; (L2) precision and recall need separate mechanisms—hard field vetoes for precision, diverse blocking/identifier indexing for recall; and (L3) transitive closure is dangerous, so verified-merge clustering (center-based assignment plus active cross-cluster verification) is a safer default. The paper includes tournament leaderboards, blocker-recall ablations, precision ablations, clustering ablations, cost/latency figures, and an explicit limitations section.

Significance. If the findings hold, the paper provides transferable engineering recipes for self-serve ER: a per-dataset matcher tournament, a recall/precision split with deterministic vetoes, and verified-merge clustering. The use of six public benchmarks with entity-level held-out splits and the explicit acknowledgement of the 10K-label regime are strengths, as is the authors' candor about single-seed point estimates and the precision floor of verified merge. However, the central quantitative support for Lesson 2 is currently unreliable: Table V is internally inconsistent with Tables III and VI on Restaurants, and the text contradicts Table V on the sparsity mechanism. The qualitative lessons may survive a corrected ablation, but the reported magnitudes—especially the +32.8 pp veto gain on Restaurants—cannot be interpreted as written. This is a load-bearing issue because Lesson 2's recommendation is based on that ablation.

major comments (3)
  1. [§V, Table V; Table III; Table VI] Table V reports Restaurants baseline cluster purity 51.6% for the 'tournament-winning matcher with global threshold'. Table III reports the LGBM winner on Restaurants with purity 1.0, and Table VI reports Pair-F1 = 1.000 and pairwise precision = 1.000 for the same dataset under every clustering regime. Under any standard cluster-purity definition, perfect pairwise precision (or Pair-F1 = 1.0) forces purity of 100%. The 51.6% value therefore implies a different threshold, clustering method, or purity definition, none of which is specified. The same row shows +Sparsity improving purity by +1.5 pp, directly contradicting §V-B(1)'s statement that sparsity-aware thresholds are 'no-op on Restaurants (full fields)'. Because the largest veto gain in Table V (+32.8 pp) is computed against this incomparable baseline, Lesson 2's quantitative support is unreliable as written. Please reconcile the ta
  2. [§V, Table V (baseline definition)] Table V's 'Baseline = tournament-winning matcher with global threshold' is not defined precisely enough to be comparable across rows and across tables. It is not stated which clustering algorithm turns matcher scores into clusters, how the global threshold is chosen, or how 'cluster purity' is computed. Since Table III's tournament winner is selected by Pair-F1 on validation and Table V uses a global threshold, the reader cannot determine whether Table V is the same configuration as the tournament winner evaluated differently or a different configuration. Please make the baseline in each ablation identical to the corresponding tournament winner's actual cluster output, or explicitly document the intended difference.
  3. [§IV, Table III (tournament evidence)] The tournament results are single-seed point estimates with no variance or significance information. This matters for the specific guidance 'always run the tournament': some winning margins are extremely small (NCV: 0.993 vs 0.992, marked 'margin < 0.001'), so the auto-selected winner may be arbitrary on those datasets. The qualitative claim that no architecture dominates may still hold, but the practitioner recommendation should be supported by at least multiple seeds or confidence intervals on close rows, or the manuscript should state clearly that near-tie rows are treated as ties for the lesson.
minor comments (5)
  1. [General (metric definitions)] Purity and cluster purity are never formally defined. Table III reports 'purity in parentheses' and Table V reports 'cluster purity'; please state the exact definition (e.g., majority-label purity over clusters) and whether it differs from pairwise precision.
  2. [Figure 2] In the figure's outcome column, 'R2↔R3 — Merged (correct)' is misleading: in the figure, R2 has true label A and R3 has true label B, so this merge is a false positive. The label should read 'Merged (above threshold)' or similar, since the point is that the matcher accepts the link even though it crosses ground-truth clusters.
  3. [§II; Figure 2 caption] There are two cross-reference typos: 'Appendix 0b' (should presumably be Appendix B) and '§VI-0b' (should be §VI-B). Please fix.
  4. [Table VI] The DBLP-Scholar dataset is abbreviated as 'DBLP' in Table VI but as 'DBLP-Sch.' in other tables. Please use one consistent abbreviation throughout.
  5. [§VI-b] The citation for 'center-based clustering' is [20], which appears to be a Swoosh paper; if so, it is not the canonical source for center-based clustering. Please cite the actual method or describe the procedure explicitly.

Circularity Check

0 steps flagged

No circularity: the paper's claims are empirical inductions from held-out evaluations on external public benchmarks, not derivations from fitted values or self-cited results.

full rationale

The paper makes no claim to derive its lessons from first principles; each lesson is an empirical induction from held-out evaluations on six public benchmarks (Tables III, V, VI). The tournament winner per dataset is the empirical argmax of Pair-F1, so the claim that no matcher dominates is a statement about the observed scores, not an assumption embedded in the method. The precision/recall ablation incrementally adds mechanisms and reports cluster purity; no target quantity is defined in terms of the fitted thresholds or vetoes. Verified-merge is a new clustering algorithm evaluated against standard baselines, not a renamed known result, and the authors explicitly disclaim individual novelty in §IX: 'None of these three mechanisms is individually novel.' All cited works are external; there is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled via citation. The paper's own limitations (§VIII) are explicit and do not conceal a circular dependency. A separate internal-consistency concern exists between Table V and Table VI for Restaurants (51.6% purity vs Pair-F1=1.0, and the text's 'no-op' vs +1.5 pp), but that is a correctness/reproducibility issue, not circularity, and does not affect the circularity score.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims are empirical observations from a fitted pipeline rather than derivations; the free parameters listed are the hand-chosen or learned settings that the reported gains depend on. The axioms are the domain assumptions that transfer the benchmark results to the claimed practitioner scenarios. No new natural entities are posited; the SOP is a configuration artifact, not a postulated entity.

free parameters (4)
  • Sparsity-aware threshold bins θ_b = per dataset, not reported
    Learned per bin of shared-populated-field counts with monotonicity; drives the +8.4 pp and +1.5 pp purity gains in Table V (§V-B-b).
  • Per-field veto floors τ_f = per field per dataset, not reported
    Hand-set floors for identifier fields (Eq. 2); central to the +32.8 pp Restaurants purity gain claimed in §V.
  • Verified-merge representative count k = 3
    Hand-chosen default; controls the O(k²) matcher-call cost and the behavior of the merge gate (§VI-b).
  • Blocking recall settings: k=20 and HNSW ensemble index configurations = k=20; M and ef_search values not enumerated
    Table IV's recall numbers depend on these hand-chosen retrieval settings (§V-B-a).
axioms (4)
  • domain assumption LLM teacher labels are sufficiently accurate to train the student matcher; teacher accuracy is never measured separately.
    The whole pipeline's matcher quality depends on the teacher's labels; the paper reports only final matcher F1, not teacher agreement with ground truth (§II, §III).
  • domain assumption The six public benchmarks and the 10K-label budget represent the real-world self-serve ER workloads the lessons target.
    The Limitations state no production outcomes or live-drift robustness are measured, so the transfer from benchmarks to deployment is assumed (§VIII).
  • domain assumption Entity-level splitting and Pair-F1 adequately capture the error profile of interest (over-splitting vs over-merging).
    Used across all experiments; purity is added to separate the two failure modes but Pair-F1 remains the primary metric (§III).
  • ad hoc to paper In verified merge, a single cross-cluster pair triggering a veto correctly blocks the merge; no weighting by matcher confidence is used.
    This asymmetry is a design choice (§VI-b) that could over-block when the representative pair is noisy; the paper does not test alternative aggregation rules.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Entity Resolution in Practice: Lessons from a Self-Serve Pipeline." pith.science (2026). https://pith.science/paper/KG5SG6M2

@misc{pith2026260726298,
  author       = {Pith},
  title        = {Pith review of: Entity Resolution in Practice: Lessons from a Self-Serve Pipeline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KG5SG6M2}},
  note         = {Machine review of arXiv:2607.26298}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We built and evaluated a self-serve entity resolution (ER) system on six benchmarks spanning 864 to 5M records, and three lessons emerged that are absent from existing ER literature. (1) No single matching algorithm wins everywhere - a self-serve pipeline cannot predict its next dataset, so we recommend training several algorithm families per dataset and letting an automatic bake-off pick the winner. (2) Precision and recall need separate fixes, not a shared threshold - precision needs hard rule-based vetoes, recall needs more diverse candidate retrieval. (3) One false-positive link can silently merge unrelated entities - assuming "A matches B" and "B matches C" implies "A matches C" lets a single bad link chain hundreds of records together, so every cross-group merge must be actively re-verified. We hope these lessons save practitioners the months of dead-end experiments that led us to them.

Figures

Figures reproduced from arXiv: 2607.26298 by Ganga Aluri, Kaushik Pavani, Kiran Sanka, Neeraj Prasad, Pravin Jadhav.

Figure 1
Figure 1. Figure 1: A preview of a typical failure mode in ER systems, and one of the lessons we discuss in this paper. Consider the task of deduplicating restaurant records so that one cluster represents one physical location (a single Sakura Sushi at 42 Oak St in Portland) rather than a brand across cities. Five similarly named records arrive (R1–R5). Most pairwise matchers will accept both R1↔R3 and R3↔R4: the sparse bridg… view at source ↗
Figure 2
Figure 2. Figure 2: Transitive closure failure on a 3-record subset of Figure 1. The sparse [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Full ER pipeline. LLM-driven stages (yellow, orange) involve humans; cost-sensitive matching/clustering (teal) run on lightweight distilled models. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: SOP excerpt for restaurant matching [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 5 linked inside Pith

  1. [1]

    An overview of end-to-end entity resolution,

    V . Christophides, V . Efthymiou, T. Palpanas, G. Papadakis, and K. Ste- fanidis, “An overview of end-to-end entity resolution,”ACM Computing Surveys, vol. 54, no. 6, pp. 1–42, 2021

    2021

  2. [2]

    A theory for record linkage,

    I. P. Fellegi and A. B. Sunter, “A theory for record linkage,”J. American Statistical Association, vol. 64, no. 328, pp. 1183–1210, 1969

    1969

  3. [3]

    Entity matching using large language models,

    R. Peeters and C. Bizer, “Entity matching using large language models,” inProc. EDBT, 2025, pp. 338–350

    2025

  4. [4]

    Cost-effective in-context learning for entity resolution,

    Y . Fan, J. Li, S. Liu, and T. Rekatsinas, “Cost-effective in-context learning for entity resolution,” inProc. ICDE, 2024

    2024

  5. [5]

    Deep entity matching with pre-trained language models,

    Y . Li, J. Li, Y . Suhara, A. Doan, and W.-C. Tan, “Deep entity matching with pre-trained language models,”PVLDB, vol. 14, no. 1, pp. 50–60, 2020

    2020

  6. [6]

    Deep learning for entity matching: A design space exploration,

    S. Mudgalet al., “Deep learning for entity matching: A design space exploration,” inProc. SIGMOD, 2018, pp. 19–34

    2018

  7. [7]

    Blocking and filtering techniques for entity resolution: A survey,

    G. Papadakis, D. Skoutas, E. Thanos, and T. Palpanas, “Blocking and filtering techniques for entity resolution: A survey,”ACM Computing Surveys, vol. 53, no. 2, pp. 1–42, 2020

    2020

  8. [8]

    Optuna: A next-generation hyperparameter optimization framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” pp. 2623–2631, 2019

    2019

  9. [9]

    autoresearch: LLM-driven autonomous research loop,

    A. Karpathy, “autoresearch: LLM-driven autonomous research loop,” https://github.com/karpathy/autoresearch, 2026

    2026

  10. [10]

    LightGBM: A highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “LightGBM: A highly efficient gradient boosting decision tree,” in NeurIPS, 2017, pp. 3146–3154

    2017

  11. [11]

    How attentive are graph attention networks?

    S. Brody, U. Alon, and E. Yahav, “How attentive are graph attention networks?” inICLR, 2022

    2022

  12. [12]

    Scalable matching and clustering of entities with FAMER,

    A. Saeedi, E. Peukert, and E. Rahm, “Scalable matching and clustering of entities with FAMER,”Complex Systems Informatics and Modeling Quarterly (CSIMQ), no. 16, pp. 61–83, 2018

    2018

  13. [13]

    HPI repeatability datasets for entity resolution: Restaurants, Cora, and DBLP-Scholar,

    F. Naumann and M. Herschel, “HPI repeatability datasets for entity resolution: Restaurants, Cora, and DBLP-Scholar,” https://hpi.de/naumann/projects/repeatability/datasets.html, 2010

    2010

  14. [14]

    Using link discovery to enrich DBpedia with equivalent entity relationships,

    A. Saeedi, E. Peukert, and E. Rahm, “Using link discovery to enrich DBpedia with equivalent entity relationships,” inProc. ADBIS, 2017

    2017

  15. [15]

    Comparative evaluation of distributed clustering approaches for entity resolution,

    ——, “Comparative evaluation of distributed clustering approaches for entity resolution,” inProc. EDBT, 2018, pp. 181–192

    2018

  16. [16]

    An overview of end-to-end entity resolution for big data,

    V . Christophides, V . Efthymiou, T. Palpanas, G. Papadakis, and K. Ste- fanidis, “An overview of end-to-end entity resolution for big data,”ACM Computing Surveys, vol. 53, no. 6, pp. 127:1–127:42, 2021

    2021

  17. [17]

    (almost) all of entity resolution,

    R. C. Steorts, R. Hall, and S. E. Fienberg, “(almost) all of entity resolution,”Science Advances, vol. 8, no. 12, 2022

    2022

  18. [18]

    Exploiting transitivity con- straints for entity matching in knowledge graphs,

    D. Baas, M. Dastani, and A. Feelders, “Exploiting transitivity con- straints for entity matching in knowledge graphs,” inarXiv preprint arXiv:2104.12589, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    TransClean: Finding false positives in multi-source entity matching under real-world conditions via transitive consistency,

    Anonymous, “TransClean: Finding false positives in multi-source entity matching under real-world conditions via transitive consistency,”arXiv preprint arXiv:2506.04006, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Swoosh: A generic approach to entity resolution,

    I. Hassanzadeh, M. A. Saeed, and A. Khodaei, “Swoosh: A generic approach to entity resolution,” inVLDB, 2009

    2009

  21. [21]

    Magellan: Toward building entity matching management systems,

    P. Kondaet al., “Magellan: Toward building entity matching management systems,”PVLDB, vol. 9, no. 12, pp. 1197–1208, 2016

    2016

  22. [22]

    Analyzing how BERT performs entity matching,

    M. Paganelli, D. Tiano, and F. Guerra, “Analyzing how BERT performs entity matching,”VLDB Journal, vol. 33, pp. 1–25, 2024

    2024

  23. [23]

    ZeroER: Entity resolution using zero labeled examples,

    R. Wu, S. Chaba, S. Sawlani, X. Chu, and S. Thirumuruganathan, “ZeroER: Entity resolution using zero labeled examples,” inProc. SIGMOD, 2020, pp. 1149–1164

    2020

  24. [24]

    Heterogeneity in entity matching,

    S. Thirumuruganathanet al., “Heterogeneity in entity matching,” arXiv:2508.08076, 2025

    arXiv 2025

  25. [25]

    Entity resolution with hierarchical graph attention networks,

    D. Yao, Y . Gu, G. Cong, H. Jin, and X. Lv, “Entity resolution with hierarchical graph attention networks,” inProc. SIGMOD, 2022, pp. 429–442

    2022

  26. [26]

    GraphER: When GDD meets GNN for entity resolution on property graphs,

    J. Hu, M. Bewong, S. Kwashie, Y . Zhang, V . Nofong, J. Wondoh, and Z. Feng, “GraphER: When GDD meets GNN for entity resolution on property graphs,”Information Systems, vol. 132, p. 102517, 2025

    2025

  27. [27]

    Graph metrics-driven record cluster repair meets LLM-based active learning,

    A. Saeedi, M. Hosseinzadeh, and E. Rahm, “Graph metrics-driven record cluster repair meets LLM-based active learning,”ACM JDIQ, vol. 17, no. 2, pp. 1–28, 2025

    2025

  28. [28]

    Match, compare, or select? an investigation of LLMs for entity matching,

    T. Wang, Y . Zhang, and S. Roth, “Match, compare, or select? an investigation of LLMs for entity matching,” inProc. COLING, 2025, pp. 89–110

    2025

  29. [29]

    Learning from natural language explanations for generalizable entity matching,

    S. Wadhwa, L. Hawkins, C. Agrawal, B. C. Wallace, and A. Agrawal, “Learning from natural language explanations for generalizable entity matching,”arXiv:2406.09330, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Fine-tuning large language models for entity matching,

    M. Steiner, R. Peeters, and C. Bizer, “Fine-tuning large language models for entity matching,”arXiv:2409.08185, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Agent-S: LLM agentic workflow to automate standard operating procedures,

    W. Liet al., “Agent-S: LLM agentic workflow to automate standard operating procedures,”arXiv:2503.15520, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    SOP-Bench: Complex industrial SOPs for evaluating LLM agents,

    M. Yinet al., “SOP-Bench: Complex industrial SOPs for evaluating LLM agents,”arXiv:2506.08119, 2025. APPENDIX Figure 3 shows the full pipeline:onboarding(SOP con- struction, blocking, LLM labeling—all human-in-the-loop, top row),training(blocker and matcher distillation plus tournament, top row right), andinference(blocking, matching, clustering, audit—bo...

    arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.