{"id":"8b77c30c-1397-4fc2-93c4-b1a02664a89e","arxiv_id":"2502.10453","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-based entity linking pipeline maps cryptoasset attribution tags to knowledge graph actors and outperforms baselines on three datasets.","lead":"This paper builds a pipeline that uses large language models to match cryptocurrency attribution tags, such as exchange names, to standardized entities in a knowledge graph. The authors find that the approach beats established baselines by up to 37 percentage points in F1-score, and that local models come close to remote ones when the prompt is chosen well.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The largest reported F1 gains rest on 67+32 author-annotated links with no inter-annotator reliability check; a few label errors could change the cross-dataset ranking.","rationale":"The reader's weakest assumption is the same one I would stress: the external test sets are tiny and their labels are unvalidated manual annotations, so the strongest evidence of generalization is built on uncertain ground truth. I checked the F1 tables and confirmed that the largest relative gains occur exactly on those two small datasets, making label noise load-bearing. The '37.4%' discrepancy is a secondary red flag that the headline is not tightly connected to the reported numbers, but the qualitative claim could survive a corrected summary; label validity cannot be repaired by rewording. The proposed second-annotator test with unanimous-label re-evaluation would settle the concern. If the labels survive, the method is a credible engineering contribution and the CONDITIONAL verdict is appropriate; if not, the external dataset results should be downgraded or removed.","tokens_in":16786,"tokens_out":8314,"duration_ms":93296,"concrete_test":"Have a second annotator, blinded to the authors' labels and to model predictions, annotate all 126 WatchYourBack and 100 DeFi Rekt records against the same GraphSense actor list following a written mapping protocol; compute Cohen's kappa. Then recompute GPT-4o and Mistral 7B-Instruct F1 on the subset of records with unanimous labels. If kappa < 0.8 or either F1 drops by more than about 0.05, the cross-dataset claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the method generalizes 'across all three datasets' depends on ground-truth actor links. GraphSense links are curated, but the WatchYourBack (126 records, 67 linked) and DeFi Rekt (100 records, 32 linked) labels in Sections 3.2 and 3.3 are manual annotations by the authors, with no annotation protocol, no inter-annotator agreement, and no independent verification. These two datasets are exactly where the largest F1 gaps appear (GPT-4o vs BM253: 0.801 vs 0.495 and 0.793 vs 0.393). With only 32 positive examples in DeFi Rekt, a single mislabeled record changes recall by about 3 percentage points, and macro-F1 is at least as sensitive; a handful of noisy labels could materially reduce the reported 79-80% F1 or change the relative ordering of models. A secondary issue is that the abstract's 'up to 37.4% improvement' is not traceable to any pairwise relative F1 difference in Table 6, which suggests the quantitative summary is not tightly anchored to the experimental record. The load-bearing premise is therefore the correctness of the manual labels; currently no evidence supports that premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage LLM-based entity linking pipeline for cryptoasset attribution tags: a candidate set generator (BM253 blocking with optional related-concept filtering) and an LLM-based candidate selector. The pipeline is evaluated on three datasets—GraphSense TagPacks, WatchYourBack, and DeFi Rekt—against BM253-threshold, UnicornPlus, and fine-tuned UnicornPlus baselines. The authors report that their approach outperforms baselines by up to 37.4% in F1, achieves 93% recall at candidate set size 5 without labeled data, and that local LLMs (e.g., Mistral 7B-Instruct) are comparable to remote models (GPT-4o). They also provide a cost-performance analysis of prompt templates and models.","tokens_in":16997,"tokens_out":5184,"duration_ms":60236,"significance":"If the results hold, the paper addresses a practical need in cryptoasset forensics: inconsistent attribution tags can be linked to a shared knowledge graph automatically. The experimental design covers multiple datasets, several LLMs, and a clear comparison to established baselines. The authors also make code and data publicly available, which supports reproducibility. The main uncertainty is the reliability of the manual ground-truth labels for the two smaller datasets, and some abstract-level claims that do not exactly match the quantitative results in the body. These issues are fixable and do not invalidate the core approach, but they are load-bearing for the paper's strongest claims.","major_comments":[{"comment":"The ground-truth labels for WatchYourBack (126 records, 67 positive actor links) and DeFi Rekt (100 records, 32 positive links) are manual annotations by the authors. The paper does not provide an annotation protocol, inter-annotator agreement, or any independent verification of these labels. With only 32 positive examples in DeFi Rekt, a single mislabeled record changes recall by roughly 3 percentage points, and macro-F1 averaged over actors is similarly sensitive. Since the largest reported F1 gains over baselines (e.g., GPT-4o 0.793 vs. BM253 0.393 on DeFi Rekt) rest on these labels, the paper should supply a detailed annotation guideline, a second annotator, agreement metrics (e.g., Cohen's kappa), and a sensitivity analysis showing how F1 changes if a small fraction of positive labels is removed or corrected.","section":"Sections 3.2 and 3.3, Table 6"},{"comment":"The abstract states that the approach outperforms baselines 'by up to 37.4% in F1-score' across the three datasets. This number is not traceable to any pairwise comparison in Table 6 or Table 7. For instance, comparing GPT-4o to BM253 yields absolute percentage-point improvements of 13.5 (GraphSense), 30.6 (WatchYourBack), and 40.0 (DeFi Rekt); relative improvements are 18.8%, 61.8%, and 101.8%, respectively. The paper should specify exactly which models are compared and whether the improvement is relative or absolute, and correct the number if it does not match any valid comparison.","section":"Abstract and Section 5.4"},{"comment":"The abstract and Introduction claim that 'local LLM models can achieve F1-scores of 90%, comparable to remote models which achieve 94%.' These figures come from Experiment 2 (Table 4), which evaluates candidate selection on the validation set where the correct actor is known to be within the candidate set. In the end-to-end Experiment 3 (Table 6), the best local model (Mistral 7B-Instruct) achieves 0.821 F1 on GraphSense vs. 0.853 for GPT-4o, and the gap is larger on WatchYourBack (0.692 vs. 0.801) and DeFi Rekt (0.547 vs. 0.793). The abstract and introduction should clearly distinguish the sub-task result from the end-to-end result.","section":"Abstract, Introduction, and Experiments 2 vs. 3"},{"comment":"The BM253 baseline threshold is optimized on the GraphSense TagPack validation set and then applied unchanged to WatchYourBack and DeFi Rekt. If the score distributions differ across datasets, this choice could understate the baseline's performance and inflate the reported improvement of the LLM approach. Please tune the threshold per dataset using a small validation split, or report results across a range of thresholds, to ensure a fair comparison.","section":"Section 5.4"}],"minor_comments":[{"comment":"The method is called 'BM253' in the text and tables (e.g., Table 3), but the equation in Section 4.1 is labeled 'BM25' and the rank_bm25 library is referenced. This naming inconsistency should be fixed for clarity.","section":"Throughout"},{"comment":"The claim of 'recall of 93% without the need for labeled data' refers specifically to the GraphSense TagPack validation set with related-concept filtering and k=5. Since the other two datasets do not have category information, the claim should be qualified as applying to the GraphSense dataset only.","section":"Abstract and Section 5.2"},{"comment":"There are several typos: 'blockhain' in Section 3.1, 'detailled' in Section 5.1, and 'Aditionally' in Section 5.3. These should be corrected.","section":"Section 3.1 and Section 5.1"},{"comment":"Reference [3] is a change.org petition rather than a peer-reviewed source. Replacing it with a peer-reviewed study on the reliability of cryptoasset tracing tools would strengthen the motivation.","section":"Section 2.2, reference [3]"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a practical and reproducible entity linking method for cryptoasset attribution tags. The core algorithmic contribution is modest but the evaluation is reasonably systematic. The main risk is the unvalidated manual ground truth for the two smaller datasets, which directly supports the cross-dataset generalization claim. If the authors can provide annotation reliability evidence and reconcile the numerical claims (especially the 37.4% figure), the paper would be suitable for publication. I would not recommend rejection; the issues are fixable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful applied paper. The new thing here is not the LLM entity-linking technique—that's established—but the application to cryptoasset attribution tags, with taxonomy-aware filtering and a practical cost analysis. The pipeline (BM253 blocking + related-concept filtering to k=5, then an LLM selector) is sensible, the evaluation covers three datasets, and the code and data are public. The end-to-end comparison against BM253 and UnicornPlus is the strongest part.\n\nThe numbers in the abstract are actually defensible: the 37.4% F1 improvement is the absolute percentage-point gap between GPT-4o (0.793) and UnicornPlusFT (0.419) on DeFi Rekt, and the 90% vs 94% local/remote comparison comes from the candidate-selection experiment on the validation set. It would be clearer to say 'percentage points' in the abstract, but it's not an overclaim.\n\nThe real soft spot is the two small external test sets. WatchYourBack has 67 manually annotated actor links, DeFi Rekt has 32, and there is no inter-annotator agreement check. No statistical significance is reported either. These are genuine limitations that should be fixed with bootstrapped confidence intervals and either a second annotator or a released annotation protocol. But the gaps over baselines are large (0.30–0.37 F1), so a handful of label errors would not flip the ranking. This is a looseness in the evaluation, not a fatal flaw.\n\nOne minor gap: the candidate-set recall of 93% at k=5 is measured on the GraphSense validation set. That's fine for component tuning, but it would help to see the end-to-end recall on the external datasets, since those have no taxonomy filtering. Also, the manual annotation process is described in one sentence; it deserves more detail.\n\nOverall, this is a competent, reproducible study that will be useful to people building crypto-forensic tools and to anyone applying LLMs to entity linking in low-resource domains. It won't change the field, but it earns its place. I'd send it to peer review with a request for the annotation-reliability work and confidence intervals.","headline":"Solid, reproducible LLM entity linking for crypto forensics; the reported gains are real, but the small manually annotated test sets need reliability work.","tokens_in":17529,"tokens_out":4277,"would_cite":false,"duration_ms":34891,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM pipeline links messy cryptoasset tags to knowledge-graph actors, beating baselines by up to 37.4% in F1.","keywords":["cryptoasset forensics","attribution tags","entity linking","knowledge graph","large language models","BM25 blocking","record linkage","data quality"],"falsifier":"Re-label the WatchYourBack (126 records, 67 with actor links) and DeFi Rekt (100 records, 32 with actor links) samples with two or more independent annotators who do not see the paper's labels, and measure inter-annotator agreement; low agreement or materially different labels would show the 37.4% F1 gain is an artifact of the original annotation.","tokens_in":16559,"feed_emoji":"🔗","tokens_out":7661,"duration_ms":56771,"temperature":0.7,"pith_summary":"Cryptoasset forensics depends on attribution tags, but tags are inconsistent: btc-e and btc-e.com denote the same exchange, yet tracing tools can read them as separate actors. This paper tries to fix that by linking tags to entities in a knowledge graph, and proposes an LLM-based pipeline that does the linking without domain-specific fine-tuning. Its central claim is that the pipeline beats existing baselines on all three datasets, with up to a 37.4% improvement in F1-score. If that holds, forensic investigations and machine-learning models trained on tags would operate on cleaner, harmonized data, reducing the risk that bad labels derail an investigation or a conviction.","feed_headline":"LLMs link crypto tags to real actors with up to 37.4% better F1","feed_subtitle":"BM25 blocking narrows candidates; an LLM picks the match, harmonizing btc-e and btc-e.com across three forensic datasets.","key_machinery":"The central mechanism is a two-module entity linking pipeline. The candidate set generator first applies related-concept filtering, which uses the taxonomy hierarchy to exclude actors whose category is unrelated to the tag's category, then BM25 blocking over trigram-tokenized strings to keep only the k=5 most similar actor labels, reducing comparison cost from O(nm) to O(nk). The candidate selector then feeds each tag and its candidates to an LLM prompt that asks for the matching entity by index or 'none', optionally with few-shot examples. The prompt template is load-bearing: zero-shot performance swings from near zero to over 90% F1 depending on template structure, so the output-format instructions matter as much as the model.","core_discovery":"On its own terms, this paper establishes that a two-stage pipeline, a cheap candidate generator followed by an LLM candidate selector, maps attribution tags to knowledge-graph actors more reliably than existing approaches. The generator combines related-concept filtering with BM25 trigram blocking to shrink each tag's candidate pool to five entities, reaching 93% recall without any labeled data. The selector then uses an LLM prompt to pick the matching actor or say 'none'; with GPT-4o it reaches 94% F1 in isolation and 79 to 85% F1 end-to-end across the three datasets, while the local Mistral 7B-Instruct reaches 90% F1 in isolation. Against the BM25-threshold and the mixture-of-experts baselines, the reported end-to-end improvements are up to 37.4% in F1.","pith_inferences":["The error analysis suggests a practical ranking: wrong-entity links are rarer but more dangerous than missed links for GPT-4o, so a deployment could treat 'none' outputs as needing human review rather than as failures.","A natural extension is to use the pipeline to flag mislabeled tags in existing databases, since a confident mismatch between a tag and its current actor link is itself a data-quality signal.","The related-concept filter only helps when a shared taxonomy exists; if the crypto forensics field adopts the INTERPOL/DWVA taxonomy, this component becomes portable to other datasets and jurisdictions.","Comparing the LLM selector against a simple string-similarity ranker on the same candidate sets would isolate whether the model adds semantic understanding beyond the blocker."],"forward_implications":["Tracing tools can automatically harmonize tags from different sources against a shared knowledge graph, so btc-e and btc-e.com resolve to the same actor.","Candidate sets of five with 93% recall mean the approach works without labeled training data, which is rare in forensic settings.","A local model reaches 90% F1, so the pipeline can run on consumer hardware without sending sensitive investigative data to a remote API.","Choosing the cheapest prompt template cuts API costs by about 90% for roughly a 1% F1 drop, making large-scale tag cleaning affordable.","The same end-to-end setup exceeds 79% F1 on all three datasets, suggesting the method generalizes beyond the dataset it was tuned on."],"supporting_citations":[{"why":"Supplies the WatchYourBack dataset and the earlier edit-distance tag harmonization approach this work builds on.","marker":"[13]"},{"why":"Supplies the mixture-of-experts baseline model that the end-to-end approach must beat.","marker":"[38]"},{"why":"Motivates the simple tf-idf-style blocking strategy used in candidate generation without labeled data.","marker":"[32]"},{"why":"Guides the LLM prompt-template design and the convention of counting invalid responses as no match.","marker":"[33]"},{"why":"Provides two local language models evaluated as candidate selectors.","marker":"[43]"},{"why":"Supplies the DeFi Rekt event database used as a third evaluation dataset.","marker":"[6]"},{"why":"Defines the BM25 scoring function used by the blocker.","marker":"[35]"},{"why":"Provides the blocking and filtering framework that organizes the candidate set generator.","marker":"[29]"}],"fun_headline_variants":["LLM method links crypto tags to entities with 37% higher F1","Crypto tag linking: LLM beats baselines, no labels needed","Local LLMs hit 90% F1 on crypto tag matching, rivaling remote","Cut crypto tag linking cost by 90% with only 1% F1 loss","No-label crypto tag recall hits 93% with LLM candidate filtering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the ground-truth actor links are correct, including the manually annotated labels for WatchYourBack and DeFi Rekt; if those labels are noisy, the reported F1 improvements are not measuring what they claim.","fun_headline_variants_meta":{"raw":{"variants":["LLM method links crypto tags to entities with 37% higher F1","Crypto tag linking: LLM beats baselines, no labels needed","Local LLMs hit 90% F1 on crypto tag matching, rivaling remote","Cut crypto tag linking cost by 90% with only 1% F1 loss","No-label crypto tag recall hits 93% with LLM candidate filtering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3153,"prompt_tokens":926,"completion_tokens":2227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2124}},"tokens_in":542,"tokens_out":2227,"duration_ms":13647,"temperature":1.0,"reasoning_tokens":2124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:00:09.207977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the WatchYourBack (126 records, 67 with actor links) and DeFi Rekt (100 records, 32 with actor links) samples with two or more independent annotators who do not see the paper's labels, and measure inter-annotator agreement; low agreement or materially different labels would show the 37.4% F1 gain is an artifact of the original annotation.","supporting_citations":[{"cited_title":"https://de.fi/rekt-database (2024), https://de.fi/rekt-database","cited_arxiv_id":null,"evidence_quote":"Supplies the DeFi Rekt event database used as a third evaluation dataset."}],"review_version":1}