{"id":"a8aa6dea-9591-45ab-8966-33b281228a53","arxiv_id":"2510.08948","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A deployed LLM risk-investigation system that augments retrieval and reflection with a domain knowledge base achieves 0.92 factual alignment and 82% expert acceptance at JD.com.","lead":"SHERLOCK combines a domain knowledge base, LLM fine-tuning, and a reflect-and-refine module to help risk analysts investigate e-commerce fraud cases, reporting an 82% expert acceptance rate and roughly 4x faster case review in a live JD.com A/B test. The paper also claims a data flywheel that recovers performance as fraud tactics shift, though that 90-day evidence appears only in the abstract.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract promises a 90-day flywheel evaluation (two decay/recovery cycles, +3.5% EAR ceiling) but the body provides only a 5-week A/B test and a qualitative §3.6; the core dynamic-adaptation claim is unsupported.","rationale":"The paper positions SHERLOCK as a dynamic, evolving system; the data flywheel is one of its three core contributions and directly supports the title's 'Dynamic Knowledge Adaptation.' The abstract promises a 90-day evaluation with two decay/recovery cycles and a 3.5% EAR ceiling gain, but the body does not report this evaluation. §3.5 describes a 5-week A/B test, §3.6 is qualitative, and Appendix B Table 6 contains only aggregate control/treatment numbers. Thus the flywheel claim is currently unfalsifiable from the manuscript. This is more load-bearing than the reader's stated weakest assumption (offline benchmark independence) because even if the offline benchmark is clean, the system's advertised ability to adapt to shifting fraud tactics — the central claim — is unsupported. The '387%' inconsistency (15.24→3.13 minutes is a 79.5% reduction, not a 387% reduction) underscores that online metrics are not carefully reported. I recommend keeping CONDITIONAL: the paper should be accepted only if the authors release the 90-day evidence or explicitly restrict claims to the 5-week A/B test. Since the reader already set CONDITIONAL, my concern does not move the verdict; hence UNCHANGED.","tokens_in":14026,"tokens_out":4648,"duration_ms":38584,"concrete_test":"Request the 90-day flywheel evaluation artifact: a time-indexed series of expert acceptance rate (or EAR ceiling) spanning 90 days, with the two tactic-shift decay events and recovery points clearly marked. If the authors cannot produce this series, or if it does not exhibit two recoveries and a ~3.5% ceiling gain, the core dynamic-adaptation claim should be explicitly downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that SHERLOCK's data flywheel enables dynamic knowledge adaptation to shifting fraud tactics — rests on the abstract's assertion of 'an additional 90-day evaluation' showing two performance-decay recoveries and a ~3.5% EAR ceiling gain through autonomous model updates. However, the full manuscript contains no such evaluation. §3.5 describes only a 5-week online A/B test; Table 6 in Appendix B reports just control vs treatment average review time (15.24 vs 3.13 minutes) and acceptance rate (0.31 vs 0.82). §3.6 is purely qualitative, with no time-series, no decay/recovery events, no second cycle, and no 3.5% figure. The 90-day result is the only evidence that the flywheel actually closes the loop; without it, the paper's title claim of 'Dynamic Knowledge Adaptation' is unverified. This is not a matter of interpretation — it is a missing reported experiment that the abstract explicitly cites. Relatedly, the '387%' throughput figure is arithmetically misstated: 15.24→3.13 minutes is a 79.5% time reduction (roughly a 4.87× speed-up), and 'reducing investigation times by 387%' is impossible. That inconsistency compounds the concern that the online metrics are not carefully reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes SHERLOCK, a deployed LLM-based framework for e-commerce risk investigation at JD.com. It builds a domain knowledge base (KB) from multi-modal sources, uses a two-stage retrieval-augmented 'Reflect & Refine' (R&R) module, and couples both with a data flywheel that routes rejected or uncertain LLM outputs to expert annotation and fine-tuning. The authors report offline benchmark results (FAR=0.92, SNR=4.34, CDR=0.64 for the full system), an expert preference study, and a 5-week online A/B test in which expert acceptance rose to 0.82 and average review time fell from 15.24 to 3.13 minutes. The abstract additionally claims a 90-day flywheel evaluation with two performance-decay recoveries and a ~3.5% EAR ceiling gain; the body does not report such an evaluation.","tokens_in":14427,"tokens_out":5002,"duration_ms":38654,"significance":"If the reported numbers are sound, the paper demonstrates an unusually complete integration of knowledge-base management, retrieval-augmented LLM reasoning, and human-in-the-loop continuous improvement in a production risk-control setting. The use of expert acceptance in live operations and the explicit 'suspect-then-rule-out' CoT synthesis are practical assets. However, the central claim of dynamic knowledge adaptation rests on the missing 90-day experiment, and the offline/online evidence as presented does not yet meet the standard needed to support that claim.","major_comments":[{"comment":"The abstract states that 'an additional 90-day evaluation shows that the flywheel successfully recovers from performance decay caused by changing tactics twice, raising the EAR ceiling by around 3.5%.' No such evaluation appears in the manuscript. §3.5 describes a 5-week A/B test; Table 6 reports only aggregate review time and acceptance rate; §3.6 is a qualitative description of feedback logging. There is no time series, no decay/recovery event, and no second cycle. This is the only evidence that would substantiate the title claim of 'Dynamic Knowledge Adaptation.' The authors should either include the protocol and results or remove the claim.","section":"Abstract / §3.5–3.6"},{"comment":"'387% faster' and 'reducing case investigation times by 387%' are arithmetically invalid. Average review time going from 15.24 to 3.13 minutes is a 79.5% reduction (or a 4.87× speed-up), not a 386.7% decrease. Since a reduction cannot exceed 100%, this misstatement appears in the abstract, §3.5, and Conclusion. Correct the metric and describe the basis for any throughput percentage.","section":"§3.5 / Conclusion"},{"comment":"No statistical or methodological confidence is reported. The 1,000-sample evaluation benchmark is said to be 'manually verified by domain experts,' but there is no statement that these samples are disjoint from the 5,000-order dataset or from the DPO/SFT training data produced by the data flywheel (§2.3). Without a disjoint split, offline FAR/SNR/CDR can be optimistically biased. Similarly, the 5-week A/B test reports point estimates only; no confidence intervals, p-values, or bootstraps are given. Error bars in Tables 2–4 and a significance test for the A/B metrics are needed.","section":"§3.1.4 / §3.5"},{"comment":"The automated judge, DeepSeek-R1, is itself an LLM and is also used as a zero-shot baseline in Table 2. Its agreement with expert labels is never measured (e.g., Cohen's κ or a held-out human-scored subset). If the judge systematically favors the fine-tuned/refined outputs, the reported FAR/SNR improvements are inflated. Please provide judge validation against human annotations.","section":"§3.1.4"},{"comment":"The text says 'Reflecting with an empty KB provides a minor boost,' but Table 4 shows the w/o Knowledge Base configuration with FAR=0.80, SNR=1.98, and CDR=0.59 — all below the single-pass w/o Reflection row (0.83, 2.98, 0.67). If 'w/o Knowledge Base' still includes reflection, the table indicates a substantial drop, not a minor boost. The discrepancy must be resolved; as written, the ablation does not support the stated conclusion about the intrinsic value of self-correction.","section":"§3.3.2 / Table 4"}],"minor_comments":[{"comment":"The notation S(d_t,e_t)∈Z∩[1,5] is awkward; a simpler integer range [1,5] would avoid confusion.","section":"§2.2.1"},{"comment":"Typos and inconsistent formatting: 'Given the the diversity' in §2.2, 'Tabluar' in Figures 1/3, 'retreive' in Figure 1, and inconsistent hyphenation of 'hotfix'.","section":"Global"},{"comment":"Figure 4 is described as a 'visual illustration of R&R module' but is largely a block diagram; a concrete example with an input case, retrieved KB entry, and refined output would improve clarity.","section":"§2.4 / Figure 4"},{"comment":"DeepSeek-R1 serves both as a baseline and as the automated judge; this dual role is not acknowledged in the main text and should be discussed explicitly.","section":"§3.1.3 / §3.1.4"},{"comment":"The CDR definition uses N_gt_core, but the paper does not state how cases with N_gt_core=0 are handled (excluded? counted as 0?). This affects the interpretation of CDR.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is an industry-systems paper with a promising deployment story, but the gap between the abstract's 90-day claim and the body is serious enough that the authors should be asked to provide the missing experiment or revise the claim. The arithmetic error and ablation contradiction also suggest the manuscript needs a careful pass by the authors before it is suitable for archival publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a genuine production system with respectable engineering, but the headline adaptation claim is not where the paper says it is. The abstract promises a 90-day evaluation showing two decay/recovery cycles and a ~3.5% EAR ceiling gain; the body gives a 5-week A/B test and a qualitative description of the flywheel. Nothing in Section 3.6 or the appendix tracks performance over time. If you're reading for the dynamic-knowledge-adaptation promise, you'll finish unsatisfied.\n\nWhat's new: it appears to be the first published deployment of an LLM risk-investigation pipeline that combines a domain KB, RAG, CoT fine-tuning, and a post-hoc reflect-and-refine module. The multi-modal knowledge distillation from docs, meeting transcripts, and code repositories is a real practical contribution. Offline numbers are solid: FAR 0.92, SNR 4.34, and human experts prefer SHERLOCK 71% of the time. I believe the system does what the offline tables say.\n\nSoft spots, in proportion:\n\n1. The flywheel evidence is missing. That is the load-bearing claim for 'dynamic knowledge adaptation,' and it's unsupported. Section 3.6 is process description, not evaluation.\n\n2. Reporting carelessness. The ablation text claims an empty KB provides a minor boost, but Table 4 shows w/o Knowledge Base is worse than no reflection on every metric — a direct internal contradiction. Also, 'reducing investigation times by 387%' is arithmetically wrong; the speed-up is 4.87x, i.e., 387% faster, not a 387% reduction.\n\n3. Evaluation hygiene. No error bars or significance tests on any online or offline metric. The offline judge is DeepSeek-R1, an LLM, not validated against expert labels. There is no statement that the 1,000 evaluation cases were disjoint from the flywheel's training data. The 82% acceptance rate comes from a single 5-week A/B with no significance testing. These are not fatal for an industrial paper, but they cap the strength of the claims.\n\n4. No code or data released, so the 'first production deployment' claim is verifiable only by JD insiders. The detail level (prompts, KB pipeline, online metrics) makes it plausible, but independent reproduction is impossible.\n\nNet: worth sending to peer review, because a real deployment with real operational metrics deserves referee time, and the flywheel gap is fixable in revision. I would not cite the adaptation claim as established until the 90-day data appears.","headline":"A real deployment with credible offline gains, but the abstract promises a 90-day flywheel evaluation that the paper never reports — the core 'dynamic adaptation' claim is unsupported as written.","tokens_in":14861,"tokens_out":2450,"would_cite":true,"duration_ms":22634,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHERLOCK claims that an LLM with a live-updating knowledge base and a self-improving data loop can run e-commerce risk investigations at near-expert quality—82% of its verdicts accepted by human experts and review times cut from ~15 to ~3 m","keywords":["e-commerce risk management","large language models","retrieval-augmented generation","knowledge base","data flywheel","chain-of-thought","fraud investigation","expert-in-the-loop"],"falsifier":"Have a second, independent panel of experts re-annotate the same 1,000 benchmark cases and re-score the LLM outputs without seeing the automated judge's labels; then check whether the 1,000 cases overlap with any samples used in SFT/DPO training or KB hotfixes. If the human re-score disagrees materially with the LLM judge, or if benchmark cases appear in the training data, the reported FAR/SNR and acceptance gains would not survive.","tokens_in":13974,"feed_emoji":"🛡️","tokens_out":6299,"duration_ms":50767,"temperature":0.7,"pith_summary":"This paper sets out to show that an LLM can move from generic chat assistant to reliable e-commerce risk investigator if it is given three things: a curated knowledge base of domain terminology, business logic, and fraud patterns; a reflection step that fact-checks and refines its output against that knowledge; and a data flywheel that turns daily expert feedback into better training data and faster policy updates. The authors report that the full system, SHERLOCK, reaches a factual alignment rate of 0.92 and a signal-to-noise ratio of 4.34 on an expert-verified benchmark, and that in a five-week online A/B test experts accepted 82% of its recommendations (vs 31% for the legacy system) while review time fell from 15.24 to 3.13 minutes per case. A 90-day follow-up claims the flywheel recovered performance twice after fraud tactics shifted and raised the acceptance ceiling by about 3.5% through autonomous model updates. A sympathetic reader would care because manual case investigation dominates daily risk operations, and this is a concrete path to automating the interpretive part of fraud analysis rather than only the detection part.","feed_headline":"LLM risk tool wins 82% expert acceptance, 4x faster reviews","feed_subtitle":"SHERLOCK pairs a live knowledge base with a self-improving feedback loop to automate e-commerce fraud case investigation.","key_machinery":"The load-bearing engine is a three-part cycle. First, a dynamic domain Knowledge Base (KB) stores 1,850 structured entries across three knowledge types: domain terminology (e.g., that a 'Treasure Island' order means an auction-channel order, not a brand), business logic (e.g., food-delivery IP clustering is normal), and risk patterns with thresholds and inter-feature relations. Second, a Reflect & Refine (R&R) module performs post-hoc fact verification on the LLM's risk factors, uses those fact-checked factors to retrieve relevant KB entries, and re-evaluates each claim. Third, a data flywheel routes rejected LLM outputs as high-value annotation samples, uses 'selection-over-creation' expert","core_discovery":"The central claim is that risk-investigation quality comes from coupling an LLM with an evolving structured knowledge base and a post-hoc reasoning module, not from the LLM alone. SHERLOCK's knowledge base contains 1,850 expert-validated entries distilled from documents, meeting recordings, and rule code, organized as domain terminology, business logic, and risk patterns. The model is fine-tuned on synthesized 'suspect-then-rule-out' chain-of-thought data and then runs a Reflect & Refine pass: it fact-checks each generated risk factor, retrieves targeted business logic and risk-pattern knowledge, and retains or discards claims accordingly. On a 1,000-case expert-verified benchmark this yield","pith_inferences":["The same three-part recipe—extractable domain KB, reflection against that KB, and a feedback flywheel—could transfer to other expert-judgment fields with scarce annotators, such as insurance claims, loan underwriting, or content moderation, provided the KB can be distilled and experts will do selection-style labeling.","The reported precision/recall tradeoff (CDR falls when R&R filters noise) hints that in practice the framework may be better at avoiding false accusations than at catching vanishingly rare fraud types; platforms should expect to tune the reflect step's strictness by business cost.","The 3.5% acceptance-ceiling gain from autonomous updates is small relative to the initial jump from 31% to 82%, suggesting the flywheel's main value is maintaining performance under drift rather than compounding gains.","A direct testable extension: measure whether the R&R module's retained claims are better calibrated than the raw model's, for example by comparing precision at equal recall on a balanced holdout, to separate genuine reasoning gains from pure noise suppression."],"forward_implications":["Risk investigation time per case can drop from about 15 minutes to about 3 minutes while expert acceptance of AI recommendations rises from 31% to 82%.","The R&R module's targeted retrieval—using the model's own risk factors as queries—is what unlocks most of the accuracy gain; non-targeted retrieval leaves SNR below the single-pass baseline.","Synthesized 'suspect-then-rule-out' chain-of-thought data plus DPO training that suppresses false positives improves signal-to-noise ratio more than vanilla CoT or conclusion-only fine-tuning.","Policy changes and new fraud tactics can be injected into the knowledge base as hotfixes and take effect immediately, without waiting for a retraining cycle; the flywheel's 90-day run shows recovery from performance decay twice.","A deployed system can maintain a self-improving loop where daily operations, expert annotation, and model evaluation feed each other, reducing annotation cost by prioritizing only cases the model got wrong."],"fun_headline_variants":["LLM + evolving KB lifts fraud investigation acceptance to 82%","SHERLOCK: 4x faster fraud probes with self-updating knowledge base","Fraud risk tool uses LLM and dynamic KB to hit 82% expert approval","E-commerce fraud sleuth: 386.7% throughput gain via adaptive LLM","Self-evolving LLM framework boosts fraud case output 4.8x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 1,000-case offline benchmark is assumed to be independent of the flywheel's training data, and the automated LLM judge is assumed to classify risk factors as reliably as a human expert panel; if either assumption fails, the reported FAR/SNR gains and 82% acceptance rate could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["LLM + evolving KB lifts fraud investigation acceptance to 82%","SHERLOCK: 4x faster fraud probes with self-updating knowledge base","Fraud risk tool uses LLM and dynamic KB to hit 82% expert approval","E-commerce fraud sleuth: 386.7% throughput gain via adaptive LLM","Self-evolving LLM framework boosts fraud case output 4.8x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1103,"prompt_tokens":804,"completion_tokens":299,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":548,"tokens_out":299,"duration_ms":6346,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:39:32.652430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second, independent panel of experts re-annotate the same 1,000 benchmark cases and re-score the LLM outputs without seeing the automated judge's labels; then check whether the 1,000 cases overlap with any samples used in SFT/DPO training or KB hotfixes. If the human re-score disagrees materially with the LLM judge, or if benchmark cases appear in the training data, the reported FAR/SNR and acceptance gains would not survive.","supporting_citations":[],"review_version":1}