{"id":"f4208c49-0cfe-4762-a4dc-3cf82fadfbf3","arxiv_id":"2607.23075","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM-based reinforcement-learning system with expert-suggestion reflection detects fake-order fraud with 85.3% macro-F1 and produces traceable reasoning traces, beating baselines and first-stage human reviewers.","lead":"DeepScrub is a fraud-detection system that uses large language models and reinforcement learning to flag fake orders and write a human-readable explanation for each decision. In tests on real order data it beat larger generic models and outperformed first-stage human reviewers, while cutting manual review workload by 94%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The auditability contribution rests on rationales trained to mimic label-conditioned DeepSeek-R1 generations, with no external faithfulness check; if these are post-hoc stories, traceable reasoning is unsupported even if F1 holds.","rationale":"I read the paper as making two separable claims: improved detection accuracy and traceable reasoning. The first is supported by offline and pilot numbers, though without error bars or released artifacts. The second is the paper's stated motivation—enforcement, appeals, and audit workflows require explanations. The weakest link is the faithfulness of the reasoning traces. The training loop uses the same generated rationales as ground truth for both SFT and RL rewards, so there is no independent signal that the rationales correspond to evidence. This is exactly the reader's weakest assumption, and it is load-bearing because the entire interpretability contribution collapses if rationales are post-hoc stories. A counterfactual ablation test would settle this: if model behavior is insensitive to evidence removal, the traces are decorative. I do not see grounds to reject the accuracy claims; the concern warrants conditional acceptance pending release of data/code and this faithfulness check. The reader already recommended CONDITIONAL, so my read does not change the verdict.","tokens_in":16227,"tokens_out":4274,"duration_ms":48348,"concrete_test":"Sample 200 held-out orders with DeepScrub's final traces. For each trace, extract cited evidence (e.g., '17 linked accounts', 'abnormal surge in transaction amount'). Create a counterfactual input that removes or contradicts that cited evidence while keeping the label and all other fields unchanged. Re-run DeepScrub. If the same label and same cited evidence persist in, say, more than 10% of such ablations, the trace is not faithful to the input; if the model changes its label or drops the cited evidence systematically, the traceability claim is supported. This directly tests the causal link between evidence and decision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DeepScrub's claimed differentiator over black-box fraud detectors is the auditability of its reasoning traces. That claim requires the generated rationales to be causally grounded in the input evidence. The training pipeline undermines this requirement by construction. In Sec. 4.2, rationales for unlabeled orders are produced by DeepSeek-R1 'conditioned on their labels' (CoT Generation), not derived from the evidence; they are then filtered only for label consistency. The accuracy reward (Eq. 4) assigns 10% weight to Sentence-BERT similarity between the model's reasoning and these label-conditioned rationales, and the reasoning reward (Eq. 10) is a product of a confidence/generation score and keyword overlap with answer-specific keywords. Both rewards therefore pull the model toward reproducing label-consistent narrative templates, not toward citing actual evidence. No experiment in Sec. 6 measures faithfulness: the case study (Tables 3-4) is qualitative, and the pilot adjudicates labels, not reasoning traces. If the traces are post-hoc rationalizations, the audit/interpretability benefit—the paper's central motivation—is unsupported, even if the 85.3% macro-F1 and pilot gains are accurate. This is not an internal inconsistency but a missing empirical foundation for the core claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DeepScrub is an LLM-based reinforcement learning framework for fake-order fraud detection in O2O platforms. It converts heterogeneous risk signals (graphs, sequences, tables) into textual evidence via semantic unification, continues pretraining on risk-control and general corpora, and trains a Qwen3-8B policy with SURE, a two-round suggest-reflect RL procedure with format, accuracy, and reasoning rewards. The model outputs a label, confidence, and reasoning trace. On a 150K-order proprietary dataset, DeepScrub reports macro-F1 of 85.3%, outperforming fine-tuned 8B baselines and a zero-shot 32B model. In a four-week live pilot, it reports 91.8% precision and 88.5% recall versus 75.2% and 49.7% for first-stage human reviewers, with a 94% workload reduction. The paper's stated contribution is improved detection plus traceable reasoning for enforcement and appeal workflows.","tokens_in":16583,"tokens_out":7002,"duration_ms":68977,"significance":"If the traceability claim were supported, this would be a practically valuable contribution: it combines domain-adapted LLMs with RL for a high-stakes fraud-review task, includes a real-world deployment pilot, and shows that a task-tuned 8B model can outperform a larger general model. The ablation design and chronological split are sensible, and the pilot provides concrete operational evidence. However, the paper's distinctive claim—that the reasoning traces are auditable and faithful—is not yet substantiated. The rationales used as training targets are themselves generated from ground-truth labels, and no external faithfulness measure is presented. The reported performance gains also lack variance estimates. Because the auditability motivation is central and currently rests on qualitative examples and a reward function with a conditioning error (Eq. 5), the paper requires major revision rather than acceptance.","major_comments":[{"comment":"The traceable-reasoning claim is trained and evaluated against label-conditioned rationales. CoT Generation (Sec. 4.2, App. A.4) asks DeepSeek-R1 to produce a rationale with the true label as input; the only filter is label consistency. The accuracy and reasoning rewards (Eqs. 4, 10) then reward similarity to these same rationales. A model can therefore receive high reasoning rewards by reproducing label-consistent narrative templates without grounding its decision in the transaction evidence. Tables 3-4 are qualitative case studies and do not measure faithfulness. To support the core auditability contribution, the paper needs an external test: e.g., human auditors rating whether each trace's cited evidence is present and sufficient, or counterfactual tests in which evidence is altered and the rationale changes accordingly.","section":"Sec. 4.2; Eqs. (4), (10)"},{"comment":"Eq. (5) defines A(ê,y) = (1/|y|) Σ_n Pθ(y_n | concat(ê, y_<n)); the input prompt x (the transaction evidence) does not appear in the conditioning set. Consequently, the 'reasoning reward' measures only local coherence between the generated reasoning ê and the answer y, not whether ê is grounded in the transaction data. The formula should be Pθ(y_n | x, concat(ê, y_<n)). As written, Δscore in Eq. (8) can reward a plausible-sounding rationale that ignores the evidence—exactly the failure mode the paper claims to avoid.","section":"Sec. 4.3, Eq. (5)"},{"comment":"All offline results are single-run point estimates with no error bars, significance tests, or multiple seeds. The headline gain over the strongest fine-tuned baseline is 2.7 macro-F1 points (0.853 vs 0.826), and the w/o r_reasoning ablation is only 0.9 points lower. Without variance estimates, it is unclear whether these differences are systematic. Please report means and standard deviations over at least three seeds, and sensitivity to the main hyperparameters α, μ, ω (Secs. 4.3, 6.1). This is needed to support the central performance and ablation claims.","section":"Sec. 6.2-6.3, Table 2"},{"comment":"The pilot's human baseline is described only as 'standard first-stage reviewers' with routine training; their workload, selection criteria, and review context are not specified. Final labels are adjudicated by senior experts, but the paper does not report inter-annotator agreement, the number of cases escalated, or the escalation threshold. The 94% workload reduction is presented without accounting for the cost of expert adjudication, downstream false-negative review, or inference/compute costs. These details are necessary to substantiate the production-impact and cost-savings claims.","section":"Sec. 7"}],"minor_comments":[{"comment":"The confidence weights 1, 2/3, 1/3 are arbitrary; no justification or sensitivity analysis is provided.","section":"Sec. 4.3, Eq. (1)"},{"comment":"Training details are incomplete: the SFT portion (20% of the CoT-augmented set) has no data size, number of epochs, or early-stopping criterion; the format reward is also never formally defined.","section":"Sec. 6.1"},{"comment":"The arrow 'without any annotation CoT generation' is misleading, since the CoT Generation prompt in App. A.4 includes the true label. Suggest rewording to 'without a manual rationale'.","section":"Fig. 2"},{"comment":"No evaluation details for C-Eval/C-MMLU (few-shot setup, prompt) are given; please add.","section":"Table 1"},{"comment":"The 'nearly one million RMB annually' saving is stated without a calculation; a short cost model or footnote would help.","section":"Abstract/Conclusion"},{"comment":"Given the reproducibility claim, the prompts and semantic templates should be released even if the proprietary dataset cannot be.","section":"App. A.1"}],"recommendation":"major_revision","confidential_remarks":"The core accuracy results may be sound, but the paper's distinctive contribution—traceable reasoning—is currently not supported beyond qualitative examples. The label-conditioned rationale generation is a real circularity risk and should be addressed head-on with a faithfulness experiment. I would also encourage the editor to require variance estimates and hyperparameter sensitivity, since several comparisons are within 1-3 F1 points. Scope fit is reasonable for a security/CS journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely engineered system with a production pilot, and the detection numbers are believable. Second, the paper's headline contribution — traceable reasoning — is not tested in any direct way; the rationales are trained to imitate label-conditioned DeepSeek-R1 generations, and the only evidence for faithfulness is a couple of case studies. That is the load-bearing flaw.\n\nWhat is new: the SURE suggest-reflect loop, the division into accuracy and reasoning rewards, and the semantic unification of graph/sequence/table signals into text. The parts are not individually new, but the assembly is sensible and the ablation study gives some credit to each component. The 8B-beats-32B result is a useful practical data point, and the four-week pilot with 10k orders, adjudicated by senior experts, is real evidence of operational value. I also give them credit for stating that the baseline reviewers were not senior experts and that the system runs after rule-based recall, not as a full-traffic detector.\n\nSoft spots, in order of severity. (1) The reasoning-faithfulness issue: CoT for unlabeled orders is generated by DeepSeek-R1 'conditioned on their labels'; the rewards reward similarity to those same generated rationales and keyword overlap with the label. Nothing in the evaluation checks whether the traces reflect actual evidence in the order. The case study is illustrative, not a measurement. So the auditability claim, which is the paper's motivation, is unsupported. (2) Eq. 5 defines A(e,y) as the probability of the answer given the reasoning alone, with no conditioning on the input prompt. That looks like a modeling mistake or at least an unexamined choice. (3) No code, data, error bars, or sensitivity analysis for alpha/mu/omega; the pilot numbers come from one four-week window. (4) Graph-based fraud detection baselines are cited in related work but omitted from experiments.\n\nNone of these are fatal to the detection claim. If the paper were reframed as a system paper about improving first-stage review accuracy and workload, the empirical core would stand. But the distinctive 'traceable reasoning' contribution needs an external faithfulness metric — human or aided verification that the cited evidence is the actual driver, or an intervention that checks whether the model's decision changes when evidence is removed.\n\nWho this is for: risk-control practitioners and LLM-RL researchers interested in domain adaptation. It deserves a serious referee, but the referee should insist on an faithfulness experiment and a release of at least a redacted dataset or code.","headline":"A plausible and well-engineered fraud-detection system whose central traceability claim is not actually measured; worth refereeing if the authors can supply a real faithfulness check.","tokens_in":17030,"tokens_out":2174,"would_cite":false,"duration_ms":20729,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeepScrub shows that a small, domain-tuned LLM with suggest-reflect reinforcement learning can detect fake-order fraud more accurately than a much larger general model or routine human reviewers, while producing inspectable reasoning traces","keywords":["fake-order fraud","brushing detection","LLM reasoning","reinforcement learning","traceable evidence","suggest-reflect","O2O platform","risk-control review"],"falsifier":"Take a sample of orders the model classified, and for each one remove or contradict the specific evidence named in its reasoning trace (for example, 'abnormal surge in transaction amount' or '17 linked accounts') while leaving the rest of the input intact; if the verdict stays the same in a large share of cases, the traces are post-hoc rationalizations rather than decision evidence. A second check: replace the reference reasoning used during training with a deliberately inconsistent rationale and confirm the reasoning reward decreases — if it cannot detect the inconsistency, the traceability m","tokens_in":16143,"feed_emoji":"🛡️","tokens_out":6176,"duration_ms":62282,"temperature":0.7,"pith_summary":"The paper sets out to prove that fake-order fraud detection on large online-to-offline platforms can be made simultaneously more accurate and more auditable by replacing handcrafted features and black-box classifiers with a compact language model that reasons in text. The proposed system, DeepScrub, translates graphs, behavioral sequences, and transaction tables into unified text, continues pretraining on risk-control material, and then trains the model with a suggest-reflect reinforcement loop in which the model either double-checks its own reasoning or receives targeted suggestions from expert models before giving a final verdict. Trained this way, an 8B-parameter model reaches 85.3% macro-F1 offline, beating a 32B general model and all fine-tuned baselines. In a four-week live pilot it reached 91.8% precision and 88.5% recall, improving on routine first-stage human reviewers by 16.6 and 38.8 percentage points, while cutting first-stage manual review workload by 94% and saving nearly one million RMB per year. The payoff the authors argue for is a deployed reviewer whose verdicts come with reasoning chains that enforcement and appeal workflows can inspect.","feed_headline":"Tuned 8B LLM beats 32B model and human fraud reviewers","feed_subtitle":"Auditable reasoning traces lift precision to 91.8% and cut first-stage manual review by 94%.","key_machinery":"The load-bearing mechanism is SURE (SUggest-REflect), a two-round reinforcement training loop. The policy first samples a group of candidate answers; a dual-branch suggestion module decides, from the accuracy reward, whether a candidate is reliable enough for intrinsic self-verification or needs extrinsic hints from a transaction-expert model and a risk-control expert model; the policy then re-reasons with those suggestions to emit a refined answer. The advantages of the first- and second-round responses are blended into a mixed advantage used in a group-relative policy optimization update. Three rewards shape the behavior: a format reward enforcing structured <reason>/<answer> output; an ac","core_discovery":"DeepScrub's central claim is that jointly optimizing prediction correctness and reasoning quality produces a fraud reviewer that is more accurate and more inspectable than rule-based screening, black-box models, or routine human first-stage review. The framework unifies heterogeneous risk evidence into text, continues pretraining on a mix of risk-control and general corpora, and trains with a SUggest-REflect (SURE) mechanism: low-confidence first-round answers receive suggestions from transaction- and risk-control expert models before the policy reflects and produces a final answer, while high-confidence answers are prompted to self-verify. The advantages of both rounds are blended and appli","pith_inferences":["The auditability benefit stands or falls on whether the emitted rationales faithfully reflect the evidence that drives the decision; the paper rewards similarity to reference rationales generated by a teacher model but does not independently verify that the traces are causally tied to the prediction. A perturbation test — deleting the cited evidence from an input and checking whether the predictio","The same semantic-unification-plus-suggest-reflect recipe could generalize to other domains with heterogeneous features and defensibility requirements, such as credit underwriting, content moderation, or insurance claim review.","The pilot compares against routine first-stage reviewers, not senior fraud experts; reported human-beating margins should be read against that baseline, and an expert-level comparison would be a stricter test of the model's value.","The found 1:15 ratio of domain to general pretraining data suggests a tunable trade-off between domain fluency and general capability; other deployments would need to retune it for their data mix."],"forward_implications":["Small, task-optimized LLMs can beat much larger general-purpose LLMs on specialized risk-control tasks, so deployment cost need not scale with model size.","Fraud verdicts can be delivered with human-readable reasoning chains usable in enforcement, appeals, and expert adjudication.","Automating first-stage review at 91.8% precision and 88.5% recall can reduce manual review workload by 94% and save roughly one million RMB per year on a large O2O platform.","The suggest-reflect training recipe is transferable to other high-stakes classification settings where both accuracy and explainability are required."],"fun_headline_variants":["Auditable LLM fraud detector beats 32B and human reviewers","RL-trained LLM with reasoning traces beats humans and 32B model","Fraud detection LLM: 8B beats 32B, cuts manual review workload by 94%","Traceable LLM fraud review: beats humans, slashes first-stage work","DeepScrub: auditable fraud reasoning lifts 8B LLM over 32B and humans"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reasoning traces the model is trained and rewarded to emit genuinely reflect the evidence driving its decisions, rather than being post-hoc narratives produced after the label is known; the paper does not independently verify this faithfulness.","fun_headline_variants_meta":{"raw":{"variants":["Auditable LLM fraud detector beats 32B and human reviewers","RL-trained LLM with reasoning traces beats humans and 32B model","Fraud detection LLM: 8B beats 32B, cuts manual review workload by 94%","Traceable LLM fraud review: beats humans, slashes first-stage work","DeepScrub: auditable fraud reasoning lifts 8B LLM over 32B and humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001613,"raw_usage":{"total_tokens":6293,"prompt_tokens":814,"completion_tokens":5479,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":5379}},"tokens_in":558,"tokens_out":5479,"duration_ms":33300,"temperature":1.0,"reasoning_tokens":5379,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T03:39:21.209684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of orders the model classified, and for each one remove or contradict the specific evidence named in its reasoning trace (for example, 'abnormal surge in transaction amount' or '17 linked accounts') while leaving the rest of the input intact; if the verdict stays the same in a large share of cases, the traces are post-hoc rationalizations rather than decision evidence. A second check: replace the reference reasoning used during training with a deliberately inconsistent rationale and confirm the reasoning reward decreases — if it cannot detect the inconsistency, the traceability m","supporting_citations":[],"review_version":1}