{"id":"f7118510-8425-4f0e-beed-196c67ca6d96","arxiv_id":"2412.08680","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of GPT-4 and Gemini prompts, fitted to 300 hand-labeled CFPB complaints, labels scam vs. non-scam fraud with reported precision .95/recall .84 on the same training set and precision .97 on a 133-complaint sample.","lead":"This paper builds an LLM ensemble that labels consumer complaints as either scams or ordinary fraud, using a hand-labeled set of 300 CFPB narratives. It provides an early benchmark for how well AI chatbots can distinguish the two.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported performance is not an unbiased estimate: F was tuned on the labeled set L, and the only external check is precision on predicted positives, leaving out-of-sample recall unmeasured.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and the weakest assumption they identify overlaps with the concern raised here. I differ slightly in emphasis: the labeling rule is real but secondary, because even granting the self-report-based labels, the reported performance is not established for new data. The iterative tuning on L means the headline precision/recall are development-set numbers, and the manual check of predicted positives addresses only one side of the error profile. A held-out evaluation with independent annotators and both precision and recall would settle the question. No part of the paper's reasoning is internally inconsistent, and the authors are transparent about limitations, but the central capability claim is currently supported by weaker evidence than the abstract and strongest claim suggest. The conditional recommendation remains the right call: the contribution is promising but requires independent validation before the high-precision/high-recall characterization is accepted.","tokens_in":6763,"tokens_out":4209,"duration_ms":53134,"concrete_test":"Select a random holdout of roughly 300 'fraud or scam' CFPB narratives from the January 1–November 13, 2024 window that were never used during prompt development. Have two annotators who did not participate in prompt design independently label each narrative using the Section 2 scam/fraud definition, recording inter-annotator agreement, then run ensemble F on the same holdout. Compute precision and recall with binomial confidence intervals. If out-of-sample precision and recall are close to the L numbers and labels are reliable, the generalization claim is confirmed; if recall falls materially, the in-sample recall overstates capability and the corpus construction needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—precision of .95 and recall of .84 on set L—is an in-sample result. Section 3 describes an iterative process in which prompts were revised after inspecting errors on L, so these numbers are fitted to L rather than estimates of performance on new complaints. The only external evaluation is a manual review of a 10% sample of the 1,333 narratives F predicted as scam; that yields precision only. It cannot detect systematic under-calling of scams, so the paper's capability claim is not supported for new data. A secondary issue is that the 'customer is always right' labeling rule makes the gold standard partly a function of complainant assertions; if F exploits the presence of words like 'scam' or complaint structure, high scores would not indicate robust understanding of deception. The decisive weakness is that the reported evaluation protocol cannot separate these possibilities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops an LLM ensemble prompt (Gemini with Prompt A, GPT-4 with Prompts B and C, combined by conjunction) to distinguish scam from non-scam fraud complaints in the CFPB database. The authors manually labeled 300 'fraud or scam' narratives (set L) through an iterative process, report that the ensemble F achieves precision .95 and recall .84 on L, apply F to 2,569 narratives from 2024 to identify 1,333 scams, and manually check a 10% sample of predicted positives (n=133) with precision .97. They also analyze error patterns related to secondary information, company reputation, narrative length, and redaction, and they note that F performs poorly (precision .25, recall .33) on complaints not labeled 'fraud or scam'.","tokens_in":6946,"tokens_out":2407,"duration_ms":26145,"significance":"If the performance claims are trustworthy, the contribution is valuable: an organic corpus of consumer scam narratives, an ensemble prompting technique, and an error analysis that identifies concrete weaknesses of LLMs in scam detection. The paper is transparent about some limitations, such as the model's specialization to the 'fraud or scam' subset. The main strengths are the manual labeling effort, the explicit prompting details, and the initial error-pattern findings. However, the central quantitative claims are undermined by the evaluation protocol, which is in-sample for the headline precision/recall and measures only precision externally. The contribution is therefore promising but not yet fully supported.","major_comments":[{"comment":"The reported precision of .95 and recall of .84 on set L are in-sample estimates. The text states that prompts were iteratively revised after evaluating against L and manually reviewing errors on L, so the final ensemble F and its conjunction rule were selected using L. Reporting performance on the same data used for model selection is fitting-then-evaluating; these numbers are optimistically biased and cannot be interpreted as expected performance on new complaints. The paper should either hold out a portion of L for final evaluation or clearly reframe the .95/.84 as training-fit diagnostics and base the capability claim on an out-of-sample evaluation.","section":"Section 3, 'Prompt Design and Iteration' and 'Final Prompt and Performance'"},{"comment":"The external evaluation—manual review of a randomly selected 10% sample (n=133) of the 1,333 predicted positives—measures only precision on predicted scams. It does not measure recall because false negatives are never sampled. Since the paper's headline capability claim includes recall of .84, and since the in-sample recall is not a valid estimate, the paper currently provides no evidence about out-of-sample recall. The authors should either annotate a random sample of all 2,569 narratives (or a random sample of narratives not predicted as scam) to estimate recall externally, or explicitly state that out-of-sample recall is unknown.","section":"Section 3, 'Final Prompt and Performance'"},{"comment":"The 'customer is always right' labeling rule makes the gold-standard labels partly a function of complainant assertions. If a complainant writes 'I was scammed' with no further detail, the label is 'scam' by construction. This creates a potential confound: F may achieve high agreement with L by detecting explicit scam language or complaint structure rather than by identifying deceptive schemes. The paper should test this, for example by reporting performance on the subset of L where the narrative does not contain the word 'scam' or explicit scam assertions, or by discussing why such a test is not feasible. Without this, the high scores on L could reflect superficial cue detection.","section":"Section 3, 'Training data'"}],"minor_comments":[{"comment":"The phrase 'high inter-related reliability' appears to be a typo for 'inter-rater reliability'; please correct.","section":"Section 3, 'Training data'"},{"comment":"There is a typo in 'complainaint' in the sentence 'the complainaint's statements would be assumed correct'; please correct to 'complainant'.","section":"Section 3, 'Training data'"},{"comment":"The sentence 'Several prompts included example complaints and labels since examples have been associated with improved performance in other contexts (e.g., [21])' cites a survey of LLM annotation; a more specific reference on few-shot prompting would strengthen the claim.","section":"Section 3, 'Prompt Design and Iteration'"},{"comment":"Figure 1 is described in text but the caption only says 'Precision and recall of the final model, F, as a function of complaint narrative length in characters.' It would help to state the binning or smoothing method used to produce the curves, since the underlying data are discrete narrative lengths.","section":"Section 4, 'Performance and Narrative Length'"},{"comment":"The sentence 'We evaluated the prompt on a random subset of CFPB complaints that are not labeled with the 'fraud or scam' issue or sub-issue and found lower precision and recall' is vague about the sample size and selection; please add details so readers can gauge the reliability of the .25/.33 numbers.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short, exploratory contribution and the core idea—an ensemble of purpose-built prompts for a specialized classification task—is reasonable. The main obstacle is the evaluation: the headline precision/recall is in-sample, and the external check covers only precision. If the authors can provide an out-of-sample evaluation with both precision and recall, perhaps on a held-out set of 'fraud or scam' narratives, the paper would meet the standard for publication. The current version overstates the empirical support and should be revised before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is the corpus: 300 hand-labeled CFPB narratives, the first organic (non-synthetic) scam narratives used for LLM detection research, plus a set of qualitative error observations that are concrete and actionable. The authors describe their labeling protocol carefully—two independent labelers, a shared codebook, reconciliation, and a documented 'customer is always right' default. The error-pattern findings (length, redaction tolerance, over-reliance on company names and secondary details like claim denials) are the real contribution, and they hold up as observations regardless of the model's exact scores.\n\nWhat does not hold up is the central performance claim. The ensemble F was iteratively tuned on L by inspecting errors on L, and then the paper reports precision .95 and recall .84 on that same L. Those are fitted numbers, not estimates of performance on new complaints. The only external check is manual review of 133 predicted positives, which gives precision (.97) but no recall; it cannot catch systematic under-calling. So the capability claim is not supported for new data. The 'customer is always right' rule also makes the gold standard partly a function of complainant assertion—if someone says 'scam,' it is labeled scam unless the narrative contradicts it. That is defensible given the definition, but it means high scores could partly reflect the model detecting the word 'scam' or a complaint's structure rather than understanding deception. The authors are transparent about the model's failure on general complaints and call the work preliminary, which is to their credit, but the numbers are still presented without the in-sample caveat until the reader digs into Section 3.\n\nThese are real weaknesses, but proportionate to the paper's modest goal. The paper is a short empirical note, not a claim to have solved scam detection. The error analysis and the corpus are worth having, and the authors plan to release data. That makes this a reasonable paper to send to peer review—it needs a held-out evaluation and a clearer statement that the headline numbers are descriptive of the fitting process, not predictive. I would engage with it, and I would cite the corpus and error findings, but not the performance numbers as validated.\n\nRecommendation: accept for review with a request for out-of-sample evaluation and explicit separation of prompt development from evaluation data. Reading group: maybe, for people working on LLM annotation and scam defense.","headline":"Useful first organic scam/fraud corpus and an honest error analysis, but the headline performance numbers are in-sample and should not be read as validated.","tokens_in":7468,"tokens_out":1868,"would_cite":true,"duration_ms":22805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an ensemble of three LLM prompts—Gemini on the trust-based definition and GPT-4 on two complementary questions—can label CFPB consumer complaint narratives as scam versus non-scam fraud with precision 0.95 to 0.97…","keywords":["LLM ensemble","scam detection","fraud complaints","CFPB database","prompt engineering","consumer narratives","error analysis","chain-of-thought prompting"],"falsifier":"Manually label a random sample of 'fraud or scam' narratives using a codebook that requires objective evidence of deception (e.g., trust-building followed by a voluntary transfer to an unknown party), then compare the ensemble's precision on narratives that contain the word 'scam' versus those that never use the word; a large drop on the word-free subset would show the model detects the label rather than the scam.","tokens_in":6560,"feed_emoji":"🤖","tokens_out":8020,"duration_ms":68888,"temperature":0.7,"pith_summary":"This paper sets out to separate two kinds of consumer harm in the CFPB complaints database: scams, where the victim is tricked into authorizing a harmful transaction, and non-scam fraud, where the transaction is unauthorized. The authors build a manually labeled set of 300 complaints, iterate on prompts for Gemini and GPT-4, and combine the two models with an 'all must say scam' rule. On that labeled set the ensemble reaches precision 0.95 and recall 0.84, and a manual check of a random 10% of its positive predictions (133 narratives) finds precision 0.97. The paper also reports error patterns: both models lean on secondary details such as customer-service disputes and claim denials, treat company names as signs of reputation, and perform worse on very long or heavily redacted narratives. If correct, these findings give researchers a high-precision corpus of organic scam narratives and concrete guidance for how consumers and developers should use LLMs for scam defense.","feed_headline":"Three-prompt LLM ensemble sorts scams from fraud at 97% precision","feed_subtitle":"Gemini and GPT-4 judgments label CFPB narratives, building a high-precision organic scam corpus.","key_machinery":"The load-bearing mechanism is the conjunction ensemble: scam is predicted only if all three prompts—Gemini with the trust-based definition, GPT-4 with the 'money stolen vs. tricked vs. neither' question, and GPT-4 with the 'potential scam vs. not scam' reputation question—independently classify the narrative as a scam. This AND rule is what buys precision, at the price of recall. The prompts themselves rely on chain-of-thought reasoning, asking each model to explain its label, and on a small set of in-prompt examples. The ensemble is built for the specific 'fraud or scam' issue tag in the CFPB database and is not a general scam detector: on complaints without that tag, precision and recall drop to 0.25 and 0.33.","core_discovery":"The central claim is that a simple ensemble of three LLM prompts, evaluated with a conjunction rule, distinguishes scam from non-scam fraud in CFPB complaint narratives at practically usable precision. The definition used is that a scam involves the complainant being tricked into voluntarily taking a self-harming financial action, whereas non-scam fraud is an unauthorized action; the prompts encode this distinction and require each model to explain its answer. The final ensemble, denoted F, predicts 'scam' only when Gemini (Prompt A), GPT-4 with the two-option question (Prompt B), and GPT-4 with the reputation-based question (Prompt C) all agree. On the 300 manually labeled narratives L, F achieves precision 0.95 and recall 0.84; applied to 2,569 recent 'fraud or scam' narratives, it labels 1,333 as scams, and a manually reviewed random 10% sample of those positive predictions shows precision 0.97. The authors state that this is the first corpus of organic consumer scam narratives, as opposed to the synthetic scam narratives used in earlier LLM evaluations.","pith_inferences":["The 'customer is always right' labeling rule likely makes 'scam' labels correlate with the presence of the word itself; a held-out test using narratives that describe deception without ever saying 'scam' would clarify whether the model detects trickery or a keyword.","The reliance on secondary evidence suggests a latent bias: complaints that mention poor customer service or denied claims may be systematically over-labeled as scams, which could matter if such models ever inform consumer advice.","The conjunction design implies a precision/recall trade-off curve; varying the number of agreeing models or the strictness of individual prompts would map that curve and may show that two models already suffice under some operating points.","Because the model only works on narratives already tagged 'fraud or scam', a practical deployment would still need a first-stage classifier or user self-selection; extending the ensemble to untagged narratives is an open problem the paper does not solve."],"forward_implications":["A high-precision subset of the CFPB database can now be treated as a corpus of organic scam narratives, supporting future measurement of LLM scam-defense behavior.","The error-pattern analysis implies that consumers should keep narratives focused on interactions with the suspected scammer and should simulate or generalize personal information rather than redact it, since redaction hurts performance and long off-topic text increases errors.","Developers fine-tuning LLMs for scam defense should treat company names and secondary events like claim denials as unreliable signals, because both models leaned on them in error explanations.","The ensemble's success on tagged 'fraud or scam' complaints does not transfer to untagged complaints, so the CFPB's issue tag or an equivalent filter is a necessary precondition for the method.","The observed length effect (performance degrades beyond roughly 3,000 characters) gives a concrete rule of thumb for prompt design and user guidance."],"supporting_citations":[{"why":"Supplies the CFPB complaint narratives that constitute the corpus and the 'fraud or scam' issue tag that defines the problem.","marker":"[4]"},{"why":"Provides the psychological definition of scam compliance that anchors the authors' labeling codebook.","marker":"[11]"},{"why":"The prompt engineering survey that the iterative prompt development builds on.","marker":"[16]"},{"why":"Supports the premise that LLMs can serve as reliable annotators with well-designed prompts.","marker":"[21]"},{"why":"Chain-of-thought prompting, cited as the reason the prompts require explanations.","marker":"[24]"},{"why":"Prior measurement study using synthetic scam narratives, which this paper extends with organic narratives.","marker":"[18]"},{"why":"Documents the CFPB redaction process that shapes the narratives and drives the redaction-related error analysis.","marker":"[22]"}],"fun_headline_variants":["LLM trio catches scams with 97% precision in CFPB data","Ensemble of three LLMs flags scams at 97% precision","95% precision LLM ensemble separates scams from fraud"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The labeling rule that 'the customer is always right' treats complainants' self-reports of being scammed as ground truth, so if people call ordinary fraud or poor service a 'scam' without objective evidence of trickery, the model's high precision could reflect keyword matching rather than genuine scam detection.","fun_headline_variants_meta":{"raw":{"variants":["LLM trio catches scams with 97% precision in CFPB data","Ensemble of three LLMs flags scams at 97% precision","95% precision LLM ensemble separates scams from fraud"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2504,"prompt_tokens":826,"completion_tokens":1678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":1620}},"tokens_in":442,"tokens_out":1678,"duration_ms":12310,"temperature":1.0,"reasoning_tokens":1620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:57.018129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually label a random sample of 'fraud or scam' narratives using a codebook that requires objective evidence of deception (e.g., trust-building followed by a voluntary transfer to an unknown party), then compare the ensemble's precision on narratives that contain the word 'scam' versus those that never use the word; a large drop on the word-free subset would show the model detects the label rather than the scam.","supporting_citations":[{"cited_title":"Consumer complaints database","cited_arxiv_id":null,"evidence_quote":"Supplies the CFPB complaint narratives that constitute the corpus and the 'fraud or scam' issue tag that defines the problem."},{"cited_title":"Scam compliance and the psychology of persuasion","cited_arxiv_id":null,"evidence_quote":"Provides the psychological definition of scam compliance that anchors the authors' labeling codebook."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting, cited as the reason the prompts require explanations."},{"cited_title":"Can LLMs be Scammed? A Baseline Measurement Study","cited_arxiv_id":"2410.13893","evidence_quote":"Prior measurement study using synthetic scam narratives, which this paper extends with organic narratives."},{"cited_title":"OFFICE OF CONSUMER RE- SPONSE","cited_arxiv_id":null,"evidence_quote":"Documents the CFPB redaction process that shapes the narratives and drives the redaction-related error analysis."}],"review_version":1}