{"id":"07d91e7e-772d-487f-9af0-38198de59529","arxiv_id":"2502.03964","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM-based turn-by-turn analysis of phone conversations can flag scams in real time with high recall, and an optional 'UNCERTAIN' label trades recall for precision.","lead":"This paper tests whether large language models can spot phone scams while a call is still happening by analyzing each spoken turn as it arrives. It finds high recall for the five models tested, and shows that adding an 'uncertain' option improves precision at the cost of some missed scams.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's high recall is computed at call level as 'ever flagged' from perfect transcripts; no time-to-detection, ASR, or latency is measured, so the real-time central claim is not supported by the reported numbers.","rationale":"The reader's weakest assumption was that clean-transcript turn-by-turn evaluation reproduces live-call conditions; my concern is closely related but more internal: even if the transcripts were perfect, the reported metrics do not establish timely detection. The paper never states whether precision/recall are computed per utterance, per call with any-flag aggregation, or per call at a fixed decision point. The examples strongly suggest any-flag aggregation, under which a model that screams FRAUD only on the last turn receives the same credit as one that alerts before the victim acts. That makes the central 'real-time' claim logically underdetermined by Table 1. I also agree with the reader that missing ASR/latency further weakens ecological validity. However, the UNCERTAIN-prompt finding is a plausible and useful qualitative result, and the paper is clearly framed as exploratory; it identifies its own limitations and defers user studies to future work. Therefore the appropriate disposition remains CONDITIONAL, matching the reader's verdict, with revisions requiring deadline-aware metrics and ideally a live or ASR-corrupted pilot before claiming real-time intervention.","tokens_in":9014,"tokens_out":5753,"duration_ms":61751,"concrete_test":"Recompute Table 1 with a time-to-detection constraint: mark a scam call as detected only if the model's first 'FRAUD' output occurs at or before the scam's first action point (e.g., first request for payment, ID, or verification code), as annotated in the transcripts. Report recall and precision under this definition for RT and UNC. If the constrained recall is substantially below the reported 0.98-1.00, the high unconstrained recall does not support real-time intervention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM turn-by-turn detection can warn users during a live call. Two load-bearing conditions are unmeasured. First, Section 4.1 and Table 1 report recall and precision without specifying the unit of analysis; the described examples (Figure 4: false positive at utterance 15; alerts at utterance 6 vs 10) indicate a call is counted as detected if any turn outputs FRAUD. Under that aggregation, recall 1.00 is compatible with a system that alerts only at the final utterance of a scam call, which would not prevent harm. Section 4.3.2 discusses the recall/timeliness trade-off only with two anecdotal utterance indices; no distribution of first-alert time is reported. Second, the evaluation feeds clean written transcripts to the LLM (Section 3.1.1), with no streaming ASR, no per-turn API latency, and no alerting mechanism. A conversational turn lasts a few seconds; LLM inference over a growing context can take longer, and ASR errors on Chinese scam calls are non-negligible. Thus the 0.98-1.00 recall and the UNC precision gain are upper-bound offline estimates, not evidence of real-time intervention. The paper's own conclusion admits a user study and notification strategy are future work, but the title and RQ1 claim real-time performance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based framework for detecting phone scams in real time. A system model with Scammer, User, and LLM-based Detector is presented, and the detection method classifies each conversational turn as FRAUD or SAFE, with an optional UNCERTAIN category to defer judgment. The evaluation uses Chinese transcripts from two datasets (authentic and synthetic) drawn from the authors' prior work [27] and five LLMs (GPT-4, GPT-4o, GLM4, Doubao-Pro-32k, ERNIE-3.5-8k). Table 1 reports accuracy, precision, recall, and F1 for binary real-time detection (RT), uncertainty-enabled detection (UNC), and a retrospective baseline (RET). The main empirical findings are that RT achieves recall between 0.98 and 1.00 across models and datasets, while precision on the synthetic dataset is lower (0.70–0.77); adding UNC raises synthetic precision to 0.83–0.85 at the cost of recall dropping to 0.90–0.98. The authors analyze false-positive categories, attribute them to limited context and keyword sensitivity, and discuss the precision–timeliness trade-off. The conclusion identifies notification strategies and user studies as future work.","tokens_in":9226,"tokens_out":2516,"duration_ms":25025,"significance":"If the reported results hold, the paper would show that LLMs can assess fraudulent intent from partial conversation context with high sensitivity, and that an uncertainty option is a workable mechanism for trading recall against precision. The contribution is potentially useful for the human-centered security community because it addresses an important harm and proposes a concrete detection approach. The paper has several strengths: it evaluates five diverse LLMs, uses two datasets, compares against a retrospective baseline, and includes concrete false-positive examples that illustrate the failure modes. However, the significance is conditional because the central 'real-time' claim rests on an offline simulation over clean transcripts, and the reported aggregate metrics do not establish that warnings arrive early enough to prevent harm.","major_comments":[{"comment":"The unit of analysis for the metrics in Table 1 is not specified, and the surrounding text and examples indicate that a call is counted as detected if any single turn returns FRAUD. Under this 'ever flagged' aggregation, a recall of 1.00 is compatible with a system that only alerts on the final utterance of a scam call, which would not provide the timely intervention claimed in the title and RQ1. The paper must report per-utterance classification performance, the distribution of first-alert utterance indices, and the time-to-detection relative to call length; without these, the high recall values do not support the real-time warning claim.","section":"§4.1, Table 1"},{"comment":"The evaluation feeds clean written transcripts to the LLM one utterance at a time, with no streaming speech-to-text, no per-turn API latency, no turn-segmentation mechanism, and no alerting component. The real-time protocol is also underspecified: the number of prior utterances included in the context, the temperature, and the exact API versions are not reported. As a result, the precision and recall in Table 1 are upper-bound estimates for an offline simulation, not measurements of a real-time system. The authors should either implement and measure an end-to-end pipeline with ASR and latency, or explicitly reframe the contribution as turn-by-turn detection on transcripts and state that real-time performance remains untested.","section":"§3.1.1, §2.2"},{"comment":"Table 1 contains no sample sizes, confidence intervals, significance tests, or repeated runs. The differences between RT and UNC on the synthetic dataset (e.g., precision 0.70 to 0.83 for Doubao) are presented as definitive improvements, but with small datasets and no variance estimates these differences may not be reliable. The paper should report the number of scam and benign calls in each dataset and provide statistical tests or confidence intervals for the key precision and recall comparisons.","section":"§3.3, Table 1"},{"comment":"The timeliness analysis is supported only by two anecdotal utterance indices from one example call: the RT system flags at Utterance 6 and UNC at Utterance 10. No distribution of detection latency is provided, and the paper does not report how often UNC delays detection or how large the delays are. Since the central value proposition is warning users before harm occurs, the paper needs a quantitative timeliness analysis, not just illustrative examples.","section":"§4.3.2"}],"minor_comments":[{"comment":"The phrase 'combat of phone scams' should be 'combat phone scams'.","section":"§1"},{"comment":"The text says 'the detection method classifies the conversion as fraud'; 'conversion' should be 'conversation'.","section":"§4.2"},{"comment":"'Generalize ability' should be 'generalization ability'.","section":"§3.2"},{"comment":"The label 'Real-time Detection' for the RT and UNC columns is misleading because the experiments use offline transcripts; consider naming these columns 'Turn-by-turn (simulated)' or adding an explicit caveat in the table caption.","section":"§4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closely related to the authors' prior work [27], which supplies both the datasets and the RET baseline; this is not inherently problematic, but the novelty relative to [27] should be clarified more sharply in the revision. The main risk is that the title and RQ1 promise real-time detection while the experiments are an offline simulation. If the authors cannot add time-to-detection metrics or an ASR/latency measurement, the paper would be better reframed as a study of incremental turn-by-turn classification, with real-time deployment left to future work. The paper is otherwise within scope for a human-centered security venue, though the current version lacks the user-study component that would make it a stronger fit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is not a real-time system paper; it is an offline study that feeds clean transcripts to an LLM one utterance at a time. Second, the one genuinely useful result is the UNCERTAIN prompt: on the synthetic dataset it raises precision from the 0.70–0.77 range to 0.77–1.00 across models, at a recall cost from roughly 1.00 down to 0.90–0.98. That is a believable precision/recall trade-off and worth building on.\n\nWhat the paper does well: it evaluates five LLMs, uses both real and synthetic Chinese transcripts, and gives concrete examples of false positive categories (identity verification, payment talk, urgent decisions). The qualitative discussion of why full-context retrospective analysis gets higher precision than incremental judgments is sensible. The authors are also honest that a user study and notification strategy are future work.\n\nThe soft spots are real but not fatal. The stress-test concern holds: Table 1 does not report the unit of analysis, and call-level recall of 1.00 is compatible with a system that only flags the final utterance. The paper gives two anecdotal utterance indices but no distribution of first-alert time, which is the central metric for a warning system. There is also no streaming ASR, no per-turn API latency, and no alert mechanism, so the title's \"real-time\" claim is aspirational rather than measured. Smaller issues: no error bars or significance tests, datasets and code are not released, and the RET baseline is taken from the authors' prior paper rather than re-run under the same protocol.\n\nNone of this is disqualifying for an exploratory paper. The core finding is not an artifact; it is an upper-bound estimate of what turn-by-turn LLM classification can do on clean transcripts. The UNCERTAIN result is new enough, and the application is important enough, to justify refereeing.\n\nSend it out, but ask for major revision: report first-alert time distributions, state the aggregation unit explicitly, release the data or a detailed protocol, and either re-run RET or explain why the old baseline is comparable. A live-call pilot would be ideal but can be future work if the title and abstract are toned down.","headline":"A competent offline feasibility study of turn-by-turn LLM scam detection with a useful UNCERTAIN prompt, but the title's 'real-time' claim outruns what was actually measured.","tokens_in":9802,"tokens_out":2346,"would_cite":false,"duration_ms":25789,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM that reads a phone call one utterance at a time flags scams with recall between 0.98 and 1.00, and an 'UNCERTAIN' option trades some of that recall for higher precision.","keywords":["phone scam detection","large language models","real-time fraud detection","conversational AI","uncertainty deferral","precision-recall trade-off","Chinese call transcripts"],"falsifier":"Run the same RT and UNC prompts on transcripts produced by a live automatic-speech-recognition pipeline, with imperfect transcription, overlapping speech, and realistic pauses, and compare precision and recall against Table 1; if recall drops materially or the alert arrives after the victim would have acted, the central claim fails.","tokens_in":8781,"feed_emoji":"📞","tokens_out":6060,"duration_ms":52093,"temperature":0.7,"pith_summary":"Phone scams remain hard to stop because existing defenses act before the call (blacklists, caller ID) or after it (retrospective analysis), leaving the call itself unprotected. This paper argues that a large language model can fill that gap by reading an ongoing conversation one utterance at a time and classifying each new turn as fraudulent, safe, or uncertain. In experiments on Chinese transcripts, all five tested LLMs achieved recall between 0.98 and 1.00, meaning almost every scam call was flagged. The paper also shows that letting the model answer 'UNCERTAIN' raises precision on a synthetic dataset (from 0.70–0.77 to 0.80–0.85 for the weakest models) at a measurable cost in recall (0.90–0.98). The point of the work is that real-time, mid-call detection is within reach, with the main open problem being the trade-off between false alarms and alert timing.","feed_headline":"Per-utterance LLM checks catch phone scams at 98–100% recall","feed_subtitle":"Five LLMs hit near-perfect recall on Chinese call transcripts; an 'uncertain' option curbs false alarms at a measurable recall cost.","key_machinery":"The load-bearing mechanism is turn-by-turn evaluation with conversation history: after each speaker's utterance, the full transcript so far is fed to an LLM with one of two prompts. The binary 'RT' prompt forces a FRAUD/SAFE decision; the 'UNC' prompt adds an UNCERTAIN category so the model can defer when evidence is thin. The deferral mechanism is what carries the precision improvement, because it prevents premature keyword-triggered alarms and lets later context resolve the call.","core_discovery":"The paper's central claim is that turn-by-turn LLM analysis can detect phone scams in real time with high sensitivity: across GPT-4, GPT-4o, GLM4, Doubao-Pro-32k, and ERNIE-3.5-8k, the binary real-time prompt ('RT') scored recall between 0.98 and 1.00 on both authentic and synthetic Chinese call datasets, with several models at 1.00. The weakness of this real-time mode is precision: on the synthetic dataset, where benign calls deliberately contain suspicious keywords, RT precision ranges between 0.70 and 0.77 for most models, and the paper traces these false positives to the model deciding on partial context, e.g., flagging a flight rebooking call when the agent says 'payment.' Its proposed fix is an 'UNCERTAIN' option in the prompt ('UNC'), which lets the model defer judgment until more context arrives; this raises synthetic-dataset precision to 0.80–0.85 (and to 1.00 for one model) while recall drops to 0.90–0.98, and it visibly delays the alert in a worked scam-call example from utterance 6 to utterance 10. Against a retrospective baseline that sees the whole call, real-time detection trades precision for timeliness, and the paper argues this trade-off is the central design problem for any deployed system.","pith_inferences":["A natural extension the paper does not run is an end-to-end test with automatic speech recognition, speaker diarization, and network latency; measurement noise in those components could lower recall below the clean-transcript numbers.","The false-positive categories the paper identifies suggest a hybrid design: use UNCERTAIN deferral only when a call touches identity, payment, and urgency keywords, and alert immediately outside those contexts, preserving both precision and timeliness.","The same turn-by-turn protocol could be applied to non-Chinese languages, but the keyword-triggered false-alarm pattern means language-specific tuning would be needed before the precision numbers transfer.","The alert-timing trade-off could be formalized as a cost model: assign a dollar or trust cost to false alarms and a harm cost to missed or late scam alerts; the optimal deferral threshold would then follow from the user's or operator's loss function."],"forward_implications":["If the reported recall transfers to real calls, a detector could warn a user mid-conversation at the first clearly fraudulent utterance, before money or personal information changes hands.","Because the precision gap appears on synthetic keyword-rich calls, legitimate customer-service conversations about payments, IDs, or deadlines will be the main source of false alarms in deployment.","Choosing the UNCERTAIN option means accepting lower recall and later alerts in exchange for fewer false alarms; a deployed system must pick a point on this curve, not a single accuracy number.","The consistency across five different LLMs suggests that the result is not an artifact of one model family, so future improvements can be compared against roughly the same recall ceiling."],"supporting_citations":[{"why":"Supplies the retrospective detection baseline (RET) and the source datasets, including the suspicious-keyword list that drives the false-positive analysis.","marker":"[27]"},{"why":"Supplies the GPT-4 model evaluated in Table 1.","marker":"[1]"},{"why":"Supplies the GPT-4o model evaluated in Table 1.","marker":"[14]"},{"why":"Supplies the GLM4 model evaluated in Table 1.","marker":"[10]"},{"why":"Supplies the Doubao-Pro-32k model evaluated in Table 1.","marker":"[4]"},{"why":"Supplies the ERNIE-3.5-8k model evaluated in Table 1.","marker":"[2]"},{"why":"Supports the claim that frequent false alerts can lead users to disable or ignore the protection system, motivating the precision improvement.","marker":"[16]"}],"fun_headline_variants":["LLM real-time scam alerts hit near-perfect recall","Real-time LLM scam detection: recall near 100%, precision suffers","LLM catches scams mid-call: 98-100% recall, but at a cost","LLM real-time scam warnings: high recall, precision cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that replaying clean, complete Chinese transcripts one utterance at a time behaves like detection on a live call; since no streaming speech-to-text, latency budget, or alert mechanism is built or measured, real-world transcription errors or delay could change the reported precision and recall.","fun_headline_variants_meta":{"raw":{"variants":["LLM real-time scam alerts hit near-perfect recall","Real-time LLM scam detection: recall near 100%, precision suffers","LLM catches scams mid-call: 98-100% recall, but at a cost","LLM real-time scam warnings: high recall, precision cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001797,"raw_usage":{"total_tokens":7106,"prompt_tokens":998,"completion_tokens":6108,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":6029}},"tokens_in":614,"tokens_out":6108,"duration_ms":39881,"temperature":1.0,"reasoning_tokens":6029,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:05:47.504846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same RT and UNC prompts on transcripts produced by a live automatic-speech-recognition pipeline, with imperfect transcription, overlapping speech, and realistic pauses, and compare precision and recall against Table 1; if recall drops materially or the alert arrives after the victim would have acted, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the retrospective detection baseline (RET) and the source datasets, including the suspicious-keyword list that drives the false-positive analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Doubao-Pro-32k model evaluated in Table 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ERNIE-3.5-8k model evaluated in Table 1."}],"review_version":1}