{"id":"9c4e0438-fc44-43a4-a27c-91a33b1103c2","arxiv_id":"2412.12701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Trigger3 uses three trained triggers to route Chinese search queries among a small correction model, an LLM, and the original query, improving F0.5 on two datasets while lowering LLM coverage.","lead":"Trigger3 is a system that decides, for each search query, whether to fix it with a cheap correction model, whether to escalate it to a large language model, or whether to leave the original query alone. The paper reports better query correction scores than the individual models and existing routing frameworks, while sending far fewer queries to the expensive LLM.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset construction is the load-bearing weakness: synthetic corruptions and a 97.8% error rate mean the reported F0.5 gains and trigger thresholds may not transfer to real query traffic.","rationale":"The reader's weakest_assumption is the same as the one I find most load-bearing. The paper is an empirical systems claim; the only evidence for the claim is Tables 2–4, computed on datasets whose construction (Section 4.1) controls the error distribution. Since CT/LT/FT thresholds are fit on those same synthetic corruption patterns, there is no reason to expect them to transfer to natural query errors without an out-of-distribution test. I also considered two other possible objections: (i) the efficiency conclusion is based on LLM coverage rather than measured latency or cost, and (ii) the Section 4.2 statement that recall never decreases is contradicted by Table 2 (GECToR char recall drops from 76.30 to 74.33 on Commercial). Both are real but secondary: the coverage proxy is common in the routing literature and would matter less if performance gains were known to transfer, and the recall sentence is a summary error that does not change the reported F0.5 comparisons. The dataset issue is load-bearing because it threatens both the performance and the efficiency parts of the central claim at once. The concrete test above—a natural-error holdout with a realistic base rate—would settle it. If the authors cannot release the Commercial logs, they can at least run such a holdout internally or on QQ with naturally occurring errors; without that, the verdict should remain CONDITIONAL rather than ACCEPT. This is not a reason to reject the paper: the framework is clearly described, the ablations support the contribution of each trigger, and the internal logic is coherent. It is a reason to require the missing validation before the broad claim is accepted.","tokens_in":18513,"tokens_out":8113,"duration_ms":76090,"concrete_test":"Construct a natural-error holdout set by sampling raw user queries from a search engine without filtering by an existing correction model's confidence, then have human annotators label correctness and provide gold corrections, yielding a realistic error rate (e.g., 10–20% erroneous rather than 97.8%). Keep the trained triggers and all thresholds exactly as in the paper; evaluate Trigger3 and Margin Sampling on this set. If the F0.5 gap over Margin Sampling narrows by more than about three points, or if CT's false-positive rate on the now-majority correct queries exceeds about 10%, the claimed advantage and efficiency are artifacts of the synthetic high-error distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—Trigger3 outperforms baselines while keeping LLM calls low—rests entirely on the Section 4.1 datasets. The Commercial test set is 97.8% erroneous, with half of all samples generated by homophone substitution, near-sound replacement, adjacent transposition, and random character addition/deletion, and the other half drawn from online correction logs under a confidence filter. QQ is a public search set filtered by a language model and then corrupted with the same operations. Thus the test distribution matches the training construction, so the reported F0.5 numbers measure performance on synthetic/supervised error patterns, not on naturally occurring user errors. This is not merely an evaluation-realism quibble: CT, LT, and FT are trained with BCE on labels derived from exactly these corruptions (Section 3.6). The CT in particular is only asked to distinguish 2.2% correct queries from 97.8% corrupted ones; with roughly 14.7k Commercial test queries, that is about 324 correct examples, so the false-positive rate of CT on correct queries—the quantity that determines both correction quality and efficiency in real traffic—is essentially unconstrained. If real user errors are sparser, more diverse, or context-dependent (voice errors, missing words, domain-specific rewrites), the measured F0.5 gains and the low LLM coverage (e.g., LC 3.84 for BART) are not evidence of production behavior. The paper gives no natural-error holdout and no public release of the Commercial data, so the central claim is currently supported only on the constructed distribution. This concern is about external validity, not internal inconsistency; the experiments appear coherent on their own terms.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Trigger3, a three-stage adaptive selector for Chinese query correction. A Correction Trigger (CT) decides whether an input query should be corrected; if so, a small model rewrites it; an LLM Trigger (LT) decides whether the LLM should be invoked using the original query and the small-model rewrite; and a Fallback Trigger (FT) decides whether to return the original query when neither model produces a usable correction. The three triggers are BERT-based binary classifiers trained with cross-entropy on labels derived from edit-level agreement among the small model, the LLM, and ground-truth corrections. Experiments compare against six routing/cascading baselines on two datasets, three small models, and two LLMs, reporting character- and word-level F0.5 as well as LLM coverage. The central claim is that Trigger3 achieves the best correction quality while making fewer LLM calls than the baselines.","tokens_in":18782,"tokens_out":8365,"duration_ms":74955,"significance":"The framework is simple, model-agnostic, and the evaluation is more extensive than is typical for this task: three small models, two LLMs, two datasets, significance tests, and an explicit efficiency metric (LLM coverage). If the results transfer to real query traffic, the contribution is a practical model-selection layer that could reduce LLM cost while improving correction quality. However, transfer is not established because the test distribution is dominated by synthetic corruptions and an unnaturally high error rate. In addition, one stated conclusion (no recall decrease) is contradicted by the paper's own Table 2, and several reproducibility details need work. Overall the internal comparisons are credible, but the external validity of the central claim remains to be demonstrated.","major_comments":[{"comment":"The efficiency claim depends on CT correctly identifying correct queries, yet the evaluation gives CT almost no correct queries to classify: the Commercial test set is 97.8% erroneous, so only about 324 of its 14,737 queries are correct, and half of all Commercial errors are synthetic character-level corruptions. The CT threshold is therefore calibrated to a distribution in which almost every query is wrong; its false-positive rate on naturally correct queries, which dominate real traffic, is effectively unmeasured. Please add a natural-error holdout, or at least a correct-query-only evaluation, and report CT specificity/fall-out. Without such evidence, the reported F0.5 and LLM-coverage numbers do not support the production-efficiency claim.","section":"4.1, Table 1; 3.3"},{"comment":"The first bullet of Section 4.2 states that Trigger3 'ensur[es] no decrease in recall rate.' Table 2 directly contradicts this for GECToR on Commercial: character-level recall drops from 76.30 (small model) to 74.33 (Trigger3). The claim should be corrected or qualified (for example, 'no systematic decrease' with a list of exceptions), and the discussion should address whether this recall loss is acceptable given the F0.5 gains.","section":"4.2, Table 2"},{"comment":"The operation of each trigger is specified only as a probability exceeding 'a certain threshold' in Eqs. (4)-(6), and Algorithm 1 writes pCT = 1 as the branch condition. The paper never states the thresholds used at inference, how they were chosen, or whether they were tuned on a validation split. Because F0.5 and LLM coverage are directly controlled by these thresholds, the reported efficiency/performance trade-off is not reproducible and could be optimistic if thresholds were selected on the test set. Please report the threshold values, the selection procedure, and the resulting operating point.","section":"3.3-3.5, Algorithm 1"},{"comment":"Margin Sampling is a key baseline, but Appendix A.1 defines it as deciding based on 'the uncertainty of the first token output by the small model.' GECToR is a Seq2Edit model that outputs edit tags, not a generated first token; the paper does not explain how a first-token margin is obtained for GECToR. If the baseline was adapted in an ad hoc way, the comparison on GECToR is not meaningful. Please specify the exact uncertainty measure used for each small model.","section":"A.1, Table 2"}],"minor_comments":[{"comment":"The title and abstract use the notation Trigger$^3$, while the body and tables consistently use Trigger3; please unify the notation.","section":"Title and Abstract"},{"comment":"Figure 1 appears to show the same three example panels under labels (a), (b), and (c), which is likely a layout error; the captions describe different correction types, but the actual examples are identical.","section":"Figure 1"},{"comment":"The row label 'Cascade (LLM)' for mT5 differs from 'Cascading (LLM)' used elsewhere in the same table and in Table 5; the terminology is inconsistent.","section":"Table 2"},{"comment":"The prompt templates in Appendix C have spaces removed (for example, 'Pleasedetectandcorrecterrors'), which makes the prompts hard to read; please reformat them.","section":"Appendix C"},{"comment":"Section 4.1 says 'The training of the LLMs and the small models can be found in Section 4.1,' but the actual training details are in Appendix A.2; the cross-reference should be corrected.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the synthetic dataset construction and the proprietary source of the Commercial data make the central claim hard to verify externally. I would encourage the editor to ask for either a natural-error evaluation or a clear statement that the results are limited to the synthetic distribution. There are no other concerns about novelty or ethics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nMy read of Trigger3 is that it is a competent industrial contribution. The core idea is a three-stage router for query correction: a correction trigger skips already-correct queries, an LLM trigger decides when the small model's rewrite is insufficient, and a fallback trigger restores the original query when neither model looks reliable. That combination is not present in the routing/cascading baselines the paper cites, and it is well matched to the correction task. The empirical work is broad: two datasets, three small models, two LLMs, ablations, significance tests. On the paper's own test sets, Trigger3 consistently improves F0.5 over the strongest baselines (e.g., GECToR from 67.48 to 74.60 Char-F0.5 on Commercial) while cutting LLM coverage.\n\nThe soft spot is the dataset construction, and it is load-bearing. Commercial is 97.8% erroneous, with half the samples synthetic character corruptions; QQ is a public search set corrupted the same way. So the test distribution is the training distribution. There is no natural-error holdout. The correction trigger is trained to distinguish 2.2% correct from 97.8% incorrect queries, so its false-positive rate on correct queries—the thing that matters when real traffic is mostly correct—is essentially unconstrained. The reported gains and low LLM coverage may not transfer to production.\n\nThere is also a concrete internal inconsistency: the paper claims \"no decrease in recall rate,\" but Table 2 shows GECToR's character recall dropping from 76.30 to 74.33 on Commercial under Trigger3. That should be fixed. And efficiency is measured only by LLM coverage, not measured latency or cost; it is a reasonable proxy but not the same thing.\n\nThe math and training procedure are straightforward, and the paper is honest about the construction details. No load-bearing flaw beyond the external-validity issue. Who gets value: practitioners building correction pipelines, and researchers working on model routing for text editing. It deserves a serious referee; the framework is coherent and the experiments are extensive, but the revision should add a natural-error evaluation or at least an explicit discussion of the error-rate mismatch, plus correct the recall statement.","headline":"A sensible three-trigger router for query correction with broad experiments, but the synthetic 97.8%-error datasets leave external validity unproven.","tokens_in":19412,"tokens_out":2599,"would_cite":true,"duration_ms":23292,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trigger3 claims that a three-trigger cascade—small model first, LLM only when needed, original query as fallback—outperforms query-correction baselines while cutting LLM calls.","keywords":["query correction","model selection","large-small model collaboration","LLM cascade","trigger classifiers","search engine","Chinese query correction","efficiency"],"falsifier":"Run Trigger3 on a held-out sample of naturally occurring user queries with human-labeled corrections, including correct queries and domain-specific terms, and compare Char-F0.5 and LLM coverage against Margin Sampling; if the F0.5 advantage shrinks substantially or LLM coverage rises above the synthetic-test values, the triggers are tuned to the corruption pipeline rather than to real errors.","tokens_in":18279,"feed_emoji":"🔍","tokens_out":6430,"duration_ms":49380,"temperature":0.7,"pith_summary":"The paper claims that search-query correction can be improved by treating it as an adaptive model-selection problem: a small correction model handles routine typos, a large language model is called only when the small model's rewrite looks inadequate, and a final check can fall back to the original query rather than accept a bad rewrite. This three-trigger cascade, named Trigger3, is trained with binary BERT classifiers and is model-agnostic; the paper tests it with three small models and two LLMs on Chinese search datasets. The reported result is higher character-level F0.5 than all correction baselines while the fraction of queries sent to the LLM stays low. A sympathetic reader would take away that the expensive LLM should be a targeted second opinion, not a default corrector, and that returning the original query is a legitimate correction outcome.","feed_headline":"Three triggers lift query correction accuracy while cutting LLM calls","feed_subtitle":"Cascade of small model, LLM, and original-query fallback beats correction baselines at lower LLM cost.","key_machinery":"The load-bearing object is the adaptive model selector made of three binary classifiers, each a BERT model using the [CLS] representation: CT scores P(Incorrect|x), LT scores P(Invoke LLM | x, ysmall) with [SEP] separating the two texts, and FT scores P(Return x | x, yc). Training labels for LT and FT are derived from character-edit indicators (TP, FP, FN) comparing the small model's edits and the LLM's edits against the gold query: LT positive when the small model misses or errs but the LLM does not, FT positive when neither model produces a correct edit. These trigger labels turn the model-selection question into three binary classification problems, which is what makes the framework trainable and model-agnostic.","core_discovery":"The central claim is that Trigger3, by chaining a Correction Trigger (CT), an LLM Trigger (LT), and a Fallback Trigger (FT), outperforms correction baselines while maintaining efficiency. Given an original query x, CT decides whether correction is needed; if so, the small model produces ysmall; LT, looking at (x, ysmall), decides whether the LLM is needed; and FT, looking at (x, yc), decides whether to return the original query. In the main tables, the best Trigger3 configuration reaches Char-F0.5 74.60 on the Commercial dataset with GECToR, compared to 67.48 for Margin Sampling, and does so with LLM coverage 32.09 versus 45.63. The paper interprets this as evidence that a cascade with correct-query filtering, targeted LLM invocation, and fallback to the original query is a better large-small model collaboration than routing or pure cascading.","pith_inferences":["The paper does not report a sweep of CT/LT/FT thresholds; tuning them per query class or per traffic mix is a natural next step and could shift the efficiency-quality tradeoff further.","The trigger labels require gold corrections for training; in a live system without gold labels, the same edit-indicator scheme could be bootstrapped from high-confidence online correction logs.","The framework's logic is generic to any rewrite task where a cheap model and an expensive model disagree, such as style transfer or query expansion; whether it transfers is an open question."],"forward_implications":["Production search engines can add Trigger3 as a drop-in layer on top of an existing small correction model, sending only a minority of queries to an LLM.","The LT threshold gives operators a direct control over LLM coverage, letting them trade a small amount of F0.5 for lower serving cost.","Formalizing 'return the original query' as a decision reduces the damage from over-correction, which the paper observes in both small models and LLMs.","Because the trigger labels are defined purely by edit-level TP/FP/FN, the same recipe can be re-trained for new small-model/LLM pairs without redesigning the framework."],"supporting_citations":[{"why":"Supplies GECToR-Chinese, the Seq2Edit small model used in the main experiments, and the ChERRANT scorer used for evaluation.","marker":"(Zhang et al. 2022)"},{"why":"Provides the BERT architecture and [CLS] representation used to build the three trigger classifiers.","marker":"(Devlin et al. 2019)"},{"why":"Provides Qwen1.5-7B-Chat, the primary LLM corrected against in the main results.","marker":"(Bai et al. 2023)"},{"why":"Defines Margin Sampling, the uncertainty-based cascading baseline that Trigger3 is compared against for quality and LLM coverage.","marker":"(Ram´ırez, Birch, and Titov 2024)"},{"why":"Defines HybridLLM, a routing baseline, and the LLM-coverage efficiency metric used in the comparison.","marker":"(Ding et al. 2024)"},{"why":"Justifies fine-tuning LLMs on 1,000 samples to improve correction performance.","marker":"(Fan et al. 2023)"},{"why":"Provides the query-filtering and corruption approach used to build the QQ dataset.","marker":"(Ye et al. 2023)"}],"fun_headline_variants":["Adaptive trigger cascade cuts LLM calls while improving query correction","Three-trigger cascade: small model first, LLM only when needed","Trigger^3: adaptive selector picks between small model and LLM","Query correction cascade slashes LLM calls with three triggers","Adaptive model selector: three triggers refine query correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation datasets—a commercial set that is 97.8% erroneous and half synthetic, plus a public set corrupted the same way—must resemble the queries a real search engine sees; if real user errors are rarer or differently distributed, the measured F0.5 gains and trigger thresholds will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive trigger cascade cuts LLM calls while improving query correction","Three-trigger cascade: small model first, LLM only when needed","Trigger^3: adaptive selector picks between small model and LLM","Query correction cascade slashes LLM calls with three triggers","Adaptive model selector: three triggers refine query correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000471,"raw_usage":{"total_tokens":2349,"prompt_tokens":957,"completion_tokens":1392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1315}},"tokens_in":573,"tokens_out":1392,"duration_ms":10008,"temperature":1.0,"reasoning_tokens":1315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:49:12.880091+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Trigger3 on a held-out sample of naturally occurring user queries with human-labeled corrections, including correct queries and domain-specific terms, and compare Char-F0.5 and LLM coverage against Margin Sampling; if the F0.5 advantage shrinks substantially or LLM coverage rises above the synthetic-test values, the triggers are tuned to the corruption pipeline rather than to real errors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GECToR-Chinese, the Seq2Edit small model used in the main experiments, and the ChERRANT scorer used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines HybridLLM, a routing baseline, and the LLM-coverage efficiency metric used in the comparison."}],"review_version":1}