{"id":"93263e9f-6776-4e86-a400-d9d77fc2f120","arxiv_id":"2505.11176","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A data-centric system for banking search recommendation combines synthetic query generation, intent disambiguation, and gap analysis, with synthetic data matching real data on Clinc150 but degrading on Banking77 and proprietary data.","lead":"This paper describes a production system at JPMorganChase that generates synthetic banking queries, refines broad intents into subintents, and mines unlabeled queries for missing intents, all to improve search recommendations. It reports mixed results: synthetic queries match real ones on a general benchmark but significantly underperform on banking-specific datasets, and some claimed benefits are not directly supported.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cold-start claim lacks a no-data baseline; synthetic replacement is compared only against full real data, so the magnitude of the claimed alleviation is unsupported.","rationale":"The paper's central contribution is the claim that synthetic query generation effectively alleviates the cold-start problem. The reader identified the weakest assumption as the representativeness of synthetic queries, supported by the missing zero-data baseline. My stress-test confirms this is the most load-bearing concern: the extrinsic evaluation in Section 5.1 only compares synthetic-replaced training data against full real data. The no-data condition, which defines the cold-start problem, is never measured. Because the generator uses 10 real few-shot examples per target label, the experiment also does not test the zero-data scenario implied by the introduction's 'almost immediately' language. Every other issue—the inequitable gap-analysis baseline, the disambiguation evaluation, and the lack of deployment metrics—is secondary to the headline cold-start claim. The paper has genuine strengths: honest discussion of synthetic data limitations, public dataset experiments, and a deployed system with human-in-the-loop review. The concern is addressable with a straightforward additional baseline, so it does not warrant rejection. The reader's CONDITIONAL verdict is appropriate; adding the no-data and zero-example baselines would either confirm or refute the central claim, and the paper should be published only after that evidence is supplied.","tokens_in":17130,"tokens_out":6640,"duration_ms":61632,"concrete_test":"Run the same masked-intent experiment with two additional baselines: (i) a 'no-data' classifier trained only on the 75% unmasked intents (masked classes absent from the label set); (ii) a zero-example variant of Synthetic Query Generation using only the intent name/description, with no few-shot queries. Evaluate macro-F1 and per-intent F1 for the masked intents on the real test set. If the synthetic-replacement model's per-intent F1 on masked intents is substantially above the no-data baseline, and ideally the zero-example variant's F1 is also reported, the cold-start claim is supported. Report these numbers alongside Figure 4.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—synthetic query generation effectively alleviates the cold-start problem—is not directly tested. Section 5.1/Figure 4 compares a model trained with 25% of intent labels' real queries replaced by 100 synthetic queries per intent against a model trained on full real data. The actual cold-start condition (no training queries for those intents) is never evaluated. A standard classifier trained without examples for a class cannot predict it, so the no-data baseline would be far lower; the paper's inference that synthetic data 'alleviates' cold start rests entirely on this uncomputed comparison. Additionally, the generation prompt draws 10 real few-shot examples from each target label, so the experiment validates few-shot cold start, not the zero-data 'almost immediately' scenario described in Section 1. Without the no-data baseline, the 0.928→0.883 Banking77 drop cannot be interpreted as success or failure; the reader cannot tell how much of the cold-start gap synthetic data closes. This is load-bearing because the abstract's cold-start statement is the paper's primary contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a production system for enhancing search-query datasets for intent-based recommendation in digital banking, with three modules: synthetic query generation, intent disambiguation, and intent gap analysis. The synthetic query module replaces real training queries for 25% of intent labels with LLM-generated queries and compares classifier macro-F1 against a full-real-data baseline on Clinc150, Banking77, and the proprietary Chat146. The disambiguation module uses an LLM to refine broad intents into subintents and evaluates against expert reannotations. The gap analysis module proposes, judges, and merges new intents from unlabeled queries, evaluated by rediscovering masked intents. The paper also reports deployment experience on the proprietary Chat186 dataset.","tokens_in":17300,"tokens_out":3575,"duration_ms":35919,"significance":"If the central cold-start claim were validated, the system would have clear practical value for rapidly onboarding new banking products without waiting for real user queries to accumulate. The paper has concrete strengths: it reports a deployed system, provides full prompts and ablation studies, and evaluates intent disambiguation against expert reannotations and gap analysis against masked intents, which are independent ground truths. The extrinsic evaluation using real test sets is the right paradigm. However, the central claim about cold-start alleviation is not directly supported by the experiments as designed, and the gap-analysis baseline is substantially handicapped.","major_comments":[{"comment":"The experiment described as simulating a 'cold start' replaces all real training queries for 25% of intent labels with synthetic queries, but the only comparison is against a model trained on the full real dataset. There is no baseline in which those intents have zero training queries, or in which they are trained with only the 10 few-shot examples used in the generation prompt. Without such a baseline, the observed drops (Banking77: 0.928 to 0.883; Chat146: 0.935 to 0.892) cannot be interpreted as alleviating the cold-start problem; the reader cannot tell how much of the cold-start gap the synthetic data actually closes. This is load-bearing because the abstract's claim that the approach 'effectively alleviates the cold start problem' rests entirely on this comparison. Please add a no-data (or few-shot-only) baseline and report the relative improvement over it.","section":"Section 5.1, Figure 4"},{"comment":"The synthetic generation prompt uses 10 randomly chosen few-shot examples from the target label. The introduction and abstract describe a zero-data scenario where new products are recommended 'almost immediately' with 'limited historical data'. The experiment therefore validates few-shot cold start, not the zero-data scenario claimed. This mismatch should be acknowledged explicitly, and either the claim should be weakened to few-shot cold start or the experiment should include a condition with no target-label examples in the generation prompt.","section":"Section 3.4 and Section 1"},{"comment":"The gap-analysis evaluation compares the proposed method against a baseline that is instructed to generate only 10 new intents 'due to context window limitations', while the proposed method can produce up to 100 proposals. Recovery rates are reported at top 50 and top 100 proposals for the proposed method, but the baseline cannot reach those proposal counts, making the comparison lopsided. The baseline should be allowed to generate a comparable number of proposals (e.g., through repeated calls) or the comparison should be restricted to N=10, where both methods are feasible. Without this, the claim that the method outperforms the baseline is not supported by the data shown.","section":"Section 3.6, Figure 6"}],"minor_comments":[{"comment":"The statistical comparison uses Welch's two-sample t-test on per-intent F1 scores, but the control and synthetic conditions are evaluated on the same set of intents, creating paired data. A paired test or a confidence interval on per-intent differences would be more appropriate; the unpaired test may understate significance, which is particularly relevant to the 'no statistically significant difference' claim for Clinc150.","section":"Section 3.3"},{"comment":"The error bars and standard deviations are not clearly defined. Please state whether the reported standard deviations reflect variability across intents, across random seeds, or across repeated train/test splits, and describe the number of repetitions.","section":"Figure 4 and Figure 6"},{"comment":"The abstract states that the deployed system 'demonstrates significant improvements in recommendation precision and operation agility', but the deployment section provides no quantitative evidence for recommendation precision beyond the Chat186 F1 comparisons. Please either add deployment metrics or soften this claim.","section":"Abstract and Section 4"},{"comment":"The prompts contain repeated typos of 'YAML' as 'Y AML' (e.g., 'Return 5 user queries in Y AML format'). These should be corrected for readability.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is industry-oriented and reports on a proprietary dataset (Chat146) that is not released, which limits external reproducibility but is not itself a blocker. The main concern is the missing cold-start baseline and the handicapped gap-analysis baseline; both are correctable within the scope of the manuscript. I would encourage the editor to require the authors to address these issues before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: this is a solid applied paper from JPMorgan about using LLMs to generate, disambiguate, and gap-analyze search query data for banking recommenders. The cold-start claim is the weakest link—not because it's false, but because the experiments never compare against a true zero-data baseline, so the reader can't tell how much of the gap synthetic data actually closes.\n\nWhat's actually new: the specific integration of three modules (sequential-conditioning synthetic generation, LLM reannotation into subintents, agentic gap analysis) and a real deployment at Chase. That's worth something. The disambiguation evaluation, with F1 0.863 against expert reannotations over 6k queries, is a genuine empirical result. The gap analysis recovery rates (up to 71% on Banking77) are also measured honestly with repeated LLM judging. The authors include ablations on batch size, temperature, few-shot counts, and they openly discuss the failure modes of their synthetic data—short, less specific, overlapping queries. The intrinsic metrics in Table 1 are a nice addition.\n\nSoft spots, in proportion: The main experiment (Section 5.1) masks 25% of intent labels and replaces them with 100 synthetic queries each, then compares to a model trained on full real data. That's a replacement test, not a cold-start test. The actual cold-start condition—zero training queries for those intents—is never run. And because generation uses 10 real few-shot examples per target label, this is few-shot, not the \"almost immediately\" zero-data scenario in the Introduction. The result is that the abstract's \"effectively alleviates\" is unsupported as stated. The authors could fix this with one extra baseline: train on the remaining 75% of labels only, and show synthetic data closes the gap. The gap analysis baseline also only generates 10 intents due to context limits, so the 71% vs baseline comparison isn't apples-to-apples. Deployment benefits are described qualitatively; no metrics like conversion or precision before/after.\n\nNone of this is fatal if you read the paper as an engineering report. The methods are sound, the evaluations are honest, and the limitations are acknowledged—the paper itself says synthetic data \"exhibits limitations\" and the gap analysis \"has yet to deliver direct results.\" For an applied IR audience, it's a useful case study.\n\nI'd send it to peer review, but I'd ask the authors to add the no-data baseline and strengthen the gap baseline before acceptance. Not a desk reject.\n\nRecommendation: engage with it; a serious referee can push on the cold-start evidence and the paper will come back stronger.","headline":"A deployed, data-centric LLM system for banking query data with a solid disambiguation evaluation, but the central cold-start claim lacks the zero-data baseline needed to support it.","tokens_in":17821,"tokens_out":2269,"would_cite":false,"duration_ms":21858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A production data pipeline uses LLMs to ease the cold-start problem for banking recommendations.","keywords":["synthetic data","cold start","intent classification","recommendation systems","large language models","intent disambiguation","intent discovery","banking"],"falsifier":"For a set of intents with no real training queries, train one classifier on 100 synthetic queries per intent and another on the first 50 real queries that arrive; if the real-data model beats the synthetic model by a large margin (or vice versa), it would settle when synthetic data actually substitutes for real data. A per-intent comparison on Banking77, where synthetic data already drops F1 by 0.44 on 'top up by card', would show which intents fail and which succeed.","tokens_in":16897,"feed_emoji":"🏦","tokens_out":4285,"duration_ms":34651,"temperature":0.7,"pith_summary":"This paper presents a production system for keeping the query data behind an intent-based recommendation engine in digital banking continuously up to date. It claims that three automated modules—synthetic query generation, intent disambiguation, and intent gap analysis—can reduce the cold-start problem for new banking products, refine overly broad intent labels, and surface emerging customer needs from unlabeled queries. A sympathetic reader would care because the system is deployed in a live banking environment and reports concrete gains in recommendation precision and operational agility.","feed_headline":"Synthetic queries ease cold-start in banking search","feed_subtitle":"Three automated modules keep recommendation intents current; synthetic data replaces real queries for new products.","key_machinery":"The load-bearing mechanism is the three-module pipeline centered on sequential conditioning in synthetic query generation: each new query in a batch is explicitly instructed to differ from previously generated ones, producing 100 diverse queries per intent. Intent disambiguation uses a single LLM annotator with few-shot examples per subintent plus a default label for ambiguous cases. Intent gap analysis is an agentic propose-judge-merge loop that proposes candidate intents from batches of unlabeled queries, judges them against the existing taxonomy, and iteratively merges duplicates ranked by query support. The extrinsic evaluation protocol—masking 25% of intents and training classifiers on synthetic replacements—is what operationalizes the cold-start claim.","core_discovery":"The central claim is that a data-centric pipeline driven by large language models can replace slow, manual dataset maintenance for intent-based recommendation. In the cold-start simulation, where 25% of intent labels have their real training queries fully replaced by 100 LLM-generated queries per intent, the classifier's macro F1 on real test queries is statistically unchanged for Clinc150 (0.953 versus 0.944) but drops significantly for Banking77 (0.928 to 0.883) and the proprietary banking dataset Chat146 (0.935 to 0.892). The paper argues these drops are acceptable and that synthetic data mitigates cold start, while qualitative failure analysis attributes the gaps to synthetic queries being shorter, less specific, and overlapping with sibling intents. For intent disambiguation, the LLM annotator matches human reannotations with F1 0.863 ± 0.127; for gap analysis, the propose-judge-merge pipeline recovers up to 71% of masked intents among its top 100 proposals.","pith_inferences":["A stronger test of the cold-start claim would compare synthetic-only training against a small amount of real data (e.g., 10-50 examples per intent); the paper only compares against the full real-data model, so the marginal value of synthetic data over a tiny real sample is untested.","The intrinsic metrics suggest a concrete fix: injecting realistic entity inventories (merchant names, product IDs) into generation prompts could narrow the specificity gap and potentially close the Banking77/Chat146 F1 deficit.","Because recovery rates plateau around 50-70 proposals, the gap analysis module is best used as a prioritization filter rather than an exhaustive discovery tool; low-support intents are unlikely to be worth product investment.","The disambiguation F1 of 0.863 with known category disagreements (e.g., debit versus credit card) implies that a human-in-the-loop should stay in the pipeline for ambiguous queries, especially those requiring product-specific business knowledge."],"forward_implications":["If synthetic data is accepted as a cold-start bridge, new banking products can be onboarded for recommendation within days rather than after months of accumulated real queries.","Broad intents like 'Open Account' can be split into precise subintents without a full manual reannotation campaign, improving recommendation mapping.","Intent gap analysis gives product teams a ranked list of unserved customer needs, so roadmap decisions can be driven by query support rather than anecdote.","The same pipeline could transfer to other domains with intent-based search, such as e-commerce or customer support, wherever a taxonomy needs to stay current.","The reported F1 drop on Banking77 and Chat146 implies that synthetic data alone is not yet equivalent to real data for domain-specific, fine-grained intents; the paper frames this as a limitation to engineer around."],"supporting_citations":[{"why":"Supplies the Clinc150 dataset, one of the three benchmarks where synthetic replacement shows no statistically significant difference.","marker":"Larson et al. 2019"},{"why":"Supplies the Banking77 dataset, the domain-specific benchmark where synthetic replacement significantly drops F1 and drives the failure analysis.","marker":"Casanueva et al. 2020"},{"why":"Provides prior evidence that LLM synthetic data helps distinct intents but fails on semantically close ones, directly supporting the paper's overlap findings.","marker":"Sahu et al. 2022"},{"why":"Establishes the extrinsic evaluation design of training on synthetic and testing on real data, which the cold-start simulation follows.","marker":"Li et al. 2023b"},{"why":"Supplies the compression-ratio and compression-ratio-POS metrics used in the intrinsic evaluation of synthetic query quality.","marker":"Shaib et al. 2024"},{"why":"Supplies DistilBERT, the classifier backbone on which all extrinsic F1 comparisons are run.","marker":"Sanh et al. 2020"}],"fun_headline_variants":["Data-centric AI pipeline scales query datasets for banking","Synthetic queries mitigate cold start in banking search","LLM intent disambiguation hits 86.3% F1 in production","Gap analysis recovers 71% of latent intents in search","Proactive data enhancement powers modern recommendation systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cold-start benefit rests on the assumption that LLM-generated queries are representative enough of real banking queries to train intent classifiers that will perform well on real user input; the paper's own metrics show synthetic queries are shorter, less specific, and overlap with related intents, and the claim is not tested against a zero-data no-synthetic baseline.","fun_headline_variants_meta":{"raw":{"variants":["Data-centric AI pipeline scales query datasets for banking","Synthetic queries mitigate cold start in banking search","LLM intent disambiguation hits 86.3% F1 in production","Gap analysis recovers 71% of latent intents in search","Proactive data enhancement powers modern recommendation systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001104,"raw_usage":{"total_tokens":4645,"prompt_tokens":1027,"completion_tokens":3618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":3537}},"tokens_in":643,"tokens_out":3618,"duration_ms":27429,"temperature":1.0,"reasoning_tokens":3537,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:55:29.797997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a set of intents with no real training queries, train one classifier on 100 synthetic queries per intent and another on the first 50 real queries that arrive; if the real-data model beats the synthetic model by a large margin (or vice versa), it would settle when synthetic data actually substitutes for real data. A per-intent comparison on Banking77, where synthetic data already drops F1 by 0.44 on 'top up by card', would show which intents fail and which succeed.","supporting_citations":[],"review_version":1}