Enhancing and Scaling Search Query Datasets for Recommendation Systems

Aaron Rodrigues; Azzam Naeem; Mahmood Hegazy

REVIEW 3 major objections 4 minor 1 cited by

Enhancing and Scaling Search Query Datasets for Recommendation Systems

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A production data pipeline uses LLMs to ease the cold-start problem for banking recommendations.

desk verdict A deployed, data-centric LLM system for banking query data with a solid disambiguation evaluation, but the central cold-start claim lacks the zero-data baseline needed to support it. read the letter →

arxiv 2505.11176 v2 pith:VKFW7A4L submitted 2025-05-16 cs.IR cs.AI

Aaron Rodrigues , Mahmood Hegazy , Azzam Naeem This is my paper · ORCID

classification cs.IRcs.AI

keywords syntheticdatacoldstartintentclassificationrecommendationsystemslargelanguagemodelsdisambiguationdiscoverybanking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents a production system for keeping the query data behind an intent-based recommendation engine in digital banking continuously up to date. It claims that three automated modules—synthetic query generation, intent disambiguation, and intent gap analysis—can reduce the cold-start problem for new banking products, refine overly broad intent labels, and surface emerging customer needs from unlabeled queries. A sympathetic reader would care because the system is deployed in a live banking environment and reports concrete gains in recommendation precision and operational agility.

What carries the argument

The load-bearing mechanism is the three-module pipeline centered on sequential conditioning in synthetic query generation: each new query in a batch is explicitly instructed to differ from previously generated ones, producing 100 diverse queries per intent. Intent disambiguation uses a single LLM annotator with few-shot examples per subintent plus a default label for ambiguous cases. Intent gap analysis is an agentic propose-judge-merge loop that proposes candidate intents from batches of unlabeled queries, judges them against the existing taxonomy, and iteratively merges duplicates ranked by query support. The extrinsic evaluation protocol—masking 25% of intents and training classifiers on synthetic replacements—is what operationalizes the cold-start claim.

What would settle it

For a set of intents with no real training queries, train one classifier on 100 synthetic queries per intent and another on the first 50 real queries that arrive; if the real-data model beats the synthetic model by a large margin (or vice versa), it would settle when synthetic data actually substitutes for real data. A per-intent comparison on Banking77, where synthetic data already drops F1 by 0.44 on 'top up by card', would show which intents fail and which succeed.

Watch

Extended reading notes

Core claim

The central claim is that a data-centric pipeline driven by large language models can replace slow, manual dataset maintenance for intent-based recommendation. In the cold-start simulation, where 25% of intent labels have their real training queries fully replaced by 100 LLM-generated queries per intent, the classifier's macro F1 on real test queries is statistically unchanged for Clinc150 (0.953 versus 0.944) but drops significantly for Banking77 (0.928 to 0.883) and the proprietary banking dataset Chat146 (0.935 to 0.892). The paper argues these drops are acceptable and that synthetic data mitigates cold start, while qualitative failure analysis attributes the gaps to synthetic queries being shorter, less specific, and overlapping with sibling intents. For intent disambiguation, the LLM annotator matches human reannotations with F1 0.863 ± 0.127; for gap analysis, the propose-judge-merge pipeline recovers up to 71% of masked intents among its top 100 proposals.

Load-bearing premise

The cold-start benefit rests on the assumption that LLM-generated queries are representative enough of real banking queries to train intent classifiers that will perform well on real user input; the paper's own metrics show synthetic queries are shorter, less specific, and overlap with related intents, and the claim is not tested against a zero-data no-synthetic baseline.

Editorial extensions

If this is right

If synthetic data is accepted as a cold-start bridge, new banking products can be onboarded for recommendation within days rather than after months of accumulated real queries.
Broad intents like 'Open Account' can be split into precise subintents without a full manual reannotation campaign, improving recommendation mapping.
Intent gap analysis gives product teams a ranked list of unserved customer needs, so roadmap decisions can be driven by query support rather than anecdote.
The same pipeline could transfer to other domains with intent-based search, such as e-commerce or customer support, wherever a taxonomy needs to stay current.
The reported F1 drop on Banking77 and Chat146 implies that synthetic data alone is not yet equivalent to real data for domain-specific, fine-grained intents; the paper frames this as a limitation to engineer around.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A stronger test of the cold-start claim would compare synthetic-only training against a small amount of real data (e.g., 10-50 examples per intent); the paper only compares against the full real-data model, so the marginal value of synthetic data over a tiny real sample is untested.
The intrinsic metrics suggest a concrete fix: injecting realistic entity inventories (merchant names, product IDs) into generation prompts could narrow the specificity gap and potentially close the Banking77/Chat146 F1 deficit.
Because recovery rates plateau around 50-70 proposals, the gap analysis module is best used as a prioritization filter rather than an exhaustive discovery tool; low-support intents are unlikely to be worth product investment.
The disambiguation F1 of 0.863 with known category disagreements (e.g., debit versus credit card) implies that a human-in-the-loop should stay in the pipeline for ambiguous queries, especially those requiring product-specific business knowledge.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The punchline: this is a solid applied paper from JPMorgan about using LLMs to generate, disambiguate, and gap-analyze search query data for banking recommenders. The cold-start claim is the weakest link—not because it's false, but because the experiments never compare against a true zero-data baseline, so the reader can't tell how much of the gap synthetic data actually closes.

What's actually new: the specific integration of three modules (sequential-conditioning synthetic generation, LLM reannotation into subintents, agentic gap analysis) and a real deployment at Chase. That's worth something. The disambiguation evaluation, with F1 0.863 against expert reannotations over 6k queries, is a genuine empirical result. The gap analysis recovery rates (up to 71% on Banking77) are also measured honestly with repeated LLM judging. The authors include ablations on batch size, temperature, few-shot counts, and they openly discuss the failure modes of their synthetic data—short, less specific, overlapping queries. The intrinsic metrics in Table 1 are a nice addition.

Soft spots, in proportion: The main experiment (Section 5.1) masks 25% of intent labels and replaces them with 100 synthetic queries each, then compares to a model trained on full real data. That's a replacement test, not a cold-start test. The actual cold-start condition—zero training queries for those intents—is never run. And because generation uses 10 real few-shot examples per target label, this is few-shot, not the "almost immediately" zero-data scenario in the Introduction. The result is that the abstract's "effectively alleviates" is unsupported as stated. The authors could fix this with one extra baseline: train on the remaining 75% of labels only, and show synthetic data closes the gap. The gap analysis baseline also only generates 10 intents due to context limits, so the 71% vs baseline comparison isn't apples-to-apples. Deployment benefits are described qualitatively; no metrics like conversion or precision before/after.

None of this is fatal if you read the paper as an engineering report. The methods are sound, the evaluations are honest, and the limitations are acknowledged—the paper itself says synthetic data "exhibits limitations" and the gap analysis "has yet to deliver direct results." For an applied IR audience, it's a useful case study.

I'd send it to peer review, but I'd ask the authors to add the no-data baseline and strengthen the gap baseline before acceptance. Not a desk reject.

Recommendation: engage with it; a serious referee can push on the cold-start evidence and the paper will come back stronger.

Referee Report

3 major / 4 minor

Summary. The paper describes a production system for enhancing search-query datasets for intent-based recommendation in digital banking, with three modules: synthetic query generation, intent disambiguation, and intent gap analysis. The synthetic query module replaces real training queries for 25% of intent labels with LLM-generated queries and compares classifier macro-F1 against a full-real-data baseline on Clinc150, Banking77, and the proprietary Chat146. The disambiguation module uses an LLM to refine broad intents into subintents and evaluates against expert reannotations. The gap analysis module proposes, judges, and merges new intents from unlabeled queries, evaluated by rediscovering masked intents. The paper also reports deployment experience on the proprietary Chat186 dataset.

Significance. If the central cold-start claim were validated, the system would have clear practical value for rapidly onboarding new banking products without waiting for real user queries to accumulate. The paper has concrete strengths: it reports a deployed system, provides full prompts and ablation studies, and evaluates intent disambiguation against expert reannotations and gap analysis against masked intents, which are independent ground truths. The extrinsic evaluation using real test sets is the right paradigm. However, the central claim about cold-start alleviation is not directly supported by the experiments as designed, and the gap-analysis baseline is substantially handicapped.

major comments (3)

[Section 5.1, Figure 4] The experiment described as simulating a 'cold start' replaces all real training queries for 25% of intent labels with synthetic queries, but the only comparison is against a model trained on the full real dataset. There is no baseline in which those intents have zero training queries, or in which they are trained with only the 10 few-shot examples used in the generation prompt. Without such a baseline, the observed drops (Banking77: 0.928 to 0.883; Chat146: 0.935 to 0.892) cannot be interpreted as alleviating the cold-start problem; the reader cannot tell how much of the cold-start gap the synthetic data actually closes. This is load-bearing because the abstract's claim that the approach 'effectively alleviates the cold start problem' rests entirely on this comparison. Please add a no-data (or few-shot-only) baseline and report the relative improvement over it.
[Section 3.4 and Section 1] The synthetic generation prompt uses 10 randomly chosen few-shot examples from the target label. The introduction and abstract describe a zero-data scenario where new products are recommended 'almost immediately' with 'limited historical data'. The experiment therefore validates few-shot cold start, not the zero-data scenario claimed. This mismatch should be acknowledged explicitly, and either the claim should be weakened to few-shot cold start or the experiment should include a condition with no target-label examples in the generation prompt.
[Section 3.6, Figure 6] The gap-analysis evaluation compares the proposed method against a baseline that is instructed to generate only 10 new intents 'due to context window limitations', while the proposed method can produce up to 100 proposals. Recovery rates are reported at top 50 and top 100 proposals for the proposed method, but the baseline cannot reach those proposal counts, making the comparison lopsided. The baseline should be allowed to generate a comparable number of proposals (e.g., through repeated calls) or the comparison should be restricted to N=10, where both methods are feasible. Without this, the claim that the method outperforms the baseline is not supported by the data shown.

minor comments (4)

[Section 3.3] The statistical comparison uses Welch's two-sample t-test on per-intent F1 scores, but the control and synthetic conditions are evaluated on the same set of intents, creating paired data. A paired test or a confidence interval on per-intent differences would be more appropriate; the unpaired test may understate significance, which is particularly relevant to the 'no statistically significant difference' claim for Clinc150.
[Figure 4 and Figure 6] The error bars and standard deviations are not clearly defined. Please state whether the reported standard deviations reflect variability across intents, across random seeds, or across repeated train/test splits, and describe the number of repetitions.
[Abstract and Section 4] The abstract states that the deployed system 'demonstrates significant improvements in recommendation precision and operation agility', but the deployment section provides no quantitative evidence for recommendation precision beyond the Chat186 F1 comparisons. Please either add deployment metrics or soften this claim.
[Appendix A.3] The prompts contain repeated typos of 'YAML' as 'Y AML' (e.g., 'Return 5 user queries in Y AML format'). These should be corrected for readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity; synthetic-data and gap-analysis claims are checked against independent real test sets and masked-intent ground truth, with only minor self-referential caveats.

full rationale

The paper's central evaluations are externally grounded rather than self-referential. Synthetic Query Generation is judged extrinsically by training DistilBERT classifiers on synthetic queries and testing on held-out real queries, so the reported F1 differences are not determined by the generator by construction. Intent Gap Analysis is evaluated by masking 25% of the known intents and measuring rediscovery against the original taxonomy; the LLM judge is separate from the proposer and is checked against known labels, so this is an independent recovery test even though the judge is an LLM. The main self-referential elements are minor: the disambiguation truth set was amended in part after seeing LLM annotations ('over a hundred cases emerged where the LLM's annotations were more accurate than the original human annotations. This led to amending the truth set', Section 5.2), which could somewhat inflate the reported 0.863 F1 against 'expert reannotations', but the evaluation still rests on human review and qualitative analysis rather than on a derivation that reduces to its inputs. The single self-citation (Hegazy, Rodrigues, and Naeem 2025) is a future-work pointer, not load-bearing. The cold-start claim is under-supported because no true zero-data baseline is run, but that is a validity gap, not circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or algorithmic entities are introduced. The free parameters are experimental settings such as temperature, batch size, and masking ratio. The axioms are domain assumptions about LLM output quality and evaluation validity, which are partially contradicted by the paper's own findings.

free parameters (6)

LLM temperature = 0.5
Chosen based on experiments; highest performance at 0.5 (Figure S5).
Synthetic queries per intent = 100
Number generated per intent; chosen without reported optimization.
Few-shot example counts = 10 random + 10 from target label
Selection strategy for synthetic generation prompts; not optimized.
Generation batch size = 5 queries per batch
Sequential conditioning batch; experiments show 5 is better than 1.
Intent masking ratio = 25% of intents
Fraction of intents replaced to simulate cold start; chosen arbitrarily.
Merge iterations = 2000
Number of intent merging iterations in gap analysis; recovery plateaus after 1000.

assumptions (4)

domain assumption GPT-4o outputs are representative of real user queries
Central to synthetic query generation; the paper's own intrinsic evaluation shows systematic differences in length, specificity, and entity usage.
domain assumption LLM judge semantic matching is accurate for recovery evaluation
The recovery rate in gap analysis is determined by an LLM judge comparing masked intents to proposed intents; accuracy is not validated against human judgments.
domain assumption Masking 25% of intents simulates cold start
The extrinsic evaluation assumes full replacement of real queries with synthetic ones is a valid proxy for cold start, but no zero-data baseline is measured.
standard math Welch's t-test assumptions hold for per-intent F1 comparisons
The paper compares per-intent F1 scores with Welch's t-test without discussing independence or normality assumptions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing and Scaling Search Query Datasets for Recommendation Systems." pith.science (2026). https://pith.science/paper/VKFW7A4L

@misc{pith2026250511176,
  author       = {Pith},
  title        = {Pith review of: Enhancing and Scaling Search Query Datasets for Recommendation Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKFW7A4L}},
  note         = {Machine review of arXiv:2505.11176}
}

abstract

This paper presents a deployed, production-grade system designed to enhance and scale search query datasets for intent-based recommendation systems in digital banking. In real-world environments, the growing volume and complexity of user intents create substantial challenges for data management, resulting in suboptimal recommendations and delayed product onboarding. To overcome these challenges, our approach shifts the focus from model-centric enhancements to automated, data-centric strategies. The proposed system integrates three core modules: Synthetic Query Generation, Intent Disambiguation, and Intent Gap Analysis. Synthetic Query Generation produces diverse and realistic user queries. Our experiments reveal no statistically significant difference when using synthetic data for Clinc150, while Banking77 and a proprietary dataset show significant differences. We dig into the underlying factors driving these variations, demonstrating that our approach effectively alleviates the cold start problem (i.e. the challenge of recommending new products with limited historical data). Intent Disambiguation refines broad and overlapping intent categories into precise subintents, achieving an F1 score of 0.863 $\pm$ 0.127 against expert reannotations and leading to clearer differentiation and more precise recommendation mapping. Meanwhile, Intent Gap Analysis identifies latent customer needs by extracting novel intents from unlabeled queries; recovery rates reach up to 71\% in controlled evaluations. Deployed in a live banking environment, our system demonstrates significant improvements in recommendation precision and operation agility, ultimately delivering enhanced user experiences and strategic business benefits. This work underscores the role of high-quality, scalable data in modern AI-driven applications and advocates a proactive approach to data enhancement as a key driver of value.

Figures

Figures reproduced from arXiv: 2505.11176 by the authors.

**Figure 2.** Synthetic query generation prompt approach. By [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Intent Gap Analysis flow and example. Starting [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Comparison of macro-averaged F1 scores between models trained on control data and data with synthetic data re [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Intent disambiguation results (F1 score) compar [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Rediscovery rate of masked intents (mean [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Comparison of F1 scores for models trained on Chat146 and Chat186 (mean [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

NILC: Discovering New Intents with LLM-assisted Clustering
cs.CL 2025-11 conditional novelty 6.0 of 10

NILC combines LLM-generated semantic centroids with hard-sample rewriting to improve new-intent clustering, but its 'consistent' superiority claim is contradicted on DBPedia.

Reference graph

Works this paper leans on

36 extracted references · 34 canonical work pages · cited by 1 Pith paper

[1]

Create diverse, natural-sounding customer queries

work page
[2]

Include queries that vary substantially in length — from very short fragments and abbreviations (common in real mobile text inputs) to slightly longer, detailed questions

work page
[3]

Vary query complexity, specificity, and length

work page
[4]

Consider both direct commands and conversational, informal questions

work page
[5]

Focus on the core functionality users are trying to access with each query

work page
[6]

Be as concise and specific as the given examples, while also presenting a realistic diversity in users’ query style

work page
[7]

Some intents capture vague or one-word queries; others are more specific

work page
[8]

<Provided Intent Label>

Queries should mimic the casual and conversational tone of real user queries. You will be given a user intent or navigation key (navkey) from the JPMorgan Chase banking app. Your task is to generate 5 diverse user queries that would naturally lead to this navkey/intent. These queries should capture the natural variability observed in real interactions: - ...

work page

Show all 36 references

[13]

Focus on nuanced language, context, and semantic meaning

Analyze each user query and assess whether it clearly fits into one of the existing intent categories. Focus on nuanced language, context, and semantic meaning
[14]

Identify queries that, despite potential similarities, demand a separate, actionable intent that would justify the creation of dedicated webpages or search options
[15]

- A proposal for a new intent capturing the emerging customer need

For every gap identified, provide: - The exact example query (as it appears in the provided data). - A proposal for a new intent capturing the emerging customer need
[16]

The chain-of-thought reasoning must include your logical steps: explain how you determined that the query is distinct from current intents based on linguistic nuances and its potential impact on digital design
[17]

inquiry” or “general

Note that “inquiry” or “general” intents are unacceptable; the proposed intents must be specific and actionable
[18]

Given that the current taxonomy is fairly comprehensive, only include examples that clearly cannot be mapped to any existing categories
[19]

If none of the queries meet these criteria, set Valid to False. Please provide your response by filling out the following Y AML template (respond only in valid Y AML): Reasoning: <A detailed analysis and explanation for why the selected examples could not be classified under t...
[20]

not novel

Not novel / handled already by an existing intent (return “not novel”)
[21]

Can be easily answered by an FAQ (return “faq”)
[22]

not relevant

Not relevant to banking/financial products and services (return “not relevant”)
[23]

consideradding

A product, service, or experience that we should consider adding to the Chase mobile app (return “consideradding”) Instructions:
[24]

Analyze the proposed intent and its example query (or queries) in the context of the existing taxonomy
[25]

not novel

Determine if the proposed intent is already covered by an existing intent. If so, classify it as “not novel.”
[26]

If the proposed intent can be addressed through a simple FAQ, classify it as “faq.”
[27]

consider adding

If the proposed intent represents a significant new customer need that warrants the development of a new product, service, or experience, classify it as “consider adding.”
[28]

not relevant

If the proposed intent is not relevant to banking or financial products and services, classify it as “not relevant.”
[29]

Provide a brief explanation for your classification decision, considering linguistic nuances, customer needs, and potential digital design implications
[30]

not novel

Mention the top most similar intents from the existing taxonomy in your reasoning. Please provide your response by filling out the following Y AML template (respond only in valid Y AML): Reasoning: <Think about the proposed intent and your classification decision, including th...
[31]

Identify pairs of intents whose examples are essentially identical
[32]

- The end-user experience, in terms of webpage content or search functionality, would be identical

Only merge intents when: - The language and examples are exactly the same. - The end-user experience, in terms of webpage content or search functionality, would be identical. - If there is any nuance suggesting that two intents serve different purposes, keep them separate
[33]

False” for the “Valid

If no single pair of intents meets these strict criteria, return “False” for the “Valid” field. Return your analysis in the exact Y AML format below (do not include any extra text): Reasoning: <Detailed reasoning on why the chosen intents are or are not redundant based on thei...
[34]

Analyze the dropped intent along with its description and examples
[35]

Compare its core meaning with each of the proposed intents and decide if it is effectively covered by one of the proposed intents
[36]

Provide a detailed reasoning about any semantic similarities or differences. Return your result using the following Y AML format (respond only in Y AML): Reasoning: <Detailed explanation including the similarities or differences between the intent’s context and the proposed in...
[119]

San Diego, California: Association for Computational Linguistics. Li, X. L.; Holtzman, A.; Fried, D.; Liang, P.; Eisner, J.; Hashimoto, T.; Zettlemoyer, L.; and Lewis, M. 2023a. Con- trastive Decoding: Open-ended Text Generation as Opti- mization. In Rogers, A.; Boyd-Graber, J...

2024 arXiv
[346]

Wang, H.; Prakash, N.; Hoang, N

Online: Association for Computational Linguistics. Wang, H.; Prakash, N.; Hoang, N. K.; Hee, M. S.; Naseem, U.; and Lee, R. K.-W. 2023. Prompting Large Language Models for Topic Modeling. In 2023 IEEE International Conference on Big Data (BigData) , 1236–1241. Sorrento, Italy:...

2023
[2022]

cold start

ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback. In Goldberg, Y .; Kozareva, Z.; and Zhang, Y ., eds., Findings of the Association for Computa- tional Linguistics: EMNLP 2022 , 3671–3683. Abu Dhabi, United Arab Emirates: Association for Computational L...

2022
[2024]

TopicGPT: A Prompt-based Topic Modeling Frame- work. In Duh, K.; Gomez, H.; and Bethard, S., eds., Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , 2956–29...

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Create diverse, natural-sounding customer queries

work page

[2] [2]

Include queries that vary substantially in length — from very short fragments and abbreviations (common in real mobile text inputs) to slightly longer, detailed questions

work page

[3] [3]

Vary query complexity, specificity, and length

work page

[4] [4]

Consider both direct commands and conversational, informal questions

work page

[5] [5]

Focus on the core functionality users are trying to access with each query

work page

[6] [6]

Be as concise and specific as the given examples, while also presenting a realistic diversity in users’ query style

work page

[7] [7]

Some intents capture vague or one-word queries; others are more specific

work page

[8] [8]

<Provided Intent Label>

Queries should mimic the casual and conversational tone of real user queries. You will be given a user intent or navigation key (navkey) from the JPMorgan Chase banking app. Your task is to generate 5 diverse user queries that would naturally lead to this navkey/intent. These queries should capture the natural variability observed in real interactions: - ...

work page