REVIEW 5 major objections 6 minor 40 references
LLM-Based Re-Ranking for Real Estate Search
T0 review · 5 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read This paper claims that adding an LLM point-wise re-ranking stage to a conversational real-estate search system improves ranking quality and downstream user actions, with production A/B results of +5.3% click-through rate and +4.8% scheduled
desk verdict Real production evidence, but the A/B stats are under-reported and the offline judge is too close to the re-ranker; still worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is point-wise affinity scoring: each candidate is evaluated independently by the LLM, which makes the re-ranking stage parallel and latency-tolerant. The scoring prompt combines the user's free-text requirement, a generated textual user profile, structured filters, property metadata, and candidate-set statistics such as price ranges and amenity prevalence. These statistics let the model judge relative value even though each property is scored alone. The user search profile—a condensed natural-language summary of behavioral history and conversational statements—is the key object that injects personalization into the scores.
What would settle it
Take a fresh sample of production traces, have multiple human raters label relevance for query–listing pairs without knowing which order came from the re-ranker, and recompute the offline ranking metrics; if the re-ranker's nDCG advantage over the baseline shrinks to zero or reverses, the central claim fails. A complementary check: run a longer production A/B that tracks completed visits or signed leases rather than clicks and scheduled visits.
Extended reading notes
Core claim
The discovery is that a point-wise LLM scorer, given a natural-language user profile and aggregate statistics over the candidate set, can reorder retrieved housing listings so that they better match the user's actual goals. The profile is assembled from previous sessions, the current conversation, and historical interaction signals, with the most recent and explicit conversational statements given priority. Each candidate property receives an affinity score in [0,1] that combines hard-constraint satisfaction, preference alignment, and relative value; candidates are sorted by that score, with ties broken by the original retrieval score. On an offline set of 960,000 query–listing pairs built f
Load-bearing premise
The offline evaluation treats an LLM judge, validated on only 1,000 human-labeled pairs (96% agreement), as ground truth for 960,000 query–listing pairs; if that judge shares the re-ranker's systematic biases, the offline gains are inflated.
Editorial extensions
If this is right
- LLM re-ranking can be layered on top of existing retrieval systems and improve engagement without end-to-end generative retrieval or a full rewrite of the serving stack.
- Candidate-set statistics add consistent, if modest, ranking gains in point-wise scoring, suggesting that even independent scoring benefits from aggregate context.
- House descriptions and textual user profiles contribute the largest semantic gains; explicit scoring-calibration instructions actually hurt slightly, so prompt design matters more than rigid rules.
- The deployed system did not yet include the textual user profile, so the production A/B may understate the approach's full potential.
- Synthetic query generation mixed with production queries and LLM-as-a-Judge labeling offers a scalable template for offline evaluation in conversational search.
Reading between the lines
- A natural extension would be to track whether the +4.8% scheduled-visit gain persists further downstream, in completed tours, lease applications, or signed contracts—the strongest evidence that re-ranking improves real outcomes rather than engagement.
- The judge win rate on sampled production traces is very high (95%), which invites a skeptical check: a blind human preference study on the same traces would separate style effects from genuine ranking improvements.
- The paper does not examine fairness across neighborhoods or demographic groups; a plausible caution is that an LLM profile re-ranker could amplify existing spatial or socioeconomic biases in housing search.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a production conversational real-estate search assistant (Concierge) at QuintoAndar and proposes an LLM-based point-wise re-ranker. The re-ranker scores each retrieved property using the user's conversational request, an LLM-generated user profile, structured filters, property metadata/descriptions, and aggregate candidate-set statistics, then sorts candidates by the resulting affinity score. The authors build a proprietary 960,000 query-item offline dataset from synthetic and production queries with LLM-as-a-Judge relevance labels, evaluate the re-ranker offline against a retrieval baseline, and report a production A/B test with claimed statistically significant improvements of +5.3% in CTR and +4.8% in scheduled visits. The paper concludes that LLM re-ranking improves ranking quality and downstream engagement while remaining operationally feasible.
Significance. If fully verified, this would be a valuable industrial case study: it demonstrates that an LLM point-wise re-ranker can improve real-estate search in production without replacing the retrieval stack, and it provides a large-scale offline benchmark plus an independent online experiment. The production A/B test is the strongest positive evidence, and the reporting of latency and cost overhead is useful. However, the current reporting of the A/B test lacks the statistical detail needed to verify the central claim, and the offline and LLM-judge evaluations rely heavily on LLM-generated judgments with only small human validation. The contribution is therefore significant but conditional on additional evidence.
major comments (5)
- [§4.4 (A/B test, Table-related text)] The abstract's central claim of 'statistically significant' production gains rests entirely on this A/B test, but the statistical reporting is incomplete. The paper reports only relative uplifts (+5.3% CTR, +4.8% scheduled visits) and states significance without giving p-values, confidence intervals, the test procedure, per-arm sample sizes, or the unit of randomization/analysis. With 'more than 200,000 production recommendation traces,' if traces rather than users are the unit of analysis, user-level correlation (pseudoreplication) could inflate significance. Please report a user-level clustered analysis, exact denominators and definitions of the metrics, and a pre-specified testing plan.
- [§4.1.3 and §4.1.5 (LLM-as-a-Judge labels)] The offline benchmark's 960,000 labels are generated by Claude Sonnet 4 with human validation on only 1,000 pairs (96% agreement). Since the evaluated re-ranker is itself an LLM and the labels are also LLM judgments, shared systematic biases (e.g., preference for fluent listing text or model-familiar patterns) could inflate offline gains. The 1,000-pair validation is not stratified across the six synthetic query types, hard negatives, or production-query segments, so it is unclear whether 96% agreement extends to the most difficult pairs. Please provide agreement by query type/candidate difficulty and report judge self-consistency.
- [§4.5 (LLM-as-a-Judge win rate)] The judge-based evaluation reports that the re-ranked list was preferred over the baseline in 95% of 3,944 production traces, with no human validation, no confidence intervals, and no checks for judge bias or position effects. A 95% pairwise win rate against a strong retrieval baseline is implausibly high and suggests possible shared bias between the judge and the re-ranker. Please report inter-annotator agreement (human vs. judge), judge sensitivity to list order/format, and calibrate the judge on a human-annotated pairwise sample.
- [§4.3, Table 1 (offline significance)] The offline results mark configurations as statistically significant (p<0.05) but no statistical test, variance estimate, confidence interval, or number of independent query/item groups is reported. Several improvements are small (0.8%, 0.9%, 1.1%), and the candidate-set construction includes anchor items that may be trivially relevant, making it unclear whether the differences are beyond sampling noise. Please report query-level paired tests with confidence intervals, and clarify whether the test accounts for repeated items across groups.
- [§4.4 (latency and confound)] The treatment arm adds an average 4.2s end-to-end latency. The observed CTR and scheduled-visit uplifts could partly reflect a differential user experience (e.g., slower but more 'deliberative' interaction) rather than ranking quality alone. No guardrail metrics are reported—abandonment, post-click satisfaction, repeat use, or visit show rate—so this confound cannot be ruled out. Please report guardrail metrics and, if feasible, a latency-matched control or a sensitivity analysis.
minor comments (6)
- [§3.3.2 (candidate-set statistics)] The aggregate statistics are described only qualitatively; please specify how many candidates are used, which statistics are computed, and how they are serialized into the prompt, so the method is reproducible.
- [§3.2 (user search profile)] The behavioral profile 'interaction weights' are mentioned but not defined. Please give the actual weighting scheme or a reference, since this is a component of the proposed method.
- [§4.1.1 and §4.1.3 (human validation samples)] Two separate 1,000-pair human evaluations are described (one for synthetic query quality, one for relevance labels). Please clarify whether the samples overlap and report the exact annotation instructions and annotator background.
- [§4.4 (Figure 6)] The CTR-over-time figure would benefit from confidence bands and explicit arm sizes per day; currently the visual comparison is hard to interpret without uncertainty information.
- [§4.5 (Figure 7)] The daily win-rate figure should show evaluation volume and confidence intervals; daily rates with very small volumes can be misleading.
- [§4.4 (baseline description)] Please describe the baseline retrieval model and ranking function in enough detail (or cite an internal reference) so readers can assess the strength of the baseline being compared against.
Circularity Check
No significant circularity: the re-ranker is not fitted to the LLM-judge labels, and the production A/B test is an independent behavioral measurement.
full rationale
After walking the derivation chain, I find no step that reduces a predicted quantity to an input by construction. Offline labels (§4.1.3) are assigned by Claude Sonnet 4 under a structured constraint-checking protocol, with two human agreement checks (94% on query generation, 96% on relevance labels). These labels are not generated by the re-ranker, and no parameter of the re-ranker is fit to them; the LLM re-ranker uses a fixed prompt (Figure 3), not trained weights, so the offline nDCG/Recall improvements are not statistically forced. The online A/B test (§4.4) is an independent behavioral measurement on >200,000 production traces, measuring CTR and scheduled visits, not the LLM judge's preferences; while its statistical reporting is incomplete (no confidence intervals, p-values, or clustering details), incompleteness is a reproducibility/correctness concern rather than circularity. The §4.5 LLM-judge preference test is also an external judge (not the re-ranker itself), and the 95% win rate is a comparison, not a derivation from the re-ranker's own outputs. The paper contains no self-citations and imports no uniqueness theorem from the authors' prior work. The legitimate worry about shared bias between LLM judges and LLM rankers is a validity threat for the offline and judge-based evaluations, not a definitional reduction of the kind required to establish circularity. The central production claim rests on the A/B test, which is external to the paper's fitted or annotated inputs. Therefore no circular step is exhibited, and the score is 0.
Assumptions & free parameters
free parameters (3)
- candidate pool size k =
10
- top-k surfaced to user =
5
- behavioral profile interaction weights
assumptions (5)
- domain assumption LLM-as-a-Judge labels validated on 1,000 pairs generalize to 960,000 pairs
- ad hoc to paper LLM-generated synthetic queries approximate realistic conversational search behavior
- domain assumption CTR and scheduled visits are reliable proxies for ranking quality in production
- domain assumption Treatment and control groups are comparable in the A/B test
- domain assumption Dense embedding similarity (e5-base + FAISS) yields meaningful hard negatives
Cite this review
Pith. "Pith review of LLM-Based Re-Ranking for Real Estate Search." pith.science (2026). https://pith.science/paper/JY24Z7U7
@misc{pith2026260714835,
author = {Pith},
title = {Pith review of: LLM-Based Re-Ranking for Real Estate Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/JY24Z7U7}},
note = {Machine review of arXiv:2607.14835}
}
read the original abstract
QuintoAndar Group operates the leading housing marketplace in Latin America for both rentals and sales. The platform replaces traditionally paper-heavy workflows with a fully digital experience, making housing transactions faster and more accessible to tenants, buyers, and landlords in the region. Finding the ideal home in such a vast catalog is inherently difficult. At the same time, the widespread adoption of conversational assistants is reshaping user expectations: people increasingly want to express their needs through open, multi-turn dialog rather than rigid filter menus and faceted search. This shift is particularly pronounced in housing, where intent is multi-dimensional, context-dependent, and rarely reducible to a small set of structured constraints. To meet these expectations, we propose a Large Language Model (LLM) based re-ranker that augments a conversational recommendation system by reordering retrieved candidates according to the nuanced, context-rich intent expressed across the user's conversation. We additionally construct a large-scale offline evaluation dataset for conversational real-estate search, containing 960,000 query-item pairs constructed from both synthetic and production queries and annotated using an LLM-as-a-Judge framework with human validation. We validate our approach both offline, on this proprietary dataset, and online, through a production A/B test. Both evaluations show consistent improvements in ranking quality, including a statistically significant increase in production of +5.3% in click-through rate and +4.8% in scheduled visits, demonstrating the value of integrating conversational context into housing recommendations.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Anthropic. 2025. Introducing Claude 4. https://www.anthropic.com/news/claude- 4
2025
-
[2]
Honghui Bao, Wenjie Wang, Xinyu Lin, Fengbin Zhu, Teng Sun, Fuli Feng, and Tat-Seng Chua. 2025. Heterogeneous User Modeling for LLM-based Recommen- dation. InProceedings of the Nineteenth ACM Conference on Recommender Systems. Association for Computing Machinery, 145–154. LLM-Based Re-Ranking for Real Estate Search RecSys ’26, September 27– October 1, 202...
2025
-
[3]
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. InProceedings of the 17th ACM Conference on Recommender Systems. Association for Computing Machinery, 1007–1014
2023
-
[4]
Yung-Sung Chuang, Wei Fang, Shang-Wen Li, Wen-Tau Yih, and James Glass
-
[5]
Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongx- iang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT’s Capabilities in Recommender Systems. InProceedings of the 17th ACM Conference on Recom- mender Systems. Association for Computing Machinery, 1126–1132
2023
-
[6]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. arXiv:2401.08281
arXiv 2024
-
[7]
Luke Friedman, Sameer Ahuja, David Allen, Zhenning Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, et al. 2023. Leveraging large language models in conversational recommender systems. arXiv:2305.07961
arXiv 2023
-
[8]
Jingtong Gao, Bo Chen, Xiangyu Zhao, Weiwen Liu, Xiangyang Li, Yichao Wang, Wanyu Wang, Huifeng Guo, and Ruiming Tang. 2025. LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations. InProceedings of the ACM on Web Conference 2025. Association for Computing Machinery, 228–239
2025
Show all 40 references
-
[9]
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). InProceedings of the 16th ACM Conference on Recommender Systems. Association for Computin...
2022
-
[10]
Google. 2026. Gemini 3.1 Flash-Lite. https://deepmind.google/en/models/gemini/ flash-lite/ Accessed: 2026-05-15
2026
-
[11]
Hammond, Robin Burke, and Kathryn Schmitt
Kristian J. Hammond, Robin Burke, and Kathryn Schmitt. 1994. A Case-Based Approach to knowledge Navigation. InAAAI’94
1994
-
[12]
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large Language Models are Zero-Shot Rankers for Recommender Systems. InAdvances in Information Retrieval: 46th European Conference on Information Retrieval. Springer-Verlag, 364–381
2024
-
[13]
Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, and Xing Xie. 2025. Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations.ACM Trans. Inf. Syst.43 (6 2025). Issue 4
2025
-
[14]
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A Survey on Conversational Recommender Systems.ACM Comput. Surv.54, 5, Article 105 (May 2021), 36 pages
2021
-
[15]
2026.How will AI reshape the news in 2026? Forecasts by 17 experts from around the world
Ada Jušić and Eleonora Lima. 2026.How will AI reshape the news in 2026? Forecasts by 17 experts from around the world. Reuters Institute for the Study of Journalism, University of Oxford. https://reutersinstitute.politics.ox.ac.uk/news/how-will- ai-reshape-news-2026-forecasts-...
2026
-
[16]
Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim, Minchul Yang, and Chanyoung Park. 2024. Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender System. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Da...
2024
-
[17]
Jiahao Liu, Xueshuo Yan, Dongsheng Li, Guangping Zhang, Hansu Gu, Peng Zhang, Tun Lu, Li Shang, and Ning Gu. 2025. Improving LLM-powered Recom- mendations with Personalized Information. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in...
2025
-
[18]
Peng Liu, Lemei Zhang, and Jon Atle Gulla. 2023. Pre-train, Prompt, and Recom- mendation: A Comprehensive Survey of Language Modeling Paradigm Adapta- tions in Recommender Systems.Transactions of the Association for Computational Linguistics11 (2023), 1553–1571
2023
-
[19]
Qidong Liu, Xiangyu Zhao, Yuhao Wang, Yejing Wang, Zijian Zhang, Yuqi Sun, Xiang Li, Maolin Wang, Pengyue Jia, Chong Chen, Wei Huang, and Feng Tian
-
[20]
Sichun Luo, Bowei He, Haohan Zhao, Wei Shao, Yanlin Qi, Yinya Huang, Aojun Zhou, Yuxuan Yao, Zongpeng Li, Yuanzhang Xiao, Mingjie Zhan, and Linqi Song
-
[21]
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query Rewriting in Retrieval-Augmented Large Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 5303–5315
2023
-
[22]
Marco De Nadai, Francesco Fabbri, Paul Gigioli, Alice Wang, Ang Li, Fabrizio Silvestri, Laura Kim, Shawn Lin, Vladan Radosavljevic, Sandeep Ghael, David Nyhan, Hugues Bouchard, Mounia Lalmas, and Andreas Damianou. 2024. Person- alized Audiobook Recommendations at Spotify Throu...
2024
-
[23]
RecRanker: Instruction Tuning Large Language Model as Ranker for Top-k Recommendation.ACM Trans. Inf. Syst.43 (7 2025). Issue 5
2025
-
[24]
OpenAI. 2024. GPT-4o Model. https://platform.openai.com/docs/models/gpt-4o Model version: gpt-4o-2024-11-20
2024
-
[25]
Enrico Palumbo, Marcus Isaksson, Alexandre Tamborrino, Maria Movin, Catalin Dincu, Ali Vardasbi, Lev Nikeshkin, Oksana Gorobets, Anders Nyman, Poppy Newdick, Hugues Bouchard, Paul Bennett, Mounia Lalmas, Dani Doro, Chris- tine Doig Cardet, and Ziad Sultan. 2025. You Say Search...
2025
-
[26]
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. arXiv:1904.08375
2019 arXiv
-
[27]
Francesco Ricci and Quang Nhat Nguyen. 2007. Acquiring and Revising Prefer- ences in a Critique-Based Mobile Recommender System.IEEE Intelligent Systems 22, 3 (May 2007), 22–29
2007
-
[28]
Wentao Shi, Xiangnan He, Yang Zhang, Chongming Gao, Xinyue Li, Jizhi Zhang, Qifan Wang, and Fuli Feng. 2024. Large Language Models are Learnable Planners for Long-Term Recommendation. InProceedings of the 47th International ACM SI- GIR Conference on Research and Development in...
2024
-
[29]
Enrico Palumbo, Gustavo Penha, Alva Liu, Marcus Eltscheminov, Jefferson Car- valho dos Santos, Alice Wang, Hugues Bouchard, Humberto Jesús Corona Pampin, and Michelle Tran Luu. 2025. AudioBoost: Increasing Audiobook Retrievability in Spotify Search with Synthetic Query Generat...
2025 arXiv
-
[30]
Jianling Wang, Ainur Yessenalina, and Roshan-Ghias Alireza. 2022. Exploring Heterogeneous Metadata for Video Recommendation with Two-tower Model. (2022). arXiv:2109.11059
2022 arXiv
-
[31]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv:2212.03533
2022 arXiv
-
[32]
Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 235–244
2018
-
[33]
Jiaqi Xi, Raghav Saboo, Luming Chen, Martin Wang, and Sudeep Das. 2026. Mine and Refine: Optimizing Graded Relevance in E-commerce Search Retrieval. arXiv:2602.17654
2026
-
[34]
Adam Yang, Gustavo Penha, Enrico Palumbo, and Hugues Bouchard. 2025. Aligned Query Expansion: Efficient Query Expansion for Information Retrieval through LLM Alignment. (2025). arXiv:2507.11042
2025 arXiv
-
[35]
Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Yanbin Lu, Xiaojiang Huang, and Yingzhen Yang. 2024. RecMind: Large Language Model Powered Agent For Recommendation. InFindings of the Associ- ation for Computational Linguistics: NAACL 2024. ...
2024
-
[36]
Joyce Zhou, Yijia Dai, and Thorsten Joachims. 2024. Language-Based User Profiles for Recommendation. arXiv:2402.15623
2024 arXiv
-
[37]
Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. 2021. TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search. InProceedings of the Web Conference 2021. Association for Computi...
2021
-
[38]
Junjie Zhang, Yupeng Hou, Ruobing Xie, Wenqi Sun, Julian McAuley, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2024. AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems. InProceedings of the ACM Web Conference 2024. Association for Computing M...
2024
-
[2023]
InFindings of the Association for Computational Linguistics: ACL 2023
Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering. InFindings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 12131–12147
2023
-
[2025]
InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Large Language Model Enhanced Recommender Systems: Methods, Ap- plications and Trends. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. Association for Computing Machinery, 6096–6106
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.