{"id":"435da913-96dd-4eb0-b240-1f0a91509e85","arxiv_id":"2607.14835","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM pointwise re-ranker using user profiles and candidate-set statistics improved ranking quality in a real-estate marketplace, with statistically significant production gains of +5.3% CTR and +4.8% scheduled visits.","lead":"This paper describes an LLM-based re-ranker at a Latin American housing marketplace that reorders candidate properties using conversational context, and reports a production A/B test with +5.3% click-through rate and +4.8% scheduled visits. It also introduces a 960,000-pair offline benchmark built from synthetic and production queries labeled with an LLM judge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Production A/B is the load-bearing evidence, but §4.4 reports no CIs, p-values, or clustering details, so the 'statistically significant' central claim is not independently checkable.","rationale":"The reader's weakest-assumption focuses on LLM-as-a-Judge bias in the offline evaluation. That is a valid concern, but the offline evaluation is not the load-bearing part of the central claim: the abstract's production claim is supported by the A/B test, which uses real user clicks and scheduled visits rather than LLM labels. The A/B result is therefore the strongest and most independent evidence. However, the published description of the A/B test is too thin to verify the statistical significance claim: no confidence intervals, p-values, or randomization/clustering details are given, and the treatment arm's additional 4.2s latency introduces a potential confound that is not addressed with guardrail metrics. These omissions make the central production claim plausible but not independently checkable. This does not change the reader's CONDITIONAL verdict, but it shifts the emphasis: the most fragile assumption is the statistical validity of the A/B analysis, not the offline label quality. The proposed concrete test would settle whether the production gain is robust or an artifact of pseudoreplication or unmeasured confounds.","tokens_in":12473,"tokens_out":6263,"duration_ms":57724,"concrete_test":"Obtain the production A/B logs and recompute the two headline uplifts with user-level cluster-robust 95% confidence intervals (e.g., block bootstrap by user or mixed-effects model with a random intercept for user). If the CI for either uplift includes 0, the 'statistically significant' claim in §4.4 is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the LLM re-ranker improves production outcomes rests primarily on the A/B test in §4.4. This leg is under-specified: the paper reports only relative uplifts (+5.3% CTR, +4.8% scheduled visits) and the phrase 'statistically significant,' with no confidence intervals, p-values, test choice, per-arm sample sizes, or unit of randomization. With 'more than 200,000 production recommendation traces' rather than users, significance computed at the trace level can be artificially strong due to user-level correlation (pseudoreplication). Additionally, the treatment arm experienced +4.2s end-to-end latency, and no guardrail metrics (e.g., satisfaction, abandonment, repeat use) are reported, so the observed uplift could partly reflect differential user experience rather than ranking quality alone. If the A/B evidence is not rigorously established, the central production claim lacks support; the offline LLM-judge bias in §4.1.3 and §4.5 is a real secondary concern but cannot carry the claim by itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a production conversational real-estate search assistant (Concierge) at QuintoAndar and proposes an LLM-based point-wise re-ranker. The re-ranker scores each retrieved property using the user's conversational request, an LLM-generated user profile, structured filters, property metadata/descriptions, and aggregate candidate-set statistics, then sorts candidates by the resulting affinity score. The authors build a proprietary 960,000 query-item offline dataset from synthetic and production queries with LLM-as-a-Judge relevance labels, evaluate the re-ranker offline against a retrieval baseline, and report a production A/B test with claimed statistically significant improvements of +5.3% in CTR and +4.8% in scheduled visits. The paper concludes that LLM re-ranking improves ranking quality and downstream engagement while remaining operationally feasible.","tokens_in":12758,"tokens_out":4894,"duration_ms":44332,"significance":"If fully verified, this would be a valuable industrial case study: it demonstrates that an LLM point-wise re-ranker can improve real-estate search in production without replacing the retrieval stack, and it provides a large-scale offline benchmark plus an independent online experiment. The production A/B test is the strongest positive evidence, and the reporting of latency and cost overhead is useful. However, the current reporting of the A/B test lacks the statistical detail needed to verify the central claim, and the offline and LLM-judge evaluations rely heavily on LLM-generated judgments with only small human validation. The contribution is therefore significant but conditional on additional evidence.","major_comments":[{"comment":"The abstract's central claim of 'statistically significant' production gains rests entirely on this A/B test, but the statistical reporting is incomplete. The paper reports only relative uplifts (+5.3% CTR, +4.8% scheduled visits) and states significance without giving p-values, confidence intervals, the test procedure, per-arm sample sizes, or the unit of randomization/analysis. With 'more than 200,000 production recommendation traces,' if traces rather than users are the unit of analysis, user-level correlation (pseudoreplication) could inflate significance. Please report a user-level clustered analysis, exact denominators and definitions of the metrics, and a pre-specified testing plan.","section":"§4.4 (A/B test, Table-related text)"},{"comment":"The offline benchmark's 960,000 labels are generated by Claude Sonnet 4 with human validation on only 1,000 pairs (96% agreement). Since the evaluated re-ranker is itself an LLM and the labels are also LLM judgments, shared systematic biases (e.g., preference for fluent listing text or model-familiar patterns) could inflate offline gains. The 1,000-pair validation is not stratified across the six synthetic query types, hard negatives, or production-query segments, so it is unclear whether 96% agreement extends to the most difficult pairs. Please provide agreement by query type/candidate difficulty and report judge self-consistency.","section":"§4.1.3 and §4.1.5 (LLM-as-a-Judge labels)"},{"comment":"The judge-based evaluation reports that the re-ranked list was preferred over the baseline in 95% of 3,944 production traces, with no human validation, no confidence intervals, and no checks for judge bias or position effects. A 95% pairwise win rate against a strong retrieval baseline is implausibly high and suggests possible shared bias between the judge and the re-ranker. Please report inter-annotator agreement (human vs. judge), judge sensitivity to list order/format, and calibrate the judge on a human-annotated pairwise sample.","section":"§4.5 (LLM-as-a-Judge win rate)"},{"comment":"The offline results mark configurations as statistically significant (p<0.05) but no statistical test, variance estimate, confidence interval, or number of independent query/item groups is reported. Several improvements are small (0.8%, 0.9%, 1.1%), and the candidate-set construction includes anchor items that may be trivially relevant, making it unclear whether the differences are beyond sampling noise. Please report query-level paired tests with confidence intervals, and clarify whether the test accounts for repeated items across groups.","section":"§4.3, Table 1 (offline significance)"},{"comment":"The treatment arm adds an average 4.2s end-to-end latency. The observed CTR and scheduled-visit uplifts could partly reflect a differential user experience (e.g., slower but more 'deliberative' interaction) rather than ranking quality alone. No guardrail metrics are reported—abandonment, post-click satisfaction, repeat use, or visit show rate—so this confound cannot be ruled out. Please report guardrail metrics and, if feasible, a latency-matched control or a sensitivity analysis.","section":"§4.4 (latency and confound)"}],"minor_comments":[{"comment":"The aggregate statistics are described only qualitatively; please specify how many candidates are used, which statistics are computed, and how they are serialized into the prompt, so the method is reproducible.","section":"§3.3.2 (candidate-set statistics)"},{"comment":"The behavioral profile 'interaction weights' are mentioned but not defined. Please give the actual weighting scheme or a reference, since this is a component of the proposed method.","section":"§3.2 (user search profile)"},{"comment":"Two separate 1,000-pair human evaluations are described (one for synthetic query quality, one for relevance labels). Please clarify whether the samples overlap and report the exact annotation instructions and annotator background.","section":"§4.1.1 and §4.1.3 (human validation samples)"},{"comment":"The CTR-over-time figure would benefit from confidence bands and explicit arm sizes per day; currently the visual comparison is hard to interpret without uncertainty information.","section":"§4.4 (Figure 6)"},{"comment":"The daily win-rate figure should show evaluation volume and confidence intervals; daily rates with very small volumes can be misleading.","section":"§4.5 (Figure 7)"},{"comment":"Please describe the baseline retrieval model and ranking function in enough detail (or cite an internal reference) so readers can assess the strength of the baseline being compared against.","section":"§4.4 (baseline description)"}],"recommendation":"major_revision","confidential_remarks":"The production A/B test is the load-bearing evidence, and I believe the authors likely have the underlying data to address the statistical concerns; the current manuscript simply does not report enough detail. The LLM-as-a-Judge circularity is a real but secondary concern, primarily affecting the offline and judge-based evaluations rather than the A/B result. I would not recommend rejection at this point, but the statistical reporting in §4.4 and the validation detail in §4.1/§4.5 must be substantially improved before the central claims can be evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine industrial deployment with a real A/B test, and the core claim — that a point-wise LLM re-ranker can improve real-estate search without replacing the retrieval stack — is plausible. But the A/B evidence, which is load-bearing, is under-specified: no confidence intervals, no p-values, no unit-of-randomization details. And the offline benchmark leans on an LLM judge that likely shares biases with the LLM re-ranker.\n\nWhat's actually new: the 960k query-item conversational real-estate dataset (synthetic + production queries, LLM-as-a-Judge with human validation), the point-wise re-ranker that uses candidate-set aggregate statistics, and the production deployment with 200k+ traces. The ablation results are coherent: candidate-set stats help, house descriptions help, textual user profiles help the most. The paper is honest about what was deployed and what wasn't.\n\nThe soft spots are real but not fatal. The A/B test reports relative uplifts (+5.3% CTR, +4.8% scheduled visits) and says 'statistically significant' without giving the machinery. With traces rather than users as the unit, user-level correlation can inflate significance; the 4.2s added latency and the absence of guardrail metrics mean part of the uplift could be a user experience effect rather than pure ranking quality. That said, the direction of the effect is consistent across online and offline, which supports the claim. The 95% judge win-rate on 3,944 traces is suspiciously high if the judge is the same family as the re-ranker; a 1,000-pair human validation on an offline dataset is not enough to bound that bias.\n\nThis paper deserves a serious referee. It is an honest, well-scoped industrial case study. The authors should be pushed to report the A/B test with full statistical detail (CIs, p-values, clustering, guardrails) and to release or share the data where possible. If they do, this becomes a cited reference for LLM re-ranking in production. I'd send it to review rather than desk-reject, and I'd tell the authors to fix the reporting before acceptance.","headline":"Real production evidence, but the A/B stats are under-reported and the offline judge is too close to the re-ranker; still worth a serious referee.","tokens_in":13202,"tokens_out":2054,"would_cite":true,"duration_ms":18052,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding an LLM point-wise re-ranking stage to a conversational real-estate search system improves ranking quality and downstream user actions, with production A/B results of +5.3% click-through rate and +4.8% scheduled","keywords":["LLM re-ranking","conversational recommendation","real estate search","point-wise ranking","LLM-as-a-Judge","synthetic query generation","A/B testing","user search profile"],"falsifier":"Take a fresh sample of production traces, have multiple human raters label relevance for query–listing pairs without knowing which order came from the re-ranker, and recompute the offline ranking metrics; if the re-ranker's nDCG advantage over the baseline shrinks to zero or reverses, the central claim fails. A complementary check: run a longer production A/B that tracks completed visits or signed leases rather than clicks and scheduled visits.","tokens_in":12395,"feed_emoji":"🏠","tokens_out":6522,"duration_ms":51386,"temperature":0.7,"pith_summary":"Real-estate search intent is often conversational and qualitative—proximity to work, neighborhood character, trade-offs like paying more for a gym—and structured filters cannot capture it. The paper tries to show that a large language model applied only at the re-ranking stage, over a small candidate set already retrieved by a standard search engine, can absorb this conversational context and meaningfully improve results without replacing the serving stack. In a production housing marketplace, the authors compared a conversational assistant with and without the LLM re-ranker. The re-ranked version produced statistically significant gains: 5.3% more clicks and 4.8% more scheduled visits, with end-to-end latency increasing by about 4.2 seconds and inference cost by about 7%. If this is right, conversational search can improve real recommendation outcomes through a narrow, pragmatic intervention rather than an end-to-end generative rewrite.","feed_headline":"LLM re-ranker lifts real-estate clicks 5.3% and visits 4.8%","feed_subtitle":"A conversational housing assistant reorders top candidates with an LLM, and the change pays off in real user actions.","key_machinery":"The load-bearing mechanism is point-wise affinity scoring: each candidate is evaluated independently by the LLM, which makes the re-ranking stage parallel and latency-tolerant. The scoring prompt combines the user's free-text requirement, a generated textual user profile, structured filters, property metadata, and candidate-set statistics such as price ranges and amenity prevalence. These statistics let the model judge relative value even though each property is scored alone. The user search profile—a condensed natural-language summary of behavioral history and conversational statements—is the key object that injects personalization into the scores.","core_discovery":"The discovery is that a point-wise LLM scorer, given a natural-language user profile and aggregate statistics over the candidate set, can reorder retrieved housing listings so that they better match the user's actual goals. The profile is assembled from previous sessions, the current conversation, and historical interaction signals, with the most recent and explicit conversational statements given priority. Each candidate property receives an affinity score in [0,1] that combines hard-constraint satisfaction, preference alignment, and relative value; candidates are sorted by that score, with ties broken by the original retrieval score. On an offline set of 960,000 query–listing pairs built f","pith_inferences":["A natural extension would be to track whether the +4.8% scheduled-visit gain persists further downstream, in completed tours, lease applications, or signed contracts—the strongest evidence that re-ranking improves real outcomes rather than engagement.","The judge win rate on sampled production traces is very high (95%), which invites a skeptical check: a blind human preference study on the same traces would separate style effects from genuine ranking improvements.","The paper does not examine fairness across neighborhoods or demographic groups; a plausible caution is that an LLM profile re-ranker could amplify existing spatial or socioeconomic biases in housing search."],"forward_implications":["LLM re-ranking can be layered on top of existing retrieval systems and improve engagement without end-to-end generative retrieval or a full rewrite of the serving stack.","Candidate-set statistics add consistent, if modest, ranking gains in point-wise scoring, suggesting that even independent scoring benefits from aggregate context.","House descriptions and textual user profiles contribute the largest semantic gains; explicit scoring-calibration instructions actually hurt slightly, so prompt design matters more than rigid rules.","The deployed system did not yet include the textual user profile, so the production A/B may understate the approach's full potential.","Synthetic query generation mixed with production queries and LLM-as-a-Judge labeling offers a scalable template for offline evaluation in conversational search."],"fun_headline_variants":["Conversational LLM re-ranking lifts housing clicks 5.3% and visits 4.8%","Point-wise LLM re-ranker: +5.3% CTR, +4.8% visits in real estate","In-production A/B: LLM re-ranking ups housing CTR 5.3%, visits 4.8%","LLM re-ranker trained on 960k pairs boosts real-estate search","Housing search re-ranking via LLM: 5.3% more clicks, 4.8% more visits"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The offline evaluation treats an LLM judge, validated on only 1,000 human-labeled pairs (96% agreement), as ground truth for 960,000 query–listing pairs; if that judge shares the re-ranker's systematic biases, the offline gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Conversational LLM re-ranking lifts housing clicks 5.3% and visits 4.8%","Point-wise LLM re-ranker: +5.3% CTR, +4.8% visits in real estate","In-production A/B: LLM re-ranking ups housing CTR 5.3%, visits 4.8%","LLM re-ranker trained on 960k pairs boosts real-estate search","Housing search re-ranking via LLM: 5.3% more clicks, 4.8% more visits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1680,"prompt_tokens":789,"completion_tokens":891,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":755}},"tokens_in":533,"tokens_out":891,"duration_ms":7005,"temperature":1.0,"reasoning_tokens":755,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T00:54:02.439715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of production traces, have multiple human raters label relevance for query–listing pairs without knowing which order came from the re-ranker, and recompute the offline ranking metrics; if the re-ranker's nDCG advantage over the baseline shrinks to zero or reverses, the central claim fails. A complementary check: run a longer production A/B that tracks completed visits or signed leases rather than clicks and scheduled visits.","supporting_citations":[],"review_version":1}