{"id":"eaf12459-9e97-4079-84f8-eb03a11f52af","arxiv_id":"2411.13584","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM fine-tuned on 60 million logistics address examples, with nearby-address retrieval and reinforcement learning from JD's geocoding service, rewrites abnormal Chinese addresses better than prior geocoding and rewriting baselines.","lead":"The authors built and deployed AddrLLM, an AI system that rewrites messy Chinese delivery addresses into standard ones using a fine-tuned large language model plus a geography-aware search step. It reported correcting about 44% of abnormal addresses offline and over 40% in a four-month live deployment at JD Logistics, suggesting LLM-based address cleaning could reduce parcel rerouting and help other location-based services.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Online claim conflates per-rewrite correction rate with system-level re-routing reduction: Section 3.5 reports ~754 rewrites/day in Zhejiang, yet the abstract claims ~43% reduction in parcel re-routing among ~2 million daily parcels.","rationale":"The reader's explicit weakest assumption concerned the LBS feedback loop: the same geocoding/reverse-geocoding service is used for the RL reward, for the direct-evaluation ground truth, and for the deployment accuracy, which can inflate offline metrics if that service is biased. That is a real concern and I do not dispute it. However, the deployment-number mismatch is more directly load-bearing because the headline quantitative claim can be checked against numbers already present in the paper. Section 3.5 gives a daily rewriting volume of 754 and an accuracy curve; the abstract and contributions convert this into 'reduced over 40% parcel re-routing caused by abnormal addresses among around 2 million daily parcels.' That conversion is not derived. It is an internal-consistency issue rather than a disagreement with external consensus, so it should be settled by requesting the missing denominator and the definition of online accuracy. The offline contribution (the SFT + RAG + objective-alignment ablations and the retriever spatial-encoding analysis) is plausible and internally consistent, and the paper gives partial credit in the form of a real deployment and large-scale data. For that reason I would not reject the paper, but the deployment claim needs either revision or additional before/after data. This is the same conditional position the reader reached, so the verdict is unchanged; the stress-test pass adds a concrete check that can settle whether the concern actually lands.","tokens_in":20638,"tokens_out":6712,"duration_ms":71939,"concrete_test":"Ask the authors for the raw daily counts behind Figure 5 together with the total number of parcel re-routing events in Zhejiang (or re-routing rate per 1,000 parcels) for the 60-day deployment window and an equivalent pre-deployment baseline period. Compute the absolute reduction in re-routing events and its percentage. If the reduction is computed only as 'accuracy × 754 rewrites/day' and not as a measured before/after difference in re-routing events, then the abstract's 'decreased rate of parcel re-routing by approximately 43%' is not supported; the correctable claim would be 'around 40% of the 754 detected abnormal addresses per day were successfully rewritten.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing unsupported step is the deployment claim. Section 3.5 states AddrLLM is called only when abnormal addresses are detected, and Figure 5 reports a daily rewriting volume averaging 754 and an accuracy curve whose average is roughly 40–70% (i.e., over 40% of those 754 are corrected). The abstract and Section 1 contributions then claim the deployment 'significantly decreased the rate of parcel re-routing by approximately 43%' and 'reduced over 40% parcel re-routing caused by abnormal addresses among around 2 million daily parcels.' These are different objects. 754 rewrites/day is about 0.04% of 2 million parcels, so a 40% correction rate affects at most about 300 parcels/day. No before/after re-routing count for Zhejiang is reported, so there is no evidence that system-level re-routing caused by abnormal addresses fell by 40%. The 43% offline figure in Table 2 is a relative reduction in station-level geocoding inaccuracies (SoP 90.0 to AddrLLM 94.3 on the mixed test set), not a measured reduction in re-routing events. Additionally, the online 'accuracy' is defined in Section 3.5 as the percentage of abnormal addresses corrected by AddrLLM, but the paper does not specify how 'corrected' is verified. Without a baseline count of Zhejiang re-routing events before deployment, the headline deployment claim is not supported by the presented data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AddrLLM, an LLM-based address rewriting framework for logistics, combining multi-instruction supervised fine-tuning (SFT), an address-centric retrieval-augmented generation (RAG) module, and an objective alignment stage trained with rewards derived from JD Logistics' LBS system (geocoding, reverse geocoding, and courier-reported delivery coordinates). Offline experiments on nationwide data show AddrLLM outperforming existing methods on address entity prediction, direct rewriting hit rate, and geocoding accuracy (e.g., Acc@Station 94.3 vs. 90.0 for SoP). The model was deployed in Zhejiang province for about two months; the paper reports an average of 754 rewrites per day and that over 40% of detected abnormal addresses were corrected. The abstract and introduction further claim that deployment significantly decreased parcel re-routing by approximately 43% and reduced over 40% of re-routing among 2 million daily parcels.","tokens_in":20985,"tokens_out":4878,"duration_ms":49879,"significance":"If the deployment claim were substantiated, this would be a notable industrial application of LLM-based address rewriting with clear practical impact. The offline comparisons are extensive, use real-world data at scale, and include ablation studies that help isolate the contributions of SFT, RAG, and objective alignment. The paper also explicitly acknowledges several limitations, including the lack of a comprehensive error-type analysis and the conservative deployment design. However, the headline 43% re-routing reduction is not supported by the presented evidence; it conflates an offline station-level geocoding error reduction with a system-level re-routing outcome. The potential circularity between the RL reward and the evaluation ground truth also needs to be examined before the offline gains can be taken at face value.","major_comments":[{"comment":"The claim that deployment 'significantly decreased the rate of parcel re-routing by approximately 43%' and 'reduced over 40% parcel re-routing caused by abnormal addresses among around 2 million daily parcels' is not supported by the data presented. Section 3.5 reports only daily rewriting volume (average 754) and an accuracy defined as the percentage of abnormal addresses corrected by AddrLLM; no before/after re-routing counts are given. With roughly 2 million parcels per day, 754 rewrites is about 0.04% of parcels, so even a 40% correction rate affects only ~300 parcels daily. The 43% figure in §3.4.1 is a relative reduction in station-level geocoding inaccuracies (SoP Acc@Station 90.0 vs. AddrLLM 94.3, i.e., inaccuracy 10.0 vs. 5.7), not a measured re-routing reduction. Please either supply deployment-level re-routing statistics or revise the abstract and contributions to state the actual measured quantity.","section":"Abstract; §1; §3.4.1; §3.5"},{"comment":"The RL reward includes a reverse-geocoding score, r_revgeo(y,c)=cos(f(y), f(reverse(c))), and the direct evaluation Hit Rate derives its ground truth by reverse geocoding the delivery coordinates using the same JD service. Because the same service supplies both the training reward and the test ground truth, the evaluation cannot detect systematic biases or errors in that service, and calling the feedback 'bias-free' in §2.2 is not justified. Please provide an independent validation set (e.g., manually inspected samples) or an analysis of reverse-geocoding error rates to show that the reward and the Hit Rate metric are not circularly reinforcing the model's outputs. If such validation is not feasible, the paper should explicitly state this limitation and temper the corresponding claims.","section":"§2.2, Eq. (3); §3.1 Hit Rate"},{"comment":"The deployed-system 'accuracy' is defined as the percentage of abnormal addresses corrected by AddrLLM, but the paper does not specify how a correction is verified after deployment. If the verification uses the same geocoding or abnormal-address detection pipeline that triggered the rewrite, the reported correction rate may partly reflect the detector's own errors. Please describe the verification procedure used in the online system and, if possible, report a sample of rewrites that were manually checked.","section":"§3.5"},{"comment":"The deployment results for Zhejiang, Yulin, and Yangjiang are reported without a comparison period or a control group. To substantiate a re-routing reduction claim, the paper needs at least a before/after comparison of re-routing rates in the deployment region, or a controlled A/B experiment. As presented, the online section demonstrates stable operation and that some detected abnormal addresses are changed, but it does not demonstrate a reduction in re-routing events.","section":"§3.5; Appendix E"}],"minor_comments":[{"comment":"The number 43% is used for two different quantities (station-level geocoding inaccuracy reduction and parcel re-routing reduction); please standardize the terminology so readers can distinguish the offline metric from the deployment outcome.","section":"Abstract; §1; §3.4.1"},{"comment":"The text says monitoring covered 60 days, but the date range in Figure 5 (2024/05/19 to 2024/07/14) spans 56 days; please reconcile this discrepancy.","section":"§3.5; Figure 5"},{"comment":"References [61] and [62] are the same paper (Zhu et al., 2024); please deduplicate.","section":"References"},{"comment":"The t-SNE visualization of station-level embeddings is qualitative; reporting a quantitative station-level classification accuracy (e.g., k-NN accuracy on held-out addresses) would strengthen RQ5.","section":"§3.4.5; Figure 3"},{"comment":"The prompt in Appendix C says 'Related Address:{addresses returned by retriever}' without specifying the number of retrieved addresses, whereas §3.2 states top-10 is used; please make the prompt template consistent with the experimental setting.","section":"§2.3; §3.2; Appendix C Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong industrial case study with large-scale offline experiments, but the headline deployment claim goes beyond the reported evidence. The mismatch between the per-rewrite correction rate (754/day) and the system-level re-routing reduction (43% of 2M parcels) is a serious overreach that the authors should be asked to fix by either adding real deployment re-routing statistics or rewriting the claim. The 'bias-free' terminology in §2.2 is also a marketing-style claim that should be supported or softened. I would recommend major revision rather than rejection because the offline framework and ablation results are likely salvageable, and the deployment claim can be corrected or substantiated with additional data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AddrLLM is a serious engineering effort with a real deployment, but the headline numbers overstate what the evidence supports. The 43% 're-routing reduction' in the abstract is actually the offline improvement in station-level geocoding accuracy (SoP 90.0 to AddrLLM 94.3 on the mixed test set), not a measured reduction in re-routing events. The deployment section reports a daily average of 754 rewrites and an accuracy curve of roughly 40–70% – not a before/after count of re-routing in Zhejiang. With 2 million daily parcels, 754 rewrites is 0.04% of the volume, so even perfect correction of every flagged address would touch a tiny fraction of parcels, not 40%. The authors need to rewrite the abstract and the contribution bullet, and show the actual re-routing numbers if they have them.\n\nWhat is genuinely new is the combination: an LLM fine-tuned on multi-task address data (parsing, entity prediction, rewriting), an address-centric RAG whose retriever is fine-tuned on a geocoding task to produce spatial embeddings, and a PPO alignment stage that uses the JD LBS system as a reward oracle. The ablation study is internally consistent and shows each component (SFT, RAG, OA) contributes. The four-month deployment at JD Logistics is real and the system appears stable at modest volume. The spatial retrieval idea is worth building on.\n\nThe bigger concern is circularity. The RL reward in Eq. 3–4 uses reverse geocoding and geocoding from the same JD system, and the direct evaluation ground truth in Section 3.1 is derived by reverse geocoding the delivery coordinates. The model is trained and evaluated on the same oracle. Calling this 'bias-free' (Section 2.2) is a strong claim without supporting evidence. If the LBS system is noisy or systematically wrong, the error is baked into both training and evaluation. The paper needs an independent gold standard, or at least a manual error analysis on a random sample, to back up the claimed magnitudes.\n\nMinor issues: no error bars, no released code or data, and the retriever quality evaluation is limited to 12 stations. These are not fatal, but they limit confidence.\n\nWho is this for? Practitioners building LLM-based address processing for logistics or geocoding, and the applied IR/database community. It is a useful systems paper, but not yet a reliable scientific claim about re-routing reduction. I would send it to peer review with major revisions: correct the claims, provide the missing deployment denominator, and address the circularity. After that, it could be a solid contribution.","headline":"AddrLLM is a real industrial LLM+RAG system with a genuine deployment, but the headline 43% re-routing claim is not supported by the evidence in the paper.","tokens_in":21526,"tokens_out":3332,"would_cite":true,"duration_ms":32443,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows a retrieval-augmented LLM can rewrite abnormal Chinese addresses, correcting 43.9% offline and cutting live parcel re-routing by over 40%.","keywords":["address rewriting","large language models","retrieval-augmented generation","geocoding","logistics","reinforcement learning","query reformulation","abnormal address detection"],"falsifier":"Take a random sample of addresses AddrLLM rewrote during deployment, verify the intended destination by direct courier follow-up or independent ground-truth inspection, and compare delivery success for rewritten versus original addresses; if rewritten addresses do not reach the correct station more often than the originals, the claimed 40% re-routing reduction collapses.","tokens_in":20413,"feed_emoji":"📦","tokens_out":6151,"duration_ms":62936,"temperature":0.7,"pith_summary":"Abnormal addresses—those with missing regions, nested cites, aliases, or misspellings—routinely send parcels to the wrong delivery station and force costly re-routing. The paper argues that a retrieval-augmented large language model, fine-tuned on address parsing, entity prediction, and rewriting tasks, can fix these errors in a unified model without retraining when new addresses appear. The model is aligned by a reward computed from the logistics company's own geocoding service and courier-reported delivery coordinates, which the authors call bias-free feedback. Offline, the system corrects 43.9% of abnormal addresses; in a live deployment covering roughly two million daily parcels, it reduced abnormal-address re-routing by over 40%.","feed_headline":"LLM rewrites bad addresses, cutting parcel rerouting by 40%","feed_subtitle":"It corrects 43.9% of abnormal addresses offline and, live in Zhejiang, cut re-routing by over 40 percent.","key_machinery":"The central machinery is a retrieval-augmented LLM whose retriever is fine-tuned on geocoding so that relevance means spatial proximity, and whose generator is trained by supervised fine-tuning plus PPO with a reward computed directly from the logistics geocoding system. The reward is the sum of a semantic cosine score, a reverse-geocoding score, and a geocoding-distance score, which lets the model learn to correct addresses that the existing system cannot handle without a trained reward model or manual labels.","core_discovery":"On its own terms, the paper claims that LLM-based address rewriting works in production. AddrLLM combines multi-instruction supervised fine-tuning on 20 million parsing, 20 million entity-prediction, and 20 million rewriting samples with an address-centric retrieval module whose retriever is fine-tuned on 200 million geocoding pairs so that retrieved addresses are geographically close rather than merely semantically similar. A PPO objective-alignment stage then optimizes a three-term reward: semantic similarity to the original address, similarity to the reverse-geocoded delivery coordinates, and geocoding distance to the courier-reported delivery point. The reported offline results include 91.8% trigger prediction and 90.3% accuracy on entity prediction, 89.7% hit rate on direct rewriting, 94.3% station-level geocoding accuracy, and 43.9% correction of abnormal addresses, with 99.9% robustness on standard addresses; the authors attribute these gains to the synergy of all three modules.","pith_inferences":["The same recipe—use a downstream operational outcome as the RL reward instead of a learned reward model—could transfer to other rewriting or normalization tasks where a measurable system outcome exists, such as product-query rewriting or clinical-text normalization.","The spatially fine-tuned retriever may be reusable as a geocoding-aware encoder for other geo-NLP tasks, since its embedding distances were shown to correlate linearly with real geographical distances.","A natural next experiment, acknowledged as missing by the paper itself, is to report correction rates by error type (missing region, nested address, alias, irrelevant words, misspelling); that breakdown would show where the LLM's gain over older systems actually comes from.","If the geocoding reward is noisy near delivery-station boundaries, the station-level gains may overstate real delivery improvements; a courier-confirmed outcome study would settle the practical value."],"forward_implications":["Offline, AddrLLM corrects 43.9% of abnormal addresses, a correction rate 24.2 percentage points higher than the best baseline.","The model keeps 99.9% of standard addresses unchanged and correct, so applying it to normal traffic poses little risk of breaking good addresses.","Station-level geocoding accuracy rises to 94.3%, corresponding to roughly a 43% reduction in station-level misrouting compared with the existing geocoding service.","Live deployment in Zhejiang corrected over 40% of abnormal addresses per day and remained stable over 60 days including a peak sales event.","Because address knowledge is stored in an external retrieval database, adding new addresses to the database does not require retraining the LLM."],"supporting_citations":[{"why":"Supplies the abnormal-address detection framework used to decide which addresses should be rewritten in the deployed system.","marker":"[18]"},{"why":"Represents the prior seq2seq address-rewriting approach for geocoding that AddrLLM extends and outperforms.","marker":"[45]"},{"why":"Provides the G2PTL logistics pre-trained model that serves as the main state-of-the-art baseline and initializes the retriever's text encoder.","marker":"[55]"},{"why":"Furnishes the retrieval-augmented generation paradigm that motivates decoupling address knowledge from the LLM's reasoning.","marker":"[14]"},{"why":"Supplies the Proximal Policy Optimization algorithm used for the bias-free objective alignment stage.","marker":"[47]"},{"why":"Provides the KL-divergence regularization formulation that prevents the aligned model from drifting too far from its initialization.","marker":"[42]"},{"why":"Supplies the BERT architecture used for the retriever and as the semantic embedding model in the reward calculation.","marker":"[9]"},{"why":"Provides the Qwen-7B base LLM used both as a baseline and as one of the two base models for AddrLLM.","marker":"[2]"},{"why":"Provides the Baichuan-7B base LLM used as the other base model and as a comparison baseline.","marker":"[3]"}],"fun_headline_variants":["LLM rewrites bad addresses, cutting parcel rerouting by 43%","AddrLLM: retrieval-augmented LLM cuts parcel rerouting by 43%","Address-rewriting LLM reduces parcel rerouting by 43% in production","Retrieval-augmented LLM corrects abnormal addresses, cuts rerouting 43%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the logistics system's geocoding, reverse geocoding, and courier-reported delivery coordinates provide an unbiased definition of a correct rewritten address; if that feedback is noisy or systematically biased, the measured correction and re-routing reductions would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["LLM rewrites bad addresses, cutting parcel rerouting by 43%","AddrLLM: retrieval-augmented LLM cuts parcel rerouting by 43%","Address-rewriting LLM reduces parcel rerouting by 43% in production","Retrieval-augmented LLM corrects abnormal addresses, cuts rerouting 43%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2485,"prompt_tokens":987,"completion_tokens":1498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1407}},"tokens_in":603,"tokens_out":1498,"duration_ms":14854,"temperature":1.0,"reasoning_tokens":1407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:04:30.971378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of addresses AddrLLM rewrote during deployment, verify the intended destination by direct courier follow-up or independent ground-truth inspection, and compare delivery success for rewritten versus original addresses; if rewritten addresses do not reach the correct station more often than the originals, the claimed 40% re-routing reduction collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the abnormal-address detection framework used to decide which addresses should be rewritten in the deployed system."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the prior seq2seq address-rewriting approach for geocoding that AddrLLM extends and outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Baichuan-7B base LLM used as the other base model and as a comparison baseline."}],"review_version":1}