REVIEW 4 major objections 5 minor 32 references
VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking
T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read This paper claims that autonomous valet parking can be performed zero-shot in unseen underground garages by pairing a bird's-eye-view trajectory generator with a vision-language model that reads signs and follows natural-language instructio
desk verdict The benchmark and architecture are the real contributions; the headline performance numbers need error bars before they can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the hybrid long-short-term memory system feeding a hierarchical decision module. The short-term memory is a queue of recent sign detections with timestamps, bounding boxes, and vehicle yaw, filtered to keep only relevant, current, same-direction cues; it stabilizes VLM decisions when no sign is visible in the current frame. The long-term memory is a directed topological graph of junction nodes, landmark nodes, and direction-vector edges, updated by user feedback and reused on later episodes. The hierarchical decision module prefers a stored graph route when one exists, falls back on the cached VLM action otherwise, and maps the chosen action onto a kinematically
What would settle it
Measure, across the Regular benchmark episodes, how often the generated candidate trajectory set contains at least one trajectory whose endpoint lies within the target area. If that frequency is materially below the reported 65.2% success rate, the bottleneck is BEV perception, not the VLM or memory modules; if the frequency is high but success is low, the bottleneck lies in decision-making.
Extended reading notes
Core claim
The central claim is that the combination of kinematically filtered candidate trajectories from BEV lane perception with a low-frequency VLM decision-maker, plus two complementary memories, produces zero-shot vision-language navigation in underground parking garages. The VLM is not asked to plan in metric space; it chooses among a small set of drivable trajectories derived from predicted lane centerlines, and the memories let it act on cues absent from the current frame. The paper demonstrates that this architecture outperforms both vision-language navigation baselines and VLM-based driving baselines, that removing either memory degrades success, and that the long-term memory reduces VLM cal
Load-bearing premise
The whole pipeline rests on the BEV lane-detection model, trained on open-road data, producing a correct lane centerline in an unfamiliar garage, because the candidate trajectories are generated from those lane lines and the VLM can only choose among them.
Editorial extensions
If this is right
- Valet parking can be initiated in an unseen garage from a natural-language instruction such as 'park near the elevator' without a pre-built HD map.
- A vehicle can safely follow a VLM's high-level commands by restricting its choices to kinematically feasible candidate trajectories, avoiding the frequent U-turns that make typical VLN policies impractical on real cars.
- Repeated trips to the same garage build a lightweight topological graph that raises success rate and cuts VLM API calls and cost by about 75%.
- The new benchmark, with ten photorealistic garage scenes and over 1,000 episodes, provides a standardized testbed for evaluating vision-language navigation in underground parking.
- Caching recent sign detections in short-term memory lets the VLM make consistent decisions even when the current image contains no visible navigation cue.
Reading between the lines
- The short-term-cue memory idea could transfer to other map-free indoor navigation settings, such as delivery robots or warehouse vehicles, where signage is the dominant navigation cue.
- The long-term topological graph, stored as direction vectors rather than steering commands, is naturally shareable across a fleet of vehicles; pooled successful routes could form a collective map-free memory.
- The directional filter that drops remembered signs when the vehicle turns assumes sign relevance is tied to yaw; multi-level or looped garages may require a position-aware or lane-aware filter.
- If a cheaper or faster VLM were substituted, the memory components might partially compensate, and the challenging benchmark subset, with composite goals like 'EV charger near the exit', would directly stress-test that substitution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VLN-AVP, a zero-shot vision-language navigation framework for autonomous valet parking. The system replaces pre-built HD maps with MapTR BEV lane-centerline perception, uses a VLM (Gemini-3-flash) for high-level decisions, and adds a hybrid memory: a short-term perception memory based on Grounding DINO detections and a long-term topological memory with validity bits and graph-based retrieval. A hierarchical decision module selects among RS-curve candidate trajectories. The authors also introduce a 3DGS-based dataset of 10 underground garages with over 1,000 VLN episodes and report simulation and real-world experiments against VLN and AD baselines, including ablations. The headline claims are an over 25% SR improvement over VLN methods, over 15% over AD methods, and 63.3% SR in 30 real-world trials.
Significance. If substantiated, the paper would make a useful contribution: it is the first VLN benchmark for underground parking, it demonstrates a map-free zero-shot parking navigation pipeline with human-in-the-loop correction, and the hybrid memory reduces VLM API calls and cost. The system design is modular and mostly reproducible from existing components. However, the central empirical claim is a quantitative performance advantage, and the evidence currently consists of single-point estimates without variance, confidence intervals, significance tests, or full protocol details. The contribution is therefore promising but not yet established at the level claimed in the abstract and conclusions.
major comments (4)
- [Abstract; Tables III, IV, V; Fig. 8] All reported success-related metrics are single-point estimates. No number of repeated runs/seeds, standard deviations, confidence intervals, or significance tests are given. The headline 'over 25% improvement' and 'over 15% improvement' are thus not statistically supported. In Table IV, for example, the difference between Ours (63.3%) and Ours w/o LTM (53.3%) on 30 trials is exactly 3 episodes, which is within plausible sampling noise. Please add repeated evaluations, per-episode counts, confidence intervals, and at least a basic significance test for the main comparisons.
- [Sec. IV-A, IV-B, V] The evaluation protocol is underspecified. Please state the number of episodes in the Regular and Challenging subsets, per-scene episode counts, the exact success criterion (e.g., stopping within the target bounding box, distance threshold), how instructions and targets are sampled, and whether all methods run on identical episode sets. In addition, the dataset is not currently released: Sec. V says the authors 'plan to further improve and publicly release' it, so the benchmark cannot yet be independently used by the community.
- [Sec. III-A, Eq. (1)] The method's entire action space T_t is generated from MapTR lane-centerline predictions in unseen underground garages. The paper provides no quantitative evaluation of MapTR on parking-garage scenes. If lane centerlines are sparse, disconnected, or geometrically wrong, the candidate trajectory set cannot contain a drivable path toward the requested target, and neither the VLM nor the memory modules can recover. Please report MapTR's lane extraction quality on the VLN-AVP scenes (e.g., centerline recall, trajectory reachability rate to target locations) and discuss how the system handles cases where no candidate trajectory leads to the target.
- [Sec. III-B, III-D, IV-E; Table VI; Fig. 8] The ablations attribute the performance gains to the memory modules, but they inherit the statistical issue. Figure 8, which is the main evidence for LTM's 'learning ability', has no error bars or aggregation details (each point is a single episode? a window? a single scene?), and its axes are unlabeled. Moreover, the memory system depends on several free parameters introduced in Secs. III-B and III-D (K_STM, delta_theta_max, delta_t_max, theta_threshold) whose values are not reported and for which no sensitivity analysis is provided. Please add robustness experiments or justify why the chosen values are not the result of tuning that favors the proposed method.
minor comments (5)
- [Sec. III-B, Eq. (4)] Eq. (4) writes the prompt as [I_front, {I_detect,j}, S, g], but the STM frame defined in Sec. III-B is m = (I_detect, B, t, theta). Clarify which components of the STM frame are actually fed to the VLM.
- [Sec. IV-B, Ref. [32]] The simulator is described as being built on the Real2Sim2Real simulator in REAP [32], which is an author preprint. Please specify whether the simulator code will be released, or provide enough implementation documentation for others to reproduce the benchmark.
- [Table II] The 'Render Method' column lists '3DGS' for all datasets, which is ambiguous. Clarify how each existing dataset was rendered and whether the VLN-AVP scenes use 3DGS exclusively.
- [Sec. IV-E, Fig. 8] Add axis labels, units, and a legend to Figure 8. Also state how many episodes each plotted point aggregates and whether the curve is over a single scene or multiple scenes.
- [Sec. IV-D, Table IV] The real-world comparison includes only LaMPilot. State why other AD baselines were omitted, and describe the real vehicle's sensor setup and the source of ground-truth localization/target positions.
Circularity Check
No significant circularity: the system is evaluated on an independently constructed benchmark and the only self-citation is simulator infrastructure.
full rationale
The paper's contribution is a modular system integration rather than a derived mathematical law: MapTR produces candidate trajectories T_t = f_optim(L_t), the VLM converts instructions into sub-goals and meta-actions, and the short/long-term memories select among candidate trajectories. None of these modules is fitted to the reported success rates, and SR/OSR/SPL are measured on the constructed benchmark rather than derived from the method's own equations. Baselines receive the same BEV perception and the same VLM core, so the comparison isolates the decision and memory components; this is a fair-comparison design, not a fitted-input-called-prediction loop. The only apparent self-citation is REAP [32], used to describe the customized simulator's design; it is shared infrastructure from the same group (Changze Li) but does not encode the target results or any success-rate numbers. The lack of error bars or significance tests is a statistical-reliability concern, not circularity, because the reported numbers are not equivalent to the method's inputs by construction. No circular step is present.
Assumptions & free parameters
free parameters (4)
- K_STM (short-term memory cache size) =
6
- Delta theta_max (directional filter yaw threshold) =
not reported
- Delta t_max (temporal filter age threshold) =
not reported
- theta_threshold (LTM trajectory cosine-similarity threshold) =
not reported
assumptions (6)
- domain assumption MapTR trained for open-road HD-map construction transfers to underground parking without fine-tuning (Sec. III-A).
- domain assumption Grounding DINO with the single prompt 'sign' detects all navigation-critical signage in parking garages (Sec. III-B).
- domain assumption Gemini-3-flash reliably maps prompt+images to the four actions {forward, left, right, stop} in the tested garages (Sec. III-C).
- domain assumption 3D Gaussian Splatting reconstruction from handheld LiDAR/fisheye scans is a faithful proxy for real vehicle sensors and dynamics (Sec. IV-A.1).
- domain assumption Benchmark ground-truth trajectories and success boxes are correct and unambiguous (Sec. IV-A.2).
- ad hoc to paper LTM validity-bit update rules (v=1 on success, v=0 on user/system failure) correctly infer environment changes and target correctness from sparse feedback (Sec. III-D).
invented entities (2)
-
Short-Term Perception Memory (STM)
-
Long-Term Topological Memory (LTM)
Cite this review
Pith. "Pith review of VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking." pith.science (2026). https://pith.science/paper/G5E4D5AP
@misc{pith2026260717767,
author = {Pith},
title = {Pith review of: VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking},
year = {2026},
howpublished = {\url{https://pith.science/paper/G5E4D5AP}},
note = {Machine review of arXiv:2607.17767}
}
read the original abstract
Existing methods in Autonomous Valet Parking (AVP) typically rely on pre-built maps, which severely restricts their scalability to unseen environments and open-vocabulary targets. Inspired by the application of Vision-Language Models (VLMs) in Vision-Language Navigation (VLN) tasks, we propose VLN-AVP, a zero-shot navigation framework for AVP tasks. By combining the precise spatial perception of a Bird's-Eye-View (BEV) model with the general intelligence of VLMs, our framework 1) eliminates the dependency on pre-built maps, 2) interprets semantic environmental contexts in parking scenarios, and 3) enables intuitive navigation following natural language instructions. Specifically, we introduce a hybrid memory system: a short-term perception memory tracks semantic visual cues to address the limitations of VLM's single-frame reasoning in existing methods, while a long-term topological memory facilitates stable policy learning from past experiences. To bridge the gap in existing benchmarks, we also present the VLN-AVP dataset and benchmark. Featuring 10 high-fidelity parking scenes and over 1,000 navigation episodes, it has the largest number of garage scenes to date and is the first VLN benchmark for underground parking. Extensive experiments demonstrate that in simulation, our method achieves an over 25% improvement in success rate compared to VLN methods and an over 15% improvement compared to other autonomous driving methods. Furthermore, it attains a leading success rate in real-world vehicle experiments, proving its practical feasibility.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Autonomous valet parking system architecture,
M. Chircaet al., “Autonomous valet parking system architecture,” in ITSC, 2015, pp. 2619–2624
2015
-
[2]
Avp-slam: Semantic visual mapping and localization for autonomous vehicles in the parking lot,
T. Qinet al., “Avp-slam: Semantic visual mapping and localization for autonomous vehicles in the parking lot,” inIROS, 2020, pp. 5939– 5945
2020
-
[3]
Less-map: Lightweight and evolving semantic map in parking lots for long-term self-localization,
M. Liuet al., “Less-map: Lightweight and evolving semantic map in parking lots for long-term self-localization,” inICRA, 2024, pp. 17 990–17 996
2024
-
[4]
Drivevlm: The convergence of autonomous driving and large vision-language models,
X. Tianet al., “Drivevlm: The convergence of autonomous driving and large vision-language models,”arXiv:2402.12289, 2024
arXiv 2024
-
[5]
Drivelm: Driving with graph visual question answer- ing,
C. Simaet al., “Drivelm: Driving with graph visual question answer- ing,” inECCV, 2024, pp. 256–274
2024
-
[6]
Lampilot: An open benchmark dataset for autonomous driving with language model programs,
Y . Maet al., “Lampilot: An open benchmark dataset for autonomous driving with language model programs,” inCVPR, 2024, pp. 15 141– 15 151
2024
-
[7]
Dilu: A knowledge-driven approach to autonomous driving with large language models,
L. Wenet al., “Dilu: A knowledge-driven approach to autonomous driving with large language models,” inICLR, 2024
2024
-
[8]
Vision-and-language navigation: A survey of tasks, methods, and future directions,
J. Guet al., “Vision-and-language navigation: A survey of tasks, methods, and future directions,”ACL, pp. 7606–7623, 2022
2022
Show all 32 references
-
[9]
Automated valet parking and charging for e- mobility,
U. Schwesingeret al., “Automated valet parking and charging for e- mobility,” inIV, 2016, pp. 157–164
2016
-
[10]
Inverse perspective mapping-based neural occupancy grid map for visual parking,
X. Muet al., “Inverse perspective mapping-based neural occupancy grid map for visual parking,” inICRA, 2023, pp. 8400–8406
2023
-
[11]
Avp-loc: Surround view localization and relocaliza- tion based on hd vector map for automated valet parking,
C. Zhanget al., “Avp-loc: Surround view localization and relocaliza- tion based on hd vector map for automated valet parking,” inIROS, 2021, pp. 5552–5559
2021
-
[12]
Think global, act local: Dual-scale graph transformer for vision-and-language navigation,
S. Chenet al., “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” inCVPR, 2022, pp. 16 537– 16 547
2022
-
[13]
Navid: Video-based vlm plans the next step for vision- and-language navigation,
J. Zhanget al., “Navid: Video-based vlm plans the next step for vision- and-language navigation,” inRSS, 2024
2024
-
[14]
Navila: Legged robot vision-language-action model for navigation,
A. Chenget al., “Navila: Legged robot vision-language-action model for navigation,” inRSS, 2025
2025
-
[15]
Navgpt: Explicit reasoning in vision-and-language navigation with large language models,
G. Zhouet al., “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,” inAAAI, 2024, pp. 7641– 7649
2024
-
[16]
Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation,
J. Chenet al., “Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation,” inACL, 2024, pp. 9796– 9810
2024
-
[17]
Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion,
M. Zhanget al., “Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion,”IEEE Robotics and Automation Letters, 2025
2025
-
[18]
Human-like navigation in a world built for humans,
B. Chandakaet al., “Human-like navigation in a world built for humans,” inCoRL, 2025
2025
-
[19]
Gpt-driver: Learning to drive with gpt,
J. Maoet al., “Gpt-driver: Learning to drive with gpt,” arXiv:2310.01415, 2023
2023 arXiv
-
[20]
Drivegpt4: Interpretable end-to-end autonomous driving via large language model,
Z. Xuet al., “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,”RA-L, 2024
2024
-
[21]
Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation,
H. Fuet al., “Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation,” inICCV, 2025, pp. 24 823–24 834
2025
-
[22]
Lightemma: Lightweight end-to-end multimodal model for autonomous driving,
Z. Qiaoet al., “Lightemma: Lightweight end-to-end multimodal model for autonomous driving,”arXiv:2505.00284, 2025
2025 arXiv
-
[23]
Drive like a human: Rethinking autonomous driving with large language models,
D. Fuet al., “Drive like a human: Rethinking autonomous driving with large language models,” inWACVW, 2024, pp. 910–919
2024
-
[24]
Maptr: Structured modeling and learning for online vectorized hd map construction,
B. Liaoet al., “Maptr: Structured modeling and learning for online vectorized hd map construction,” inICLR, 2023
2023
-
[25]
Optimal paths for a car that goes both forwards and backwards,
J. Reeds and L. Shepp, “Optimal paths for a car that goes both forwards and backwards,”Pacific journal of mathematics, vol. 145, no. 2, pp. 367–393, 1990
1990
-
[26]
Grounding dino: Marrying dino with grounded pre- training for open-set object detection,
S. Liuet al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” inECCV, 2024, pp. 38–55
2024
-
[27]
3d gaussian splatting for real-time radiance field rendering
B. Kerblet al., “3d gaussian splatting for real-time radiance field rendering.”ACM Trans. Graph., vol. 42, no. 4, pp. 1–14, 2023, art. no. 139
2023
-
[28]
Sups: A simulated underground parking scenario dataset for autonomous driving,
J. Houet al., “Sups: A simulated underground parking scenario dataset for autonomous driving,” inITSC, 2022, pp. 2265–2271
2022
-
[29]
Development of occupancy prediction algorithm for under- ground parking lots,
S. Wang, “Development of occupancy prediction algorithm for under- ground parking lots,”arXiv:2409.00923, 2024
2024 arXiv
-
[30]
Letsgo: Large-scale garage modeling and rendering via lidar-assisted gaussian primitives,
J. Cuiet al., “Letsgo: Large-scale garage modeling and rendering via lidar-assisted gaussian primitives,”ACM Trans. Graph., vol. 43, no. 6, pp. 1–18, 2024
2024
-
[31]
A point cloud dataset and deep learning method for semantic segmentation of underground garages,
H. Yueet al., “A point cloud dataset and deep learning method for semantic segmentation of underground garages,”CACAIE, vol. 40, no. 23, pp. 3726–3749, 2025
2025
-
[32]
Reap: Reinforcement-learning end-to-end autonomous parking with gaussian splatting simulator for real2sim2real transfer,
C. Liet al., “Reap: Reinforcement-learning end-to-end autonomous parking with gaussian splatting simulator for real2sim2real transfer,” arXiv preprint arXiv:2605.08713, 2026
2026 arXiv
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.